Summary
Notes
Transcript
Absolutely.
All right. Rock and roll. Are we ready? Ready. Roll the punch. Thursday. Dig deep. Yes. Woo! Woo-woo! Let's go. Okay. My distinct left, hold on for a second. Close the buzz. Okay, one plus. Good timing. Can you just speak English? So it's--My pleasure to introduce you to my friend Manuel Bronstein. Manuel is a storied prophetProduct leader. In fact, two little stories. We have a mutual friend who's at Amazon, Alexa.
So at one point in his career, Manuel wasAnd this guy literally said that they could tell the difference in terms of how good the competition was before and after Manuel was at a system. When he left, all of a sudden everything got easier. And funny enough or not funny enough, after he left Roblox, he then, after at Google he was this chief product officer at Roblox, after he left Roblox, the stock went down. So either he's good luck or bad luck.
It depends how you look at it. We need to turn that switch before you leave. Exactly. Just tell us what you're doing next. And I'd like everybody to pull out your agent and start playing. So... Anyway, so the plan today is to have an open discussion about product. And now with this new tool, product-making AI, how that's changed. Um. And I have a bunch of questions, and obviously we'll do it the other day. As questions come up for you, feel free to bomb away. So first of all, I'm going to welcome you to the room. So start by just giving us, you know, these guys like to know your resume, so a little bit where you started. I love the interview. And the story made me feel old.
A story brought it to me many years.
So, as Jonathan mentioned, it's great to be here. I grew up in Venezuela, major in electrical engineering, and finally ended up doing what most engineers do, I went to Stanford, Virginia to work for Granville, which is not a common path, it seems like it's a common path in Venezuela. You know, EMT is a great company to learn business, marketing, and a lot of things, I wanted to go to the United States.
I mean, what's a good opportunity to-Join a multinational company and then see if the opportunity is. I then left it for them to work at a startup For those who love the World Cup, we were doing all the cards in 2000. We were all kinds of panini and all these collections and we wanted to do that online. Finally, at that time, there was no mobile phones or apps. You know, headphones was the best thing that you could find. There was no social graph.
There was no tokens or anything on the blockchain. So we were doing it the old fashioned way. You get your cards, you trade them through email, and then you try to complete the album. It was fun. The company lasted many years, but it became more of a life of business. So I decided to come here to the Bay Area and went to the other school in the Bay Area, Berkeley, to do my MBA. And then after that, I had the fortune to join Xbox three years before the launch of the Xbox 360.
So it was the second console that Microsoft was doing, the first one was a bloodbath, not only in terms of market share but also how much money the company lost. The second question actually pretty well. I had an opportunity to work on many things at Xbox, from controllers to Xbox Live, to building a game studio, to bringing Netflix to Xbox, which by the time, I know it sounds cold place right now, there was no streaming on TV in 2016.
At some point I was in Seattle and I decided that I wanted to do something more entrepreneurial and came here to join Zynga. And I went to a company that was about 400 people. for those of you who have gone to Singapore, maybe you know Farbill, or maybe you know the Singapore Poker, or some of the games that the company did. It was a big company, a lot of learnings for what happens when you build on top of another platform that you don't control.
The company went crazy on, and we were like the hardest thing in town, went public, and then the platform changed, and then also the mobile transition happened. And then essentially all consumer products at YouTube. So YouTube in the living room, YouTube on the whole app, on desktop, YouTube social, YouTube gaming, launch YouTube music, launch YouTube kids. And it was a fun time when YouTube was growing for about 400 million users, 4 million users a day, and about a billion hours a day. So it was a super fun experience. And then I got asked to move to the consumer product for the Google Assistant.
This is Google Assistant 3 LMS. For those of you who are tracking, the curve was OK. It wasn't that great. A lot of the things that are easier right now because of your building the right models and the right architecture were really hard there. Having said that, the ideas were the same. The notion that there was going to be an agent that was going to do stuff for you were the things that we talked about.
In fact, the mission statement was that Google Assistant is the best way to get things done. My joke at the time was that if you do work out the committee hours you were watchingAll right, Mark. After that,I had the fortune. in late 2020, beginning of 2021, and was there for five years leading first as chief product officer and then took over also the business side of the company for the last two years, and more recently getting advice from the company.
And now I'm on that phase again where I'm investing advice and I'm figuring out how what's next and how can I help John and the few guys here. I joined the world in your times in 2021 and recently joined the power of match group for those of you who like that dating apps. If you want to complain about the media or if you need a date.
Whatever you need. It's all here. Okay, great. So we've been talking about products for a long time. Thank you. I rememberBut I want the question I want to be with. If you think about the call of the top three to fiveOkay. What kind of biggest takeaways you've had in your career about building products?
Which of them changed because of AI and AI? And maybe when-When you read that, when you're looking to the answer perhaps, you read the things you started doing in robots with AI and how that changed how things were. The way or the thinking that goes into the product doesn't inform application with AI. It's enabling a lot of things to go faster. And over time, it may replace some of the things that it hasn't replaced just yet.
But any good product starts with--A point of view. and people may say it's for the data and analysis, and maybe that's part of it. Call it an evolution, call it a worldview. But it's someone that believes that there's something that could be better in the world, or could be done better. Mark Pinchus who founded Liga, he's used to the framework, he's wrote a book and he used to say that the company who would better than you?
So most products or more things actually have something that has already been proven, maybe something that is better than the other product, and then there's something that is new. But all of those things, I mean, intuition, it's used very loosely, but to me, intuition starts with what you know about the world that surrounds you, what you know about the customers, what you know about the space, or any idea or need that you personally have.
That isn't changing, right? That's still the core and the exit. Maybe you say I put a lot of data and maybe help you synthesize the data to come up with ideas. You actually can brainstorm with AI. Sometimes they tell you that everything you're saying is amazing. So you have the... What was that, Bing? Bing, yeah.
Bing, we talked about the fact that it's your best friend. Gemini.
We talk every day.
Just talk to Adam, and tell him that you're on the phone, and make some things with them.
But it's good, you can help us, but you can help us, and I say that at the end of the day, the beginning of that program for you, of where the world is going, where this product could be better, what needs to change. Maybe in the future AI decides that, I mean, just to use as an example, People should create videos by themselves and post them on their website. I don't know if that's something that AI is going to come up with or some individual is going to say.
Maybe we should build that. Now the difference that you have right now and one of the things that changes is that from that core idea, Do something, Adelon. Now that's moving at the speed of light. So what used to take maybe I have an idea, I have to write it down in a document, I need to find a designer, three engineers, someone to prototype this for me so that I can see it and I can feel it, I can touch the phone or whatever.
That's actually something that now I can do by myself. And at least that first phase of seeing what you're thinking about materializing in some way, shape, or form, it's moving incredibly fast. And that helps a lot with the other parts of the probability that doesn't necessarily Change which is When you release a product, or when you have a product, one of the things that you start doing is iterating and testing and validating your assumptions. You have assumptions.
I'll tell you--You think about like just station product also. It can be in the sense that, I'll give you my personal example. I started building a product by myself in this free time and it was very easy to go in the prototype from what you would have considered in the past an MVP, a Minimal Viable Product, which should have been like a subset of a subset of a subset of the features. I actually went all the way to the long-term product in the prototype.
Which by the way is wrong because I'm not validating anything. I'm actually adding all the flows just for the sake of completeness and that's not giving me the opportunity to test the core concept, test the core idea, test the core loop of the product to make sure that there's an audience for it.
Again, Martin said the-you know, prices that we use in the company, which is... Get married in some way to your intuition and your insight, but not to your ideas. And what he's meaning is that actually you may come up with a problem that you want to solve, and you understand the problem. You want to come up with one idea to solve it. That idea, when you test it, may be wrong, but that doesn't mean that the problem statement was wrong.
You may have read that the way you went about solving it was wrong. So yes, sometimes you can go too fast, and by going too fast you can have too many bells and whistles, and before you get to really test the product you build something very complicated, and you don't get to test the core idea. they have to add a third one. So in general, if you think about a product cycle, Start with some vision, mission statement, and idea of where you want to go.
If you are a serious company about it, you then set some goals and some strategy of how you're going to win and how you're going to quantify that what you're building is good. That doesn't change. Then you're going to go build it, test it, validate it, then you're going to get iterations because you're going to get data. When you get that product in the market, you're going to iterate on that product. And then you're going to refine and continue the look. And at the end of the day, hopefully when you're inside a company and you're building products or you're building features, it's with a one goal in mind.
No, yeah, let me build on that. So... What your building a product? Usually, You kind of said that defines or tries to capture the value that the product is delivering to the customer. So I'll give you an example. YouTube eventually, when YouTube started, the company started counting views. How many views you get in a video, how many views you get in a video. And then we quickly realized that views was the wrong metric because views is something that if you have a very sexy thumbnail, actually a misleading thumbnail, you can get a lot of people to click on that and watch it, right? It's clicking. And we moved to watch that.
If you look at Zynga Poker right now, maybe it has changed over the years, but eventually while I was running it and we were building things, we had... We have poker tables of different states, we have tournaments, we added slot machines, we had some bonus thing that you can get on a list. At the end of the day, the core loop was open the app,Seat up over the table, I'm gonna play ahead. and the government of India.
Right. If people actually like opening the table, and they want to come back and play again, you have something that is good. Now you can start layering other things. You can layer a tournament. You can decide that you want to have a different type of poker game. Maybe you have this slot machine and so forth. So yes, with AI, I could have tried to build all those things. But it would have been really hard to validate if the code group was working.
Because then you have seven different models in order with different things. And it's a lot harder to understand what the problem is. So it better still to do those things in some incremental fashion. So even though I can do it faster, you actually. You're better of having conviction of what you believe the core product is and validating that piece. So yes, you could go-you could call that going slower, validate the core concept, and then continue.
And this company, for example, called Osmo, is providing an interface where you describe a smell and it synthesizes the component that actually make that smell happen, it kind of creates a formulation for that compound and then sends that to a machine, similar process that you will have in any, you know, when you're adding fragrance to a detergent or a cologne or something like that, then you use that and then try to replicate it.
What is, how does AI take the workflow, the current work that we're all doing, and makes that faster, more efficient, cheaper, right? That means, for example, if in the past I needed a call center, and I needed a bunch of people answering the phone and answering questions, can I now train an AI to do that, right? And that is not necessarily your company. That's a piece of work that you do in your company that now AI is helping you do it back.
In the same way, using the surveys example, if you were a marketer and you did a big survey and now you get data from back and you needed to segment that, analyze it, do correlations or do some conjoint analysis or whatever, probably that took maybe weeks or weeks or maybe a week. Now you can probably get that done in an hour or in a couple of hours. So that's basically looking at when you look at your company and the work that you're doingEverything that takes up time or resources in the company It's worth thinking, is there a way to do this better, faster, cheaper, or do something that we were not able to do before? So that's one thing for the RBI, and that's more about the process, the operation, how you run it. The second part of the question is whether AI is a part of your product.
or AI fields report. So for example, at Roblox, one of the things that is important for us is that people can be engaged to join the sport. We are a user-generated content platform, so the more people that can create games, the better. We'll have more content on the platform, more content attracts more users, more users attracts more friends, and it's a flywheel that just keeps growing. And when you have a lot of users, developers want to build on your platform.
So for Roblox, part of building a better product is saying, how can I use AI to make it easier for a kid that is 13 to build a game from scratch? When a traditional game, when I was working at Xbox, could have required a studio of 100 people and 3 years. How can I actually change that to something that goes from 200 or 100 people and 3 years and a $20 million budget to something that a kid in Idaho, Bogota, or whatever, or El Cairo, and actually get to a computer, build it, and publish it. And that's where now you're using AI to make your apartment.
Yeah, look, I mean, it's interesting. At the beginning, the coding tools weren't that great, but they have become amazing. So there was a little bit of resistance to how am I gonna code and how much do I need to check? You can actually see it in, When a developer in the team is writing code, how much code gets written by the developer versus AI? At the beginning, it was 99% the developer, and I'm using the AI to check.
Now it's more like the AI is writing the code and the humans change. So there's a flip right now in that model. And I have a friend who works at another company and he was told, by the way, he's a college grad. He just graduated college as a computer engineer. He started in that company six months ago or whatever, started coding, and one day his boss came and said, "I don't want you to do it anymore. "I want you to use agents to build everything "that you're doing. "I want you to learn to do it with the agents." And your role is to what they call orchestrate with the agents to do the work, create the work plan, validate the work plan.
And look, one thing that I think I mentioned to Jonathan, Accountability still resides in the people. The CEO of a company cannot say, "You know, I gave the product to the AI, the AI did this, and that's what I shipped, and now the product doesn't work and my stock went down." The accountability is still going to be the CEO. The accountability is still going to be an engineer that had to deliver that feature, the product leader that built that feature.
What's been amazing is that the wait cycle between my idea and my feedback and seeing something on the screen is close to zero. That's insane. By the way, it used to take, I needed to make a phone call or write to someone, send them a spec, that person will call me for a meeting, ask me what do you want to do, and then they will tell me it's going to take 48 hours a week. And now that disappeared. But I'm still the one actually seeing the thing on the screen and deciding, is this good?
Very few people now. No, we have JP. Where's JP? He just hires, what would you call them, agentic engineers?
Yeah, yeah, agentic engineers. Agentic engineers.
Then you have the engineer, then you have the designer, then you have the product marketer, then you have blah, blah, blah, blah, blah. And the way I describe this is that The atomic unit of building a product ends up being a bunch of people with different specializations. And the more specialized you want people, the more people you will have in building a product. I mean, this was the way that I was having calls with our CEO, and especially when you're a founder, CEO, and you're very technical. When Roblox started, it was the CEO and his co-founder, two engineers building a product. They didn't have perfect managers, they didn't have data scientists, they didn't have designers, and so how they built something.
Was it great the first day? Probably not, but they built something, right? Eventually they added more engineers to the team. Then slowly they added the first product manager. So now to build a feature, you need a dependent linear spread analysis program. Then someone convinced them that they needed a designer. So now, the same work that they were doing initially with two engineers went from three engineers to ten people teaching different functions and then, oh, I need a data scientist, oh, I need a user researcher, whatever.
I think that a lot of these things are going to begin to collapse. What it means is that My expectation would be that people can do more of those functions as single person. So maybe you can at least as a product manager get to a clear, simple prototype of the thing that you want to do. Maybe you actually get to write the first line of code. Maybe you can analyze the data. The data scientists may be able to arrive to a conclusion and present it in some mocks and design that in the past would have required a designer.
That's what I think is going to happen. So you're going to get what they call builders these days, or where are you builders, or we're looking for builders. It's at the end of the day, it's a single person that can actually build something. on their own, right? And can actually have the agency and the drive to win. It doesn't fully replace the function. A designer that is a really good designer and has a keen eye and has amazing taste and can tell you whether, again, this color palette is the right one or not, that still has a ton of value, right? The marketer who knows how to go to market and put the product and write the right messaging and the right copy and the right ads or whatever, that's still a skill.
The fact that an engineer can actually build a video doesn't necessarily mean that they know that is the food advertising, right? So there's an element, I mean, people have been telling me, right, like the thing that's gonna last is taste. But taste to me is that degree of expertise that you bring and some discipline because You have an innate skill to be really good at that thing because you have been trained in that craft and you're really good at that craft.
Yeah, look, I think, as I said, the first thing that I think is going to happen is that the size of the team is probably going to be smaller.
That's to me very clear, right? So when you take 10 people, maybe you can do more with-Those three people, maybe one has a key product expertise, but it's a builder, like can do, it's a generalist, but with a product, kind of call it a major and a minor. They all can build products, but I don't have an expertise in product. Maybe there's a person that has the expertise, depending on what products they're building. Inside, maybe there's someone that has an expertise in engineering, right? Because I still need some of those. Uh...
So we outsource a lot of the coding of the same organization. Should it matter if we outsource to companies that use AI to code or not? Does it make a difference other than the price? Well, I don't think that the criteria-at the end of the day, if you're outsourcing the code, you as a customer are wanting a good outcome. That good outcome is the quality of the product that you're getting back. The velocity of what you're getting back, the cost of the.
My intuition is that the company that is using AI is going to give you all those things better than the one that is not. But at the end of the day, you have your criteria for accepting what the developer is bringing to you. And you set the price, you set the timelines, and you set all those things. So I think that it's intuitively that a company that is using AI is going to do these things a lot faster and potentially cheaper than the one that is not using it.
Can you describe where we are? The product velocity is real. Going from, I have a roadmap of 20 features that it was gonna take six months to build to maybe doing it in three months is real. Now, Jonathan's point, more volume doesn't necessarily mean better, but the reality is that you can build more faster. As I mentioned, the move from one step to the other in the process is real. It gets a lot faster to go from, I can now be very patient with the team from when we got together and we said we were gonna do this, and one day get it done.
to actually get all these things. And every year, by the way, companies that are doing well, every year they have to improve. Right now the reality is that now you need to make a trade off in head down and movements. And the budget became not just how many people you have, but how many people and how much are you investing in profits, because in some way, tokens are some fraction of a person, right? And so it's very cool to see, hey, you know, at Roblox we said, hey, by 2030 we're gonna be making this amount of money and we're gonna have this headcount.
How much do you rely on synthetic versus real? Look, I cannot speak of direct experience because all of those things are becoming very new right now. I don't see a reason why you're not tested. I think that everything that is replacing something, you need to test that synthetic audience against the real audience and see what's the delta that you're getting between those two and see if those things are integrated in a way thatMakes sense, right.
that you don't get the full picture, right? And then there's the things that consumer research itself, you do sometimes one-on-one interviews, sometimes you observe the person using the product, sometimes you ask them a survey. You now have three methodologies of gathering data that give you three different data points. I don't know if synthetic audio replaces all of that in one single shot. Especially when it's something novel, like the data in the past, you need to generate so it might be hallucination, like there's a lot of when you're testing a new feature, you don't know how they're gonna really interact with each other, the data doesn't interact with each other.
So there's actually research, some of them are former student and former colleague. created this massive data set where they essentially gave 2,000 battery questions, a lot of them from my pastAnd they created digital twins for those people. The digital twins don't do nearly as well as the hype going into digital twins suggested that you know. And I say that despite the fact that one of my colleagues raised $100 million to create these synthetic participants.
I would be extra careful because at least for the research now, it doesn't bear it out necessarily. I'm happy to share those. Do they know why? They don't know why, they know that it doesn't. You look at the correlations and we're actually pretty low. I'll share it. There's so much complexity. There are products that do well because of the social viewpoint of the product. There are products that do well because of what it's not.
It's potentially really hard to capture those things in There's also a problem at its core with common preference. But preferences are expressed along lots of different ways and in lots of different dimensions. And any decision that you make isSorry. of a set of expressions. Some of which are signals that you can I don't even know. Thank you. And so in order to be able to create the synthetic participant, you have to be able to really make that, have that whole thing.
And under the companies that we have to play. Yeah, so a couple of things. One of the things that we started doing at Rolls-Royce was, in the past it used to be a headcount. And then we really converted every $50. And the discussion got a lot easier when you converted $50. And just to give you guys a nuance, if I tell you 10 people, there's a difference in price if those 10 people are 10 vice presidents or 10 junior people, right?
Look, if I think about it in the context, for example, of. The goat of Arnhem ride So what are the parameters on a Fraud in UI? Doesn't matter. The engine, the game without an engine would not, right? If you felt that our engine was completely blown, We would go to battle for it. So I think that it's more where the core differentiation of your company and the core of your company is. You're a... I don't know.
If you're an insurance company, maybe the source code doesn't matter. You just need it to work for you and so forth. You have an amazing algorithm to validate how you underwrite a question.
It's new territory. I can tell you that I'm on the other side of some of those things. I mean, I could run away with the e-calls, but... New York Times and others have sued LLM's because they use all of their content without paying or without asking for permission. But it was uploaded by them. And they knew that this is public--Right, well, yes and no, but at the end of the day, there's a debate of what is public domain, what's not public domain.
And you will get that, by the way, Satya just wrote an article about, Satya, the CEO of Microsoft, about, hey, you still need to think about the ecosystem. You take that to the end. And then you don't need journalists. then will the AI actually capture what's happening today or will someone still need to report that? so that the AI can actually disseminate it, right? So I do think that there's something that you could debate of whether that, it was funny, right?
You are very comfortable using the work of journalists in a company. So this is still up in the air. The legal framework for this is being built by these cases. and what's gonna become the precedent in these rulings is gonna take a lot of our business.
For example, on the software side, you're saying you're speaking marketing stuff. It's for your product development cycle. Yeah. It's instantaneous. Yeah, I'm sorry. There are a lot of things that matter, right? So, I mean, for those of you who do marketing and think about the four Ps, I mean, distribution still matters, right? Like, I can build a game, you can build a game. Where do you publish it? How do you get your first 10 users? How do you get to a million users?
Do you build in monetization or the transactional technology or do you leverage the one that the platform has built? So I would think that there are things that are still, if I'm Procter& Gamble and I have a brand, maybe someone can build a detergent now faster than it used to be. They still need to promote that from market that they're searching, build a brand, get distribution. It might be cheaper and faster for people to attend them. But in those physical industries where distributions still matter, manufacturers still matter, shipping the product still matter, I mean, that's not going to get sold by beats.
I would ask that person the hard question, and if they don't know how to answer them, I would say that, you know, you need to learn about it and so on. Prioritization remains the same. I don't know that prioritization changes. I mean, if you have a capacity to do more, that doesn't mean that you have infinite capacity. So there's still going to be things that you cannot do. Either because the tokens are too expensive and you don't have the budget to spend it, or because you cannot ship 20 features because you're-how do you communicate in the market that you just launched 20 features and how are you going to validate?
you're still gonna have to make trade-off and decision decisions. And it goes back to the things that I was saying in the beginning. If you're the product leader and you have 20 features, what are the goals? What's the most important? What comes with the needle? How am I gonna validate those things? That's still the same. So, If someone comes to me and says, hey, now we can talk about the future, I may still ask, do we need those kind of pictures?
and the resources that are required for something to be produced. I have an intuition for what that is. I have an intuition for time. I have an intuition for how much time something takes, what that costs, how much a head count costs me. But my feeling is that tokens are completely abstract. Actually, I bet none of us in this room, even JP, who loves this stuff, has any idea when he enters the question, how much actually was the cost of getting that question answered or building that agent, how much was the cost of that agent?
You have to say no clue, even though yesterday you had that presentation, it said a million tokens. How much is a million tokens? Is that a lot or a little? We just don't know the denominator. On everything else, we know the denominator. So how do you like I remember there was the stat that Uber employees used up all of their year's token budget in one quarter. And I don't think it's because they were being wasteful.
When you're working on the Coding Edge and you're working on things that are new, You Estimate, guesstimate, you come up with some assumption, you can stake on the ground, and the best thing that you can do is hopefully use some measuring so that you understand, am I on a trajectory or not? So instead of waiting a year, yeah, you may say, hey, I'm going to I want to restrain and release a pound. I want to measure for a month.
We'll see how much we will be spending this amount of money. How many people were asking for more? You need to put those systems in place that didn't exist before.
And when you learn, and you're learning on budget, I don't know the answer. I don't know how to budget for it. today, but I would know how to create a process to learn how to do it. If you don't know how to do it, divide the process to work. How do you create a budget for something that you're not even aware of? And so you don't even know how much something is totally out of focus. How am I supposed to think about that?
Well, I mean, it starts with a first book thread, which is I know how much I can spend. Right?
Like I have a number at the end of the year that says I want to spend $10 million in product development. I have $5 or $7 million of that in people. So now I can only spend $3 million in token. Now I have $1 million. It's $3 million in token until I start hiring people. Right? That's a very good number. It's a number.
At the end of the day, it's a number that we're managing. And so then I could constrain the budget to 3 million, but then what happens is that my gives are going to come back to me and say, I'm exceeding the budget, I need more. Then I'm going to say, what are you getting out of those 3 million dollars? Are we getting more than $300,000 worth of value? Then maybe I off the budget. In the same way that when I was budgeting HeadGum and halfway through the year, the teams came back and said, like, you know, I told you that I needed 20.
This is definitely another question, something that's really bothering me this whole week is a lot of people have talked about whether it's faster, better, cheaper, whatever. There's a bunch of things that have to be learned in the implementation of this new workforce. The problem with the speed is it moving too fast for organizations to actually be able to internalize the learning. To internalize a learning, you have to have a realization, you have to observe it, and then you have to internalize it, and you have to interpret what it means, and then you have to implement it.
Peace. Right? And they now are now. Other people will be different and say, I want to be more rational about it. I want to set a budget. And that has to do a lot with company culture. I could see robots moving pretty fast and learning a thing or two. I could see maybe a Google being more methodical about how they work. Right? And I think that company culture will dictate how we move. And some people will learn-You know, because the bill came back and it was too high, or they ended up building a lot of AI slot and the product became really, really bad. All the food learned because of the food losses. But I think ultimately the good news is that the collective learning will happen.
It's happening right now. And six months from now, a year from now, we'll be talking about best practices.
I cannot sit down here today and talk about best practices. That would be completely... also for me to come back and say, I know what the answer is. This is just beginning. I mean, it's kind of like saying, You know, I mean, this transition is very unique, but anybody who did the transition first at Webb, then all of those have course corrections, right?
The first student at Wave had a massive course correction in 2021, in 2001.
You want 20 to the 4. We at Roblox have online 2022 tour investors and said, "From now on, we commit to you that we're going to grow 20% year-over-year and we're going to foresee a new future." Q1 2024. it was looking like 12%. We all win, we'll have some day. We started discussing what are the things that we can do to get this thing better than 4%. And we have 20 ideas on the board. it was easy to cut 10 of those 20 videos.
The other 10, it was not easy to cut. If I had all the luxury of time I would have said let's actually wheel it down so that we can find the top three. At that moment in time, the practical reaction was, "We want to be in these ten different things." And... To this day, if you ask me which one exactly worked, it would work. I don't know which one exactly worked. I know it may have been a compounded effect of doing the first one.
I learned actually that I could execute and build those things. I learned that the intuition was relatively right because it moved the needle. And it actually carried on for multiple quarters. But yes, the process wasn't perfect. In that scenario, We did a lot more than maybe we have to. It's hard to tell. Generally speaking, if you have the luxury, you want to build a better product. Now, the process to build an answer is going to be one of two.
Look, I think that the amazing thing that is happening is that And I got the idea because I haven't been there in a while. There's probably a lot of rebuilding to be done. And rebuilding, what I said is wrong, right? The infrastructure and energy, whatever data system, topology system, economy and imports, exports, how do you get people to invest in the country and all those things. So it's hard to pick one.
I mean, I think that what's exciting though is that It feels that many things could work. just because of how behind things are in so many of the core elements of building an economy and building a market, right? I mean, I haven't gone in detail to see, hey, you know, would Venezuela be amazing for data centers? Maybe. Land may be not that expensive these days, but then you don't have the energy infrastructure, so now you need to actually, or maybe use solar. But there might be some of these opportunities that are interesting.