The one metric that matters
Ben Yoskovitz, author of Lean Analytics, talks about the one metric that matters.
Paul: You say we’re all liars, and a certain amount, particularly when you’re starting out on a new venture with a big vision and no backing, you need to lie to yourself maybe, about the complexity of what you’re about the face or the magnitude of what you’re about to face. It’s not all a bad thing, but we need to try and ensure that the data we’re receiving is as unbiased by our own personal reality distortion field as possible.
Ben: Absolutely. I believe we are all liars, but I think to a certain degree it’s a necessity. To get up in the morning and face the universe as an entrepreneur, you need to have a bit of a reality distortion field. You’re selling something to people that doesn’t exist yet. You’re selling a vision for something that maybe people don’t even realize they need yet. You do need to be able to lie to yourself, that that is a reality and that matters, and you’re the one that’s capable of doing it. Then you need to lie to other people in a way, because you’re pitching them on something that doesn’t exist.
The problem with many entrepreneurs, and it happened to me, is that we insulate ourselves so much that we believe in our own hype so much that we lose complete focus on what we’re actually trying to accomplish and we just run into a wall and we just crash. For me, analytics is not about blowing that reality distortion field completely away, it’s about poking holes in it and saying, “We get it. You’re going to lie to yourself; you’re going to lie to other people. Temper that with some data so that you can figure out how to get to proving that the lie was true.”
Paul: That’s a nice way of putting it, ‘proving that the lie was true.’ Now we’re a startup who acknowledges that we need to measure, we need to collect data, and we need to be informed by that data. The temptation is to jump straight in and create dashboards that have 100 different metrics on them. You talk about the one metric that matters as a way of disciplining yourself of focusing on the given problem space at any given time. I wonder if you could go into a bit more depth into that, and maybe explain a little bit about the concept and the advantages that that brings.
Ben: The way we structure the [inaudible: 02:57] book, and very quickly [inaudible: 03:00]. The metric that you track, or the metrics that you focus on, are dependent on the business that you’re in, you’re model, which is largely around how you make money, even though for some businesses, making money is much farther down the road, and the stage that you’re at, very simply. We break these stages, down but what stage you’re at in your business. If you take those two things, you should be able to find a single metric for that moment in time that matters to you, that you need to first of all measure, and then most probably improve on.
For us, the one metric that matters is a true thing, in the sense that I think you can actually just look at one number and move your business, and improve your business. It’s also similar to scoring [inaudible: 03:52] interviews, it’s a way of thinking; it’s a mindset around, “No, I shouldn’t have a dashboard with 1 million things. Maybe I can’t get down to one because I think that that’s too myopic, but I can get down to 3 or 4. Or maybe I get down to 4 or 5, but those all bubble up into one specific number that really matters.” Most numbers don’t live by themselves. Churn is tied to conversions, and lifetime value is tied to churn; these things are all related. We look at it as a concept of focus; be disciplined, simplify your life, and look at one thing based on the business you’re in and the stage you’re at.
You will probably be looking at other things; you will probably be tracking a whole bunch of things. If you can focus on one thing, and then the other key concept of the metrics that matters is you need to have a line in the sand. This is something we saw at Year-1 labs a lot, and you see it with a lot of companies. “Okay. I’m focused on …” I’ll use churn; “We’re focused on churn. I get it. I’m going to pick this one number. I’m going to look at churn, and I’m going to run experiments to improve churn. What should churn be? What should the value be? How do I know when I’ve gotten churn to the point where I can now go to the next step and focus on something else?” That’s really, really hard.
We make an attempt in the book to give you benchmarks, and we call them lines in the sand. It’s not concrete, because these lines can move, but it’s your best guess of where you think this number has to be in order for you to feel confident in going to the next step. How do you get the benchmarks? We share some in the book, but a book is not a great format for that because there is so much research out there all the time, and people are sharing all the time, that these numbers change. They are not universal and there’s a lot of variables. Again, you can see trends. You can see pretty clearly that in a B-to-B SaaS business, if your churn is 5%, early, early on maybe that’s okay. As you move along, 5% churn is too high.
Time and time again you talk to people; investors, entrepreneurs, business managers, 2%, 2 ½ churn in a B-to-B SaaS business, that’s kind of the number you’re looking for. That’s really high-level, it’s really broad, it doesn’t include all businesses and all types, but that to me, is so important. Pick one number, maybe some numbers bubble into it, pick a number for the business you’re in and the stage that you’re at, and then find a line in the sand; experiment your way to that line, and if you can’t get there, you have to really go back further and look at the fundamentals of the business. Why can’t I get churn from 5% to 2%? Maybe my value proposition is off. Maybe my pricing is off. Now you have ways of going back and rolling back the business to really look at the fundamental assumptions you made early on.
Paul: That’s great. Very helpful. We’re now kind of honing in on the fact that if you can focus on one metric, or maybe a couple of metrics that roll up into one metric at any given stage of your business, you’re going to be able to have the discipline to focus in on that; you’ve got the line in the sand. What constitutes a good metric? We have absolute numbers, and absolute numbers are great for vanity sometimes, and seeing a number always going up. What constitutes a workable metric that can be applied; general principles that can be applied in many situations?
Ben: I think that there are some basic things that make good metrics versus bad metrics. A ratio or a rate, very simply 2% churn versus some absolute number of churn is better; numbers that you can compare one another. Often this means doing things in a cohort way, one group versus another group. If I can compare those numbers easily, that helps. It has to be easy to understand. We say in the book you should be able to go to anybody, and without explaining your business, say, “Here’s the number I’m tracking,” and they’re going to understand what you’re doing and why you’re doing it. Those are some of the things, for me, that tell you that you have a good number or a bad number. Number that always go up to the right, more often than not, is a vanity number. A percentage typically doesn’t just go up and to the right, that doesn’t make sense, that’s already better a ratio or a rate; numbers that you can compare and are easy.
We also talk in the book about different types of metrics; for example, a leading indicator versus a lagging indicator. Very simply, a lagging indicator, churn is a good example of that. By the time you’ve measured it, the people have already left. That’s what churn is. You can’t get those people back, probably. You’ve discovered a problem too late. You can try to fix it, and then re-measure it and hopefully improve it, but a leading indicator is much more powerful because it helps you predict the future. There’s some basic math around these numbers that’s important for what’s a good metric versus what’s not, and then there are some characteristics of numbers that also matter.