How do you grow? In recent years this has been a difficult question for paysites.
I’d like to propose an answer.
If you want to grow, you need to build an organization that is ruthlessly, tenaciously in search of the truth. Your organization also has to be very efficient about finding out what is true through testing.
Test ideas quickly. That’s it.
We all have lots of ideas that we hope will make our companies grow. The only way to know if they will move the needle is to test.
In a recent article in Harvard Business Review, Microsoft’s head of testing (and former head of testing at Amazon) shared his experience with testing new ideas.
80 percent of them don’t work. Yep, 8 out of 10 ideas don’t move the needle for Microsoft or Amazon. Most of your ideas won’t help your company grow either. 80 percent sounds like a terrible ratio.
It’s also known as the Pareto principle. 80 percent of effects come from 20 percent of causes. The 80/20 rule is everywhere. Here are some examples: 20 percent of customers produce 80 percent of sales, the richest 20 percent have 80 percent of the money, etc.
It’s a fact of life. It’s a fact about your business, too. What can you do about it?
The only logical thing to do is test lots of ideas. And test them very quickly.
If you want to grow, you need to build an organization that is ruthlessly, tenaciously in search of the truth. Your organization also has to be very efficient about finding out what is true through testing.
Here’s the problem. The way a normal paysite tests ideas is very expensive. Consider a typical A/B test. Let’s say you are going for a 15 percent improvement on some key metric. To find the 15 percent you need to send half of your traffic to whatever new experience you think will make a difference. This is your challenger. It is trying to beat the status quo. Let’s call it option A.
You send the other half to the control group. These people don’t see the new experience you are testing. Let’s call that one option B.
The longer you run the test, the more traffic you send to the loser. That loser could be A or B. It doesn’t matter. If there is a real difference then it can cost you a lot of time and money to find it.
We recently did an A/B split test with a client.
It was Vendo versus another biller. We were A, they we were B. It took a while to work out the kinks of the test.
By the time it was done and everyone was satisfied with the results the company had sent over a million dollars’ worth of traffic to each biller. There was a double-digit difference between the two billers.
Each visitor that the client sent to the other biller (B) made the client over 10 percent less money. They lost over $100,000! What an expensive test!
Paysite companies have many, many ideas for growth. Not every industry has such a wealth of ideas. How do you choose between all your options? How do you prioritize? It’s the question that every paysite company is asking.
Let’s flip that problem on its head for a second. What if the problem isn’t that you have too many ideas and not enough resources to pursue them? What if your problem is — as it is at Microsoft and Amazon — that 80 percent of your ideas aren’t going to help your company grow?
I’d like to propose that you work on your ability to test new ideas faster before moving on to your priority list. If you really believe that 80 percent of your ideas won’t matter then why spend time prioritizing them? Don’t even do them. Chances are they’re a waste of time.
The first thing to do is build an organization that can use tools and processes to test quickly and cheaply. The focus becomes less about the ideas for growth and more about becoming proficient at using tools and processes to know which of the ideas matter.
In my experience the best tool available for testing new ideas is artificial intelligence. When Google’s AlphaGo beat Lee Sedol at Go last year the AI had learned from more than 30 million Go moves.
An average game of Go is 300 moves. It also takes about an hour. AlphaGo had 300,000 hours of game play. An average person spends 92,000 hours working during their life (assuming 40 hours per week from age 18 to 67). Lee Sedol was only 33 when he was beaten. He had been recognized as a Go prodigy and started young. He worked long hours. Let’s be generous and say that he had 30,000 hours of Go playing time.
AlphaGo had learned from 10 times as many moves as Lee Sedol. The advantage the AI had over the human was that AlphaGo had tested its ideas 10 times more, had learned from 10 times more mistakes.
We recently launched a new version of Aria (our AI) for risk. How do you win at risk as a biller? You let through the largest number of transactions with the smallest number of chargebacks below a threshold defined by the payment method. In some cases that is 1.5 percent, in other cases it is 1 percent.
How do you know if someone is likely to chargeback? It won’t surprise you to hear me say that we don’t, and 80 percent of our ideas about whether a transaction is risky won’t make a difference. Most of our ideas won’t help us get more good transactions and fewer bad ones.
There is another problem with testing ideas about risk. Chargebacks are a lagging indicator. Because they take some time to come in (some in the first 30 days, more in the next 30 days and the balance over two years) you don’t immediately know whether your ideas made a difference.
It is very hard for humans to identify patterns in systems with delayed effects. It’s just not how our brains work. We move on to other topics. We forget the details. Time causes havoc for humans.
Another challenge is the huge number and variety of factors that influence a chargeback. Will your end user’s girlfriend look at his bank statement and will he lie and call his bank? This is very hard to predict. But there are patterns among variables that a machine can learn from.
If an AI can evaluate all previous chargebacks and look at time of day, location, site, characteristics of the site, cross sales purchased and dozens of other variables it can detect fraud patterns that humans can’t see. And it can test many, many ideas about how to reduce risk.
Only AI can effectively test lots of ideas and learn from them quickly. It is a testing and learning machine. Intelligence isn’t defined by having the best ideas. Intelligent people have lots of ideas and sort through the bad ones more efficiently than other people.
Walter Isaacson has just released a new biography of Leonardo da Vinci, who is considered one of the greatest geniuses that humanity has ever produced.
As part of his research Isaacson pored over Leonardo’s notebooks. There are a lot of them. Over 7,000 pages survived. And inside these notebooks you find … a lot of crap. Leonardo lived 500 years ago.
Even today, using the latest technology, we can’t make many of his “inventions” work. Why not? Because this invention won’t fly, that one won’t roll and this other one won’t actually scale a castle wall and help the invading army end the siege. The ideas just aren’t good.
Why did such a genius have so many bad ideas? Because ideas come in fives and only one of them is good. It’s the same for Leonardo as it is for you and me and everyone at our companies.
The best thing you can do to help your company is to set up the right tools and processes to test — faster, better — in pursuit of the 20 percent of ideas that actually move the needle and will lead to growth.
Mitch Platt is co-founder of Vendo Services, which uses artificial intelligence to power its billing platform that allows merchants to continuously improve and grow their businesses.
