There are many products in Mango, some consider only large ones and it turns out 12. It so happened that for 20 years Mango services have been developing on the telephony platform, and many of them are still available only within the personal account page.
In the context of the task, our hypotheses revolve around different visual codes: for a large and small B2B client. By convention, an iPhone user has more money than a push-button phone user and is also accustomed to a better, more minimalistic and functional design. Our hypothesis is that customers will be richer among conversions under the new design than among conversions under the old design.
Actually we'd been running tests for around a year with different versions and minor hypotheses, what you see here is the final result.
It would be possible to follow the popular way to measure the conversion rate. In our case we go from general to specific, so we also track all side metrics, but to decompose the final results only. In this case, I would prefer to measure the money directly. Namely, the payment of the first month and the repeated payment. These indicators correlate very well with LTV, and therefore directly affect the company's ebit. In addition to increasing the margin, in proportion to ARPPU, the average payment does not increase the fixed costs as well. Which further raises the income, even with same revenue.
Additionally, it is worth noting that the conversion rate and other side metrics may not increase, but even decrease, based on our hypothesis about richer B2B customers. Therefore, let's add the measurements of the total income from equal traffic for each design version.
Thanks to the collaboration of the site team, CRM and product marketing, we were able to implement one important mechanism. Now we know which version of the design the client saw before particular deal.
|Ver||Deals||Avrg paym||Total||Minus fixed costs|
If only customers or conversions are counted within this model, then the first version might be concidered as more profitable. But if you count the money, then the second option is far ahead. With a small difference in the primary figures, you can increase revenue from new customers by 10% with one visual design. And fewer customers equals less overhead, which adds profitability. Yet we ignored overheads in calculations, because they're proportional to revenue.
In addition to the money from the sale, customers also have such important characteristics as: churn, upselling, and much more. Our team will enrich the measurements on them in the following tests for precise improvements, but for now, let's focus on the above model.
Someone might ask: why don't you measure LTV? Answer: we are not even worried about LTV itself, but its proportional increase. The fact is that we have a very predictable smooth outflow graph, so the difference between clients in the first months pretty accurately reflects the difference in LTV. And in B2B products, there is a very long cohort life cycle, such tests would take many years.