Cashbe is a cash back service operating in CIS and Russia. Its’ mission is to provide the best cashback experience and an extra benefit for online shoppers.

I held a CPO role, and this case study focuses on how we improved the return on marketing investment by 32% in less then 60 days.

Task

My objective was to make the product profitable within six months. To achieve that, I decomposed the objective into several sub-goals:

1. Increase conversion rates of new signups through the product funnel
2. Improve long-term retention and LTV
3. Increase number of new signups

Action

First, I have focused on issues one and two with a structured customer-centric approach. Next, I tackled the last problem by establishing and scaling a non-standard acquisition channel. 

This case study focuses on step 1: increasing conversion rates through the funnel. 

To address the issue, I have executed a 4-step plan:

1. Defined main stages of the funnel
2. Constructed hypotheses about barriers to conversion
3. Validated them with surveys, in-depth interviews, and user tests and gathered insights
4. Hypothesized about potential solutions and tested them with users

I will discuss each of the stages in more detail below.

Defining main stages of the funnel

The funnel of the service was composed of the following steps:

Steps 3 and 5 were optional, but the data showed they had a strong positive correlation with customer LTV.

Validating hypotheses 

To test the assumptions, I have used an exit survey, email surveys, and in-depth customer interviews.

It was crucial to understand when to approach users with a question: we had to be sure the user was not on the verge of converting to have meaningful results. I have analyzed «time-to-conversion» data for each step, determining when the majority of those who would convert to the next step has already made it:

Website Exit Survey

The survey was shown on website exit to users who did not signup. The goal was to determine whether the user had had any interest in the service; if yes, what prevented him or her from signing up.

Unfortunately, it turned out most of the sample had no real interest in the service, and their answers provided no value. I had to switch to offline live product test to gain insights about perceived website trustworthiness.

Email surveys

Next, I have built simple surveys to be emailed.

I was hesitant if surveying non-converting customers would be something we would need to do on a regular basis. To move fast and not to build a potentially unnecessary solution, I’ve sent these surveys myself via simple PHP wrapper for Sendgrid API on a daily basis for a week.

In-depth interviews

To understand why users who ordered with us at least once stopped using the service, I invited some of them for a phone interview, using the following approach:

• First, we identified the time, after which 90% of the users who would make a second order have had already placed it (55 days after first order) – it was our initial sample
• Next, I’ve asked the users whether they have had made an order without cashback during this timeframe.
• Third, I invited them to participate in an interview in exchange for a gift card

Later, I have drafted the main hypotheses to be tested during the interview and the scenario to address them.

Sample discovery questions:

1. Tell me about a time you made your last online purchase?
2. Tell me about how your experience has been so far trying to get cash back on your online purchases?
3. Tell me about the last time you have used Cashbe website or used an extension. What was your goal?
4. Have you ever been in a situation when you have made an order with cashback, but it did not appear on your profile? How did you proceed with the problem?

Next steps

I have iterated in a cycle “ask questions – get answers – capture insights” until I could predict possible responses. As a result, we now had a list of validated problems for each conversion step:

1. Not signed up: Does not understand how cashback works; Does not trust the website
2. Email not confirmed: A confirmation email’s subject is not appealing; A confirmation email is too long [not validated] *
3. First cashback not activated: The user does not understand the mechanics of cashback
4. Browser extension not installed: The user does not comprehend why he needs a browser extension
5. First purchase not made: The user has purchased with our service, but the purchase did not appear on his account (missing cashback)
6. Repeat purchase not made within 90 days after sign up: The user forgot about the service; The amount of cashback received was different from what the user expected; The user had bad missing cash back experience

* Note: I have deliberately skipped validating this problem as it was easier to implement the solution right away.

Hypothesizing and testing potential solutions

Next, for each problem, we have built a list of possible features to solve it. To understand which features we should build first, I have ranked them by a composite indicator maximizing ROI of development efforts:

“Revenue Impact * Our Confidence in Impact / Development Complexity”

Here, Revenue Impact is measured in percentage points; Confidence is a percentage (0% – not confident at all, 100% – very confident); Development complexity is the number of story points assigned to the task.  

Managing Missing Cash back

Our priority was an issue of missing cashback. We can track and reimburse most of these missing cashback cases if the users communicate us the problem.

The problem was the users did not understand we could track missing cashback in these cases. After the order did not appear on their Cashbe account, they became disappointed with the service and never came back.

Our goal was to inform users adequately that we could track missing cash back. To achieve that, we:

• Improved communication in onboarding and trigger emails campaigns
• Introduced a new email campaign «On the way to cash back» sent to the users after first cashback activation if no order was made within 1 hour to explain next steps if cashback is missing
• Added an ability to track missing cash back from anywhere on the website

All these features combined gave us an increase in the topic-related tickets by 30%, meaning we could capture more problems and increase customer satisfaction. 

Addressing “Repeat purchase not made within 90 days after sign up”

The user forgot about the service

As a primary hypothesis, I suggested that if we transition from capturing purchase intent to evoking it, we could become a single starting point for online shopping and would increase long-term retention.

To address that, I designed a weekly email campaign with the best deals of the week at selected stores. It was a curated email with 8-10 best coupon offers. We have collected and sent it in a manual mode for a month to track results. Unfortunately, the campaign did not produce any significant results – we decided that it happened due to the lack of personalization, and approached the problem from the other side.

We re-designed it as a campaign with double cash back stores of the week, featuring stores with x2-x4 cash back of the week tailored to user’s preferences. This action immediately gave us a boost of 5% in repeat orders and led to an increase in long-term retention.

Educating new users about cashback

To address the issues related to poor understanding of cash back mechanics, we improved our onboarding process. I decided to tear current onboarding down and to rebuild it from scratch as less than 5% of users were completing it. 

I designed a 5-step process as a simple popup to see if we are moving in the right direction before investing in a more native solution. It explained:

1. What cashback is and how to start a cashback trip
2. What will happen after the order, and what to do if the cashback is missing
3. How to withdraw funds
4. Why the user might need Cashbe browser extension
5. That we had a help center. It also provided a link to a sample article explaining service mechanics

One of the challenges I faced was explaining the team who had invested a significant amount of time in the previous version of onboarding the need for a new one. I had to seat with each team member one by one and explain the benefits of the new solution, as well as drawbacks of the previous one based on the data collected.

After UX prototype testing had indicated no significant issues, we have rolled the new feature out in an A/B test. As a result of the test, the cohort with the new onboarding process has shown a 32% increase in 3-month LTV. See the results of the test below:

• Email confirmation CR: no change
• Extension Installed CR: + 36%
• Cashback Activated CR: no change
• Order Placed CR: +13%
• LTV-90: +32%

The process: Onboarding

Next steps

What we have completed here was one iteration of the famous “Build-Measure-Learn” feedback loop, and it brought significant results. We have proceeded with the approach, testing more than 70 different hypothesis in a 6 weeks span. Unfortunately, these efforts were not enough to make the product profitable with a full-time development team on board, so we had to lay off part of the team.

Results

• Extension installation rate has increased by 36%
• Long-term retention improved by 5%
• LTV-90 and company revenue from new users has increased by 32%

Key takeaways

• I am yet to discover an approach to website exit surveys that provide meaningful results
• The structured, iterative approach allowed us to improve business economics quickly, maximizing ROI of development efforts.
• Focusing on low-hanging fruits gave us some fast results, but we needed something much more powerful to sustain the business in the long-run. In the aftermath, I believe it was essential to invest in radically changing the product. Reaching a new market that could potentially yield exponential growth (rather than fine-tuning the business with an average revenue) may have saved the development team.
• More, it was crucial to involve the development team in customer development process rather than doing it almost exclusively by myself. Not being aligned has led to the team burning out, and the following development speed was much lower