Playbooks — How we increased signup to KAV by 6.8%
Our Onboarding engineering team is dedicated to ensuring that every new user’s journey, from the initial signup to the full use of Klaviyo’s features, is seamless and intuitive. We believe that every user deserves an onboarding experience that is personalized and educational. This post is the story of a project we did early last year to shrink a gap in our onboarding experience. I talk about what we changed and how we measured the impact.
When we looked at how users onboarded, we recognized a difference between self-serve customers and our larger customers. The latter often had help from dedicated onboarding specialists, while the former navigated Klaviyo’s features on their own. The need for an enhanced, personalized, and structured onboarding experience for our self-serve segment became apparent. It was not just about providing information but doing so in a manner that was structured, sequential, and aligned with each user’s specific needs, technical environment, business type, and growth aspirations. The Playbooks feature was envisioned as a solution to this identified need.
To start with, we needed to decide how we would measure the success of the new experience. Our team’s north star metric is achieving KAV or Klaviyo Attributed Value. This metric refers to new customers who achieve revenue to their business attributed to Klaviyo. For example, not only do they set a flow live, but a message coming from that flow results in a customer of our customer making a purchase.
In order to gather data effectively and be able to confidently make judgments about the impact of our intended changes, we launched Playbooks as an A/B experiment with KAV as our primary metric.
The old experience
The previous onboarding experience relied on users exploring Flows, Campaigns, etc. on their own, or, clicking on “tasks” listed on the “Objectives” tab of our dashboard, as shown in the screenshot above. The tasks listed in the Dashboard Objectives tab were disconnected from each other and did not have a single clear order that the tasks should be completed in. Additionally, through customer conversations, we discovered that most users did not see the Dashboard Objectives tab as the personalized step-by-step setup guide it was intended to be, but as an “endless feed of information” or “series of unrelated suggestions.” Representative feedback:
“Still confused how best to use this esp to improve my store performance — would love to have the step-by-step guide accessible so I can use now that I understand what I want to do next.”
“It’s overwhelming — step by step, easy to follow instructions are NOT that easy. So easy to miss a setup.”
Despite this, a quarter of new users did click on tasks in the Objectives tab, and those users were much more likely to publish a key object (see below).
The new experience
Our goal with Playbooks was to transform the onboarding experience from a generic, one-size-fits-all model to a more personalized one. Playbooks is engineered to offer users a clear, logical path through the essential setup tasks and feature exploration. Every user is greeted with a personalized “Get Started” page, which provides a clear, sequential guide through the essential setup tasks. Each task and step is not just informational but is aligned with the user’s specific business type, needs, and growth targets.
In order to set users up for success, some tasks are gated until certain foundational steps like integrating with an e-commerce provider are completed. In the old onboarding experience, we found that users would often jump into the most exciting features first, without laying the foundation that would allow them to get the most out of those features. Logically ordering and gating tasks allowed us to gently nudge users into completing the most important tasks first.
At Klaviyo, we really care about the data. It’s important to get accurate numbers so we can understand the impact new features are having on our customers. Many teams have an embedded product analyst. In order to get these analysts the data they need, we often launch new features as A/B experiments. A/B experimentation is a method in which two versions of something (like a webpage, app feature, or advertisement) are compared to see which one performs better. Imagine having two different designs for a website’s homepage. With A/B testing, half of the visitors would see Design A, and the other half would see Design B. The performance of each design is measured by how well it achieves specific goals, such as more clicks, sales, or sign-ups. The goal of the test is to determine which version of the feature achieves your goals more effectively. My team almost exclusively launches new features in this way.
During the Playbooks experiment, we observed a number of statistically significant shifts, including an increase to the primary metric mentioned above. Here’s what we saw:
Signup -> Engage: +28% (relative change)
Do new users engage with our onboarding guidance? In the old onboarding experience, this metric represented users that clicked tasks on the Objectives tab. In Playbooks, this metric represents users that click tasks within Playbooks. The increase shows enhanced initial user interest. We believe users found the personalized, streamlined tasks offered by Playbooks more accessible and engaging.
Signup -> Create: -4.0% (relative change)
Create means creating a key object (e.g. Flows, Campaigns, and Signup Forms) but not necessarily publishing it. Here, we saw a decrease. Initially this drop was puzzling. However, when contextualized with other metrics, especially the increase in signup -> publish (below), this may show that users better understood the tasks and those who did them were more intentional.
Signup -> Publish: +3.3% (relative change)
Here we saw new users engaging more actively by publishing key objects. We were excited about this. After all, if you sign up for Klaviyo, but don’t actually, for example, get a flow live or a campaign out there, it’s hard to get value.
Signup -> KAV: +6.8% (relative change)
This is the metric we were trying to move! New signups achieving KAV (their first revenue attributable to Klaviyo) is our team’s north star metric. Traditionally, achieving KAV has been hard to increase and so we were very excited about this shift. As a result, after the experiment ended, we decided to adopt our new Playbook feature for all users.
Signup -> Paying: +5.8% (relative change)
Paying means the new user becomes a paying customer of Klaviyo. Always nice to see, and a sign that the user found tangible value and that turned into a financial commitment.
Conclusion
While the engineering and user experience facets of Playbooks are compelling, the metrics offer a quantifiable testament to its success. The metrics are evidence of enhanced engagement and increased value realization. (We wish all experiments went this way!) By addressing the disjointed and unfocused steps users previously encountered, we designed an onboarding journey that is clear, simple, and aligned with user intent. Stay tuned for a follow-up post on how we engineered this feature from a technical perspective.
