Advice to Self — Lessons Learned Building Portfolio Reporting
I joined Klaviyo right out of undergrad in 2019. In retrospect, the projects I tackled in my first 2+ years were straightforward, not because they didn’t have technical challenges, but because a) I only needed to touch code that belonged to my team, and b) the features I worked on were incremental enough that my colleagues in product could get the details right before we started building.
In this blog post I want to talk about the lessons I learned from portfolio reporting, my first project where neither of those two things were true!
First…some background. Since Klaviyo was founded, our data model was centered on individual companies. Everything was tied together at the company level; all of our reporting views for campaigns, flows, profiles and more were at the individual company level or below.
As Klaviyo has grown over the years, we’ve seen an expansion from entrepreneur and small business users to larger mid-market and enterprise customers. These users are often responsible for a collection of companies (corresponding to multiple brands, regions, subsidiaries, etc.) and want to see how they perform in aggregate. This identified a gap where a user would have to manually switch into several different accounts, take screenshots and/or export data, and repeat on a regular schedule.
This gap motivated a new data model at Klaviyo, the portfolio. A portfolio is a collection of linked accounts that allows for aggregate reporting and management. It allows for a single-pane view into data across multiple accounts, summarizing key metrics like attributed revenue, deliverability data, and channel performance. This new data model will save our customers a huge amount of time spent manually exporting data from individual accounts, allow them to more quickly assess the performance of their portfolio, and overall make Klaviyo easier to use.
I started on the portfolio project in early 2023, and last week it went into general availability. The project spanned multiple teams over an extended period and I learned a lot! If I could go back in time to the start of the project, here’s the advice I’d give myself:
- For big new features, get feedback early and often.
- For long-running projects, write everything down.
- As soon as work crosses team boundaries, establish new lines of communication.
For big new features, get feedback early and often.
This seems obvious in hindsight, but this was a hard lesson to learn. This is where iterative development is your friend. Show core user stories being accomplished through demos to your team, to people outside your team, and to customers, and expect there to be changes to specifications and implementation based on the feedback.
When you’re building a feature to the product specification, you can develop a habit of having blinders on for how customers might actually interact with your feature. This is why it’s good to demo early and often. Don’t forget that your team can develop collective blinders, which is why it’s useful to demo outside of your team including to customers.
Here’s a prime example. We thought things were in pretty good shape in the summer of 2023, getting ready for a demo to our VP Product. This was the first time she was seeing the feature since it was presented at the product specification meeting several months prior. She pointed out that in our report, we were showing accounts with different currencies (e.g. euro, US dollar) all plotted on the same graph. For example, say we had a portfolio with a US dollar account and a Japanese yen account. We would show bars for say campaign-attributable revenue from both accounts on the same graph with the X-axis being units of currency. This would be confusing, strange, and difficult to interpret, especially since the ratio of US dollar to yen is ~1:150!
Of course this made no sense. We just hadn’t noticed due to being focused on other technical challenges and not having multiple currencies in our dev environments. How did we solve it? We added a currency conversion feature, where we take all of the values for the linked accounts in a portfolio and convert them to a single user-selected currency.
There were many other examples from the beta period. During beta (limited availability), we’re able to solicit early customer feedback and tackle any high-priority bugs prior to a wider release (general availability). I was able to hop on several customer calls with our product managers to be a “fly on the wall” while a customer used the report. We quickly noticed things that we needed to prioritize before the GA launch. These were mostly ease-of-use items, such as adding additional filter options and persisting filter selections as the user navigates through their various portfolio pages. They might seem small, but they made the customer experience that much more delightful, which was a huge win for us. And it was definitely motivating to see the problems happen with my own eyes vs only hearing about them later from a product manager.
Specs are a starting point. For the features I worked on earlier in my time at Klaviyo, I could expect that the finished feature would be very close to the original spec. But for more involved features, you should expect the spec to evolve just as code evolves. It’s good to keep in mind that your job isn’t to implement the spec; it’s to create great software.
For long-running projects, write everything down.
Don’t assume a static team of people with perfect memories. Write absolutely everything down. It’s important to maintain a source of truth for why things are the way they are.
Projects may start as investigations, go on the shelf, and later come off the shelf. People switch teams, priorities change, new people join the company. Projects get blocked due to dependent technology needing to get created first. That’s all completely normal even for organizations much smaller than Klaviyo. This was my first time being the single person who saw a project from brainstorm to ship. It made me a convert for documentation. I learned that if you assume from the start that you’re writing not only for your current stakeholders but also for new people who will need to get up to speed, you’ll do a better job.
A sample of the documents we and other teams wrote:
- A vision document for portfolio reporting with patterns and principles.
- A detailed product specification for the V1 portfolio reporting feature.
- A report on a proof-of-concept we built in early 2023 that showed what was possible with our code as it existed back then and what we’d need to develop to support the feature.
- Backend and frontend RFCs.
- An RFC on scaling.
- An RFC on enhancements to our Computed Metric Service Engine.
- An RFC on portfolio metrics and metric mapping for accounts.
- Reports on running performance benchmarks.
Having all of these documents available to all stakeholders proved to be crucial. It allowed us to have everyone on the same page for what we were going to deliver, and how we were going to do it. We referenced them as we were building, and, especially for new people on the team, after the fact to understand why we made certain tradeoffs, how the data model worked, etc. Being responsible for explaining how we’d tackle shipping the project in a way that was maintainable and scalable for the future was a great way to improve my technical writing skills.
Another tip: Writing documents and going into discussions about those documents with a notion of “strong opinions, loosely held” is the healthiest way to end up with a good design which everyone is bought into. It demonstrates that there’s already been some investigation into why things are a certain way, and allows for hypotheses to be worked out async. Then, during discussion, people are already informed, and the goal is to align on a path forward. (It’s part of our culture at Klaviyo. Look at any RFC and you’ll see lots and lots of comments and issues that get resolved ahead of time leaving the interesting stuff for live discussion.)
As soon as work crosses team boundaries, establish new lines of communication.
Develop a practice of “looking around corners” when dealing with cross-team development. This along with having the system in your head will help solidify how your changes affect other parts of the platform.
Part of the fun of working at a company is collaborating with your team and you get pretty good at it — you learn how to communicate, you’re usually looking at the same information (e.g. sprint board, code, slack channels relevant to the team) which makes it easy to stay on the same page. In short you get into a state of team flow. But once other teams are involved things get more complicated. You need to do more looking around corners.
Here’s a practical example. We needed to work in code owned by other teams. We use CODEOWNERS files in our various repos to map files to their owning teams for PR approval. We were driving hard to get to beta in advance of a call scheduled with our first beta customer. We knew we would need to make changes in files owned by other teams, including by our internationalization team. That team is based in London (a different timezone) and had a bank holiday coming up at a time where we anticipated making late-breaking changes. We couldn’t afford to lose a day or more blocked on approvals. A month into joining Klaviyo I wouldn’t have seen around that corner, but now it was second nature. We coordinated with the London-based team and they volunteered to have someone checking in during the holiday to do PR reviews.
Another example is that for us to demo the feature, and for others within Klaviyo to play with the feature, we needed a set of linked demo accounts with demo data, simulating a larger company with a portfolio of brands. We could have assigned someone on our team to create these accounts, but we also knew that other teams working on other (non-reporting) features within portfolio would have a similar need. So instead of doing the “easy” thing and taking care of this ourselves, we worked across teams to create a set of demo linked accounts used by everyone. This proved to be crucial, as it allowed us to see the full picture with all the portfolio features on a single set of accounts and catch several game-breaking bugs prior to the beta period.
Conclusion
The journey to shipping portfolio reporting was long and winding. It’s been over a year since I started on the project, and I’ve learned a lot from it and grown as an engineer. It taught me the importance of getting feedback early and often, writing great technical documentation, and emphasizing communication when working with people outside your own team. The project was a huge effort across multiple teams in engineering, product, and design. From a quick proof-of-concept early last year to shipping the production feature last week, I’m so excited to see it in customers hands!
