Problem
Crunchyroll didn't have any real way of testing out ideas on the fly, and so the company wanted a tool that could be used to do just that. Having an A/B testing tool for the various new features that are being developed for Crunchyroll 2.0 was a must-have for the future success of Crunchyroll. Features like newly created paid tiers for US and International, as well as testing marketing placement for e-commerce, Crunchyroll Expo, as well as other more in depth experiments were all being requested by various teams, and therefore needed to be part of Config Delta.
Overview
Config Delta is a robust A/B testing tool developed for the many different facets of the Crunchyroll platform. Its main focus is to help facilitate the retention of users through various funnels, such as supporting marketing tests, audience development needs, as well as business development requests. Through Config Delta, users are able to create in depth experiments that they are then able to test on free, paid, or anonymous users, on all of the different Crunchyroll platforms.
Role
Being a more technical rounded designer, I was asked to help design Config Delta in what was a very quick turnaround. In around four months time, we went from kickoff to first release, ready for the introduction of our new Crunchyroll paid tier lists.
Constraints
The two biggest constraints that I had when designing this tool were time, and resources. The turnaround time from concept to release was around four months, to be exact. With that in mind, I had to hit the ground running, and start wireframing up flows as well as being in constant communication with Product Managers. I had two great engineers that I worked with, and so resources for this project were limited to me making sure that I stayed within a framework that both engineers could understand and work within. That meant that design work fell in-line within the Material-UI framework.
Process
We started off with an initial kickoff where stakeholders gave a quick overview of the project. Over the next couple of weeks, I sat down with PMs, and we went and created quick whiteboard user flows so that I would have a better understanding of the complexities of Config Delta.
/w=3840,quality=80)
/w=3840,quality=80)
After figuring out and identifying pain points within the user journeys, I went ahead and started to create wireframes of those journeys. Once again, there was a healthy amount of back and forth communication with the devs and stakeholders involved in this project, so I never felt like the project was going over my head. Through an iterative process, I was able to hone in on the most important features of Config Delta, and create wireframes that satisfied both stakeholders, PMs and engineers.
One of the most important features required for Config Delta was the ability to create experiments using logical operators. This would allow users from different teams within the company to create A/B tests that were either very broad in their reach, or extremely focused and targeted specific Crunchyroll users. For example, if Marketing wanted to create an experiment on the Crunchyroll homepage that targeted 50% of registered, tier 1 subscribers, and they wanted to see if the new CTA, promoting the new Ultimate Fan pack was clicked through, that was all possible within Config Delta.
/w=3840,quality=80)
Outcome
When it came time for deliverables, I provided engineers with mockups of the various features for Config Delta, separating them into their specific functions. Throughout my design process of Config Delta, being able to sit down with stakeholders and engineers, and iron out the functionality was critical to the success of delivering this app on time. Having Config Delta ready for the global rollout of Crunchyroll+ felt like a real accomplishment, and I was happy to see real quantitative results due to the use of this app and the global rollout of Crunchyroll+
Below are some video demonstrations of Config Delta in action, as well as notes on what's happening.
Campaign Acquisition Funnel (single tier)
As a marketing team member, I would be able to create a campaign using Config Delta portal.
- I can create a start and end date, and I'm also able to define the targeted users.
- You can see the changes being applied within the video, we've targeted 100% of the US users, and we would like to have a custom URL for our campaign.
- We will use a default variant weight, click create, and let's check the new custom campaign URL. As the user lands on the campaign URL, they can see a landing page, as we have not customized the landing page yet, it's displaying our default configuration of images and text.
Customized Landing Page
- Through Config Delta, we are able to customize copy, images, CTAs, as well as many other nuanced things.
- In the video above, you can see how we’re configuring all the different translation keys and inserting a custom image URL. When a user lands on this new customized campaign URL, they will see a completely different landing page!
- You can see that the images and copy have been changed in this version.
Complete Page
A complete page is the last page in our user acquisition funnel. Users land there after going through the checkout process.
- We have a default version of the complete page; it has messages about different series, news, manga, etc.
- As a marketing team member, I would be able to change each individual part of the complete page for, let us say, a new marketing campaign we want to promote.
- Therefore, I can change all of that information, and display new promotional material, and talk about various exciting things. In the demo video above, we're showing off Crunchyroll Expo banner, that when clicked, takes them to that particular page.
- Config Delta allows our marketing department to do in depth experiments on whatever they want!
A/B Testing
As a marketing team member, I would like to have multiple variants of a campaign, to understand which one has a better conversion.
- So within this existing campaign, I can add a new variant. I want to customize it, having a 50/50 split weight.
- Some users will see variant A, while others see the customized variant B.
- Being able to do a variety of different A/B tests will allow teams to create in depth experiments on all of our Crunchyroll users.
