START is one of the leading online streaming platforms in Russia. It uses data to develop business, personalise user experiences, and optimise content strategies. We spoke with the company’s Chief Data Officer, Dmitriy Zolotukhin, to find out how the data function is structured, what methodologies are applied, and how AI and ML impact the future of subscription-based services.
What role does data play in START’s business, and how do you set priorities when working with it?
At START, data is crucial for business growth, especially for expanding the subscriber base. The detailed user data (viewing habits, engagement metrics, trial behaviours) allows us to build models that optimise discount offers.
At START, the main directions for Data analytics are:
- Product Analytics: a team focused on uncovering insights to improve the user experience.
- Marketing Analytics: optimising traffic, identifying profitable channels, and highlighting unprofitable ones.
- Content Analytics: providing producers with insights on content quality and viewership and evaluating the effectiveness of content acquisition contracts.
- Machine Learning Teams: delivering customer data products, such as recommendations and uplift modelling.
Due to the high user acquisition cost, our priorities are stability and interpretability.
What are your key metrics and business goals for the data team?
Key metrics and business goals for a data team align with measurable business impact. Of course, the main objective is to increase revenue and conversions.
Elaborating on the metrics and the speed and reliability of experimentation (how quickly new models are tested and deployed) is vital. They show how the business adapts to changing user behaviour, reducing the risk of deploying unproven strategies. We also consider effective segmentation and maximising data usage. These efforts help us estimate other outcomes – churn reduction and customer lifetime value.
Why did you choose uplift modelling, and what problems does it help solve?
We chose uplift modelling because it addresses the main challenge – identifying users who will convert only if targeted. The models estimate the incremental impact of an action (offering a discount, for example). This helps us to exclude users who would convert without an offer, improving the business’s ROI. At START, user acquisition is costly, but retention is critical. Uplift modelling also meets our need for data-driven subscriber growth strategies.
Why is it important not to offer a discount to users who are already willing to buy?
Building on the previous answer, it is important not to offer discounts to users who are already willing to subscribe, since doing so reduces revenue. The uplift modelling allows us to protect full-price revenue where possible. Hence, the efficient use of discounts positively impacts START’s profitability.
Which metrics and approaches were the most useful — Qini, stratification?
Both of them are useful. The Qini coefficient helps us accurately evaluate uplift models by measuring the actual incremental effect of targeted offers and specifically considering users whose behaviour changes incrementally due to the intervention. Therefore, the metric helps validate whether our models are impactful. On the other hand, stratification (especially registration date and active days) improves model stability by providing more accurate validation results.
Where did Thompson’s Sampling prove useful, or on the contrary, where did it fail?
Thompson Sampling proved beneficial in selecting data subsets most likely to improve model performance by focusing on those with the highest potential impact during training. At START, we used an algorithm to guide sample selection from stratified clusters, updating the training set only when the model’s Qini score on both training and validation remained stable. However, Thompson Sampling needed to be combined with a data split for consistency, as it often failed to converge reliably due to variability in external conditions and model sensitivity.
What must be taken into account when running A/B tests in subscription-based services?
A/B testing must be well-designed to provide valid results. In this case, user segmentation is essential, as randomisation should reflect the main differences – registration date or engagement level, to avoid bias.
Furthermore, control groups must remain untouched to measure accurate uplift. The test should run long enough to capture delayed conversions and potential churn, not focusing only on immediate results.
Finally, it is crucial to prevent data leakage by ensuring users are not exposed to mixed treatments (for instance, initially seeing no discount and later receiving 10%).
Which metrics do you consider the most indicative for evaluating the effectiveness of data projects in an online streaming service?
Besides conversion uplift and Qini coefficient, we consider retention by cohort. These help quantify the real impact of the interventions, such as content recommendations. We also track subscription recovery after churn and trial conversion behaviour because they show how well our models influence user activity.
What competencies do you rely on when building a data team? Which specialists are especially in short supply today?
When building a data team, we look for specialists who develop models, understand business processes and can transform data into measurable insights; the principal roles include data scientists, ML engineers, and data and product analysts. However, there is a need for experts in experimental design, akin to A/B testing and uplift modelling, and data engineers working with event-based data processing.
What methodologies or algorithms would you like to use in the future?
In the future, we are looking to apply methodologies that improve personalisation and long-term user value. For example, that could be deep learning for content recommendation across modalities (viewing, search, ratings) or learning for dynamic offer strategies, where the systems adapt actions based on user activity in real time. Finally, we want to use automated model monitoring and ML ops practices to provide model reliability.
What role do you see for LLMs and generative AI in subscription services or online streaming platforms?
The main potential of LLMs in streaming services, as I see it, lies in flexible content search and choice. By considering a user’s past preferences and allowing them to describe in their own words the type of content they are looking for (or the emotions they want to experience), we can offer a whole new level of content recommendation.
This requires strong technical capabilities on the service side and the ability to communicate and “sell” this new way of finding content to the user.
How do you see the evolution of the CDO role and the data function in digital business?
The primary role of the CDO is the storage, management, and interpretation of data. The volume of data is growing exponentially year over year, which naturally increases the workload and responsibilities of this department.
Another challenge for the CDO is spreading LLM and AI solutions on the market. Their effectiveness also needs to be appropriately analysed and assessed by the data team. I would not be surprised if, in the future, LLMs will assist analysts in evaluating the performance of other LLMs.