Payout Structure Analysis
The aim of this project was to ascertain if new payout structures had any significant effect on applicant behavior.
Objective
This endeavor was driven by a notable financial inefficiency I uncovered during an extensive examination of our creator payment records in relation to prevailing industry benchmarks. A comprehensive market study further disclosed that our organization had been exceeding appropriate compensation levels for nano and micro creators (1,000 - 50,000 followers) by an estimated margin of 25%.With micro and nano creators constituting approximately 75% of our creator base, this overpayment presented a significant financial burden and potential loss for the company.
The primary challenge we faced was determining whether these creators would have perceived a 25% reduction in their starting compensation and how this cut would have impacted their behavior. We needed to understand whether these adjustments would have been noticeable to the creators and whether the decreased compensation would have influenced their productivity, engagement, or willingness to collaborate.
Problem
Our team devised an experiment wherein two groups of creators, a treatment and control groups, would be invited to apply to an ad campaign. Starting compensation for the treatment group would be 25% less than control group. To ensure our results could be evaluated properly, we conducted a comprehensive group stratification to establish our treatment and control populations, followed by several chi-squared tests to ensure our populations were as balanced as possible on the target variables.
Solution
Fig 1. Contingency Table for one of the several target variables tested, creator Race/Ethnicity.
With the treatment and control groups established, testing could begin with each group invited to apply to the respective campaigns. Upon conclusion, a dataset of all applicants, totaling around 2,200 individuals, was prepped for analysis. The logistic regression model was selected for this analysis as it is well-suited for modeling binary outcomes, and in this context, to to assess the impact of a price cut in the treatment group compared to the control group in terms of who applied to the campaign. The dependent variable, "Group," represents the group membership (treatment or control), and the aim was to determine if the price cut influenced the likelihood of individuals applying to the campaign. By analyzing the coefficients of the independent variables, we can determine whether the price cut had a significant impact on the likelihood of individuals applying, while controlling for other factors such as race/ethnicity and different payment tiers. Read the code here!
Fig 2. The logistic regression model did not yield statistically significant effects for any of the variables, including race/ethnicity, payment tiers, and historical campaign participation. The model's ability to differentiate between the two groups, as indicated by the area under the ROC curve (AUC) of 0.53, was limited.
Overall, the results suggest that the price cut did not have a significant impact on group membership, and other unconsidered factors may play a more substantial role in determining who applies to the campaign. These results provide reassurance that the price cut did not introduce any unintended consequences or biases in the creator's decision-making process, supporting the stability and consistency of creator behavior in response to pricing changes. Consequently, we anticipate that a potential reduction of 25% in campaign expenditure can be achieved without significant consequence, enabling the redistribution of these resources across the entirety of our organization.