Problem
Comparing experiments with different traffic mixes and calling the difference a clean creative result.
Solution
Root Cause / Diagnostic:
Comparing an experiment that received 70% Search traffic with a control video that received 80% Browse traffic introduces severe traffic mix bias. Search viewers possess high intent and seek specific answers, generating naturally higher CTRs and different retention patterns than casual Browse scrollers. Calling the metric difference a clean creative result confuses audience acquisition surface dynamics with content quality.
Actionable Fix:
1. Normalize experimental comparisons by filtering both test and control assets to a single identical traffic source (e.g., Browse Features only).
2. Inspect the Traffic Source breakdown in Advanced Analytics before comparing aggregate CTR, AVD, or engagement ratios.
3. Ensure experimental assets are engineered and packaged specifically for the primary distribution surface of the control asset.
Pro Tip:
Always compare experimental metrics within a single traffic source silo; an apparent retention victory is completely invalidated if the test video was carried by Search intent while the control faced cold Browse scrolling.
Comparing an experiment that received 70% Search traffic with a control video that received 80% Browse traffic introduces severe traffic mix bias. Search viewers possess high intent and seek specific answers, generating naturally higher CTRs and different retention patterns than casual Browse scrollers. Calling the metric difference a clean creative result confuses audience acquisition surface dynamics with content quality.
Actionable Fix:
1. Normalize experimental comparisons by filtering both test and control assets to a single identical traffic source (e.g., Browse Features only).
2. Inspect the Traffic Source breakdown in Advanced Analytics before comparing aggregate CTR, AVD, or engagement ratios.
3. Ensure experimental assets are engineered and packaged specifically for the primary distribution surface of the control asset.
Pro Tip:
Always compare experimental metrics within a single traffic source silo; an apparent retention victory is completely invalidated if the test video was carried by Search intent while the control faced cold Browse scrolling.