Case Study — Product Data Scientist Portfolio
Product teams need to model revenue impact and rollout risk before launching experiments. Poor experiment design leads to false positives and revenue leakage.
Built an interactive simulator that takes traffic, conversion, uplift, and ARPU as inputs and outputs expected revenue lift, confidence intervals, statistical significance, and a GO/ITERATE/KILL recommendation.
Web-based engine with sliders for traffic, conversion %, uplift %, ARPU, duration, and segment. Real-time visualization of confidence band and risk meter.
Ship when significance ≥95%, false-positive risk <5%, and uplift ≥5%. Otherwise iterate or kill.
| Time Horizon | Milestone | Success Criteria |
|---|---|---|
| 0–30 days | Baseline established | Pre-experiment conversion/ARPU measured; traffic logs validated |
| 30–60 days | Experiment run | Minimum sample size reached; no SRM detected |
| 60–90 days | Rollout decision | Statistical significance achieved; revenue impact quantified |
Success: Revenue lift >5% with p<0.05; false-positive rate <5%; no degradation in core retention.
Tripwire (anti-success): Sample ratio mismatch (SRM); cross-contamination between control/variant; external market events skewing results.
| Event Name | Required Properties | Notes |
|---|---|---|
| experiment_impression | experiment_id, variant, user_id | Fired on assignment |
| experiment_conversion | experiment_id, variant, user_id, revenue | Conversion with revenue |
| experiment_session | experiment_id, variant, session_id, duration_sec | Engagement proxy |
Staging: Raw event logs ingested into stg_experiment_events. Deduplication and timestamp normalization.
Marts: fct_experiment_results — experiment_id, variant, conversions, revenue, sample_size. fct_experiment_statistics — lift, p_value, confidence_interval.
Tests: Uniqueness on (experiment_id, user_id, variant); not_null on revenue for conversion events; referential integrity to experiment config.
Documentation: dbt docs; column-level descriptions; lineage to upstream sources.