Case Study — Product Data Scientist Portfolio
Optimization systems (e.g., fraud detection, risk scoring) face a tradeoff: aggressive thresholds reduce loss but increase user friction; conservative thresholds protect experience but increase operational risk.
Built a simulator that models precision vs recall as a function of sensitivity and risk tolerance. Outputs live risk probability, alert trigger state, and estimated loss avoided.
Interactive controls for sensitivity and risk tolerance. Real-time precision-recall curve visualization and dollar impact of loss prevention.
Calibrate thresholds so alert triggers at risk probability ≥60% while keeping false-positive burden acceptable.
| Time Horizon | Milestone | Success Criteria |
|---|---|---|
| 0–30 days | Baseline model in place | Historical precision/recall measured; threshold options documented |
| 30–60 days | A/B test on thresholds | Friction metrics (block rate, appeal rate) tracked by variant |
| 60–90 days | Optimal threshold rollout | Loss avoided > baseline; friction within SLA |
Success: Reduction in operational loss; precision > target; recall adequate for risk coverage; appeal rate below threshold.
Tripwire (anti-success): Spike in customer complaints; increase in manual review queue; model drift detected via monitoring.
| Event Name | Required Properties | Notes |
|---|---|---|
| risk_score_computed | entity_id, score, model_version | Every scoring invocation |
| alert_triggered | entity_id, score, threshold_used | When score exceeds threshold |
| alert_outcome | entity_id, outcome, loss_prevented | True/false positive resolution |
Staging: Raw scoring events in stg_risk_events. Normalize timestamps and entity identifiers.
Marts: fct_risk_scores — entity_id, score, threshold, outcome. agg_precision_recall — model_version, precision, recall, date.
Tests: Uniqueness on (entity_id, timestamp); not_null on score and model_version; valid score range.
Documentation: dbt docs; schema for downstream monitoring dashboards.