Role Playbook · 2026

ML Engineer

Feature stores with provenance you can defend. Trace every feature to source tables and catch schema drift before AUC drops.

22 min read Runbooks Metadata only Apache 2.0

Executive summary

Register feature tables and training pipelines in Metroflow. Alert on upstream schema changes. Document feature DAGs for compliance in one click.

Features look stable until an upstream column changes overnight. Without lineage, you debug in notebooks for days. Metroflow traces features from raw ingest to production models.

2hDebug vs 2 days without lineage
100%Production features with documented lineage
EarlySchema drift alerts before silent AUC drop
01

What broken lineage costs you

Typical patterns for ML Engineer teams. Ranges, not guarantees.

Overnight schema change

Silent AUC degradation for days

With Metroflow: Alert when upstream columns change.

Compliance review

Stale diagrams and notebook exports

With Metroflow: Live feature DAG export from graph.

Undocumented feature sources

Cannot explain model behavior

With Metroflow: Trace churn_features → dim_customers → raw_hubspot.

Training-serving skew

Different SQL in batch vs online

With Metroflow: One lineage path for both pipelines.

◇

Metadata only. Metroflow crawls schemas, job names, manifests, and dashboard definitions. Your production data rows never leave your network.

02

Your stack, one graph

Metroflow sits above the data path, not inside it. One searchable map for your entire role.

Trace churn_features → dim_customers → raw_hubspot. When orders_v2 changes, see every training job that consumes it.

03

Where are you today?

Most teams land at L1 or L2. Target L4 in 90 days.

L1Siloed docsWikis and spreadsheets disconnected from production.
L2Partial lineageSome tool lineage exists. Cross-layer gaps remain.
L3Unified graphOrchestration, warehouse, and BI in one map.
L4Certified metricsKPIs owned, enforced, and traced end-to-end.
L5Proactive opsPre-merge gates. Stale assets caught early.

Quick self-check

  • Answer "what breaks if X fails?" in under 15 minutes?
  • Cross-functional teams share one definition of core KPIs?
  • Incidents include downstream dashboard impact without Slack archaeology?
  • Changes include cross-layer impact checks before merge?
  • Named owners for certified metrics and critical pipelines?

0–2: Start Week 1 connect · 3–4: Certify metrics · 5: Add change gates

04

Choose your path

Every org is different. Pick the track closest to your context.

Warehouse-native features

Features built in dbt/SQL, batch training

Priority: Register first: top 5 production model feature sets

Feature store (Feast/Tecton)

Online + offline feature serving

Priority: Register first: link store entities to warehouse sources

Real-time inference

Stream features + batch backfill

Priority: Register first: stream-to-feature lineage

05

Who owns what

Assign decision rights up front. Metric fights are governance problems.

RoleOwnsOn Metroflow
ML engineerFeatures, training, deploymentOwn feature DAG documentation. Schema drift monitors.
Analytics engineerUpstream dbt modelsNotify ML on breaking schema changes.
Data governanceModel risk, complianceExport lineage for reviews.
Data platformPipelines, computeLink training jobs to orchestration metadata.
06

30 · 60 · 90 day rollout

A program with gates, not just a connector checklist.

Days 1–30

Map features

  • Register top production models
  • Trace features to warehouse sources
  • First schema drift alert
Gate: One model with end-to-end feature lineage
Days 31–60

Guardrails

  • Alerts on upstream column changes
  • Compliance export template
  • Training job registry
Gate: 100% prod models with lineage docs
Days 61–90

Operationalize

  • Pre-merge ML impact for AE PRs
  • Monthly model risk review
  • Automated feature DAG for auditors
Gate: Schema drift caught before AUC impact
07

Incident runbook

When churn_model_v3 is in trouble. Follow this timeline.

T+0 · Detect
AUC drop in monitoring

Production metric below threshold. Check recent schema changes.

T+15 min · Trace
Feature lineage
"Which production models consume columns from orders_v2?"
T+30 min · Root cause
Upstream change

Column renamed in staging. Training-serving skew confirmed.

T+2h · Fix
Retrain or patch

Coordinate with AE on schema fix. Retrain with updated features.

T+24h · Verify
AUC recovery + doc

Update feature DAG. Post-mortem with lineage proof.

Slack template

[ML] churn_model_v3 AUC drop: investigating Suspected: schema change on orders_v2 (column device_type renamed) Impact: 3 training jobs + online feature store entity customer_churn Owner: @ml-oncall · Retrain ETA 6h Metroflow feature lineage: [paste link]
08

Metric certification pack

Copy into your governance doc. One definition. One owner. Full lineage.

Feature freshness (SLA)

Track first
Formula
Max lag between source table update and feature store refresh.
Threshold
Alert if > 24h for tier-1 features.
Owner
ML Eng + Platform

Feature lineage coverage

Track second
Formula
% of production features with documented path to raw source.
Target
100% by day 60.
Owner
ML Eng + Governance
09

Daily workflows

Four situations you will hit every week.

🔬

New model launch

  1. Document features

    Register in graph.

  2. Compliance export

    DAG for model risk.

  3. Drift monitors

    Upstream column alerts.

📉

AUC drop investigation

  1. Check schema changes

    Last 7 days upstream.

  2. Trace features

    Training vs serving paths.

  3. Retrain plan

    Coordinate with AE.

🔀

Before AE schema PR

  1. ML impact query

    Affected features and models.

  2. Sign-off

    Approve or request migration.

  3. Update docs

    Feature DAG refresh.

📋

Quarterly model review

  1. Lineage audit

    100% coverage check.

  2. Stale features

    Deprecate unused.

  3. Export for auditors

    One-click report.

10

Copy-paste queries

Company Brain or lineage search. Context included.

Schema drift
"Which production models consume columns from orders_v2?"
Feature trace
"Trace churn_features to raw_hubspot and every training job."
Compliance
"Export feature DAG for churn_model_v3 for model risk review."
Upstream change
"What changed upstream of fct_customer_features in the last 7 days?"
Training jobs
"List all training pipelines that depend on dim_customers."
Serving skew
"Compare lineage path for batch vs online churn features."
11

Glossary

Plain English. "Why it matters" tells you when to care.

Feature store
Central registry for ML features online and offline.
Why: Must link back to warehouse sources.
Schema drift
Upstream column changes that break features.
Why: #1 silent AUC killer.
Training-serving skew
Different logic in batch vs real-time features.
Why: Lineage exposes mismatched paths.
Feature DAG
Directed graph from raw data to model input.
Why: Compliance teams require this.
Model risk
Governance review of production ML systems.
Why: Exports from Metroflow, not notebooks.
Company Brain
Plain-English feature and lineage queries.
Why: Faster than manual notebook tracing.
12

Outcomes checklist

Measure if the program is working.

Success metric90-day target
Time to debug feature / schema issue< 2 hours
Production features with lineage100%
Schema drift caught before AUC impact≥ 90% of cases
Compliance export timeMinutes vs days

Ready to put this playbook to work?

Week 1: connect your stack and run your first blast-radius query. Week 4: certify your first KPI. Week 8: operationalize the runbook.

Try the live demo Self-host Talk to us