Role Playbook · 2026

Data Platform Engineer

Debug DAG failures with full stack context. Link orchestration to dbt models and BI dashboards in one on-call view.

22 min read Runbooks Metadata only Apache 2.0

Executive summary

Register every DAG, pool, and warehouse connection in Metroflow. Trace failures to stale dashboards in minutes, not hours.

Platform engineers are on-call for Airflow, Snowflake, and the 2am pages: but logs alone do not tell you which exec dashboards will go stale. Metroflow links orchestration metadata to downstream impact.

60%Fewer escalations to analytics engineers
<2hMean time to resolve with full context
1Pane for DAG → model → dashboard
01

What broken lineage costs you

Typical patterns for Data Platform Engineer teams. Ranges, not guarantees.

DAG retry at 2am

2–4 hours across five browser tabs

With Metroflow: Agent reads logs + links to downstream models and dashboards.

Snowflake rate limit

Blind retries waste slots

With Metroflow: See which dashboards go stale by 8am if throttle continues.

AE pulled into every page

Context lives in one person's head

With Metroflow: Self-serve blast-radius for on-call.

Pool slot contention

Cascading DAG delays

With Metroflow: Dependency graph shows critical path and affected owners.

◇

Metadata only. Metroflow crawls schemas, job names, manifests, and dashboard definitions. Your production data rows never leave your network.

02

Your stack, one graph

Metroflow sits above the data path, not inside it. One searchable map for your entire role.

When extract_charges retries, trace: Airflow task → staging tables → dbt models → Looker explores. Platform fixes with full downstream picture.

03

Where are you today?

Most teams land at L1 or L2. Target L4 in 90 days.

L1Siloed docsWikis and spreadsheets disconnected from production.
L2Partial lineageSome tool lineage exists. Cross-layer gaps remain.
L3Unified graphOrchestration, warehouse, and BI in one map.
L4Certified metricsKPIs owned, enforced, and traced end-to-end.
L5Proactive opsPre-merge gates. Stale assets caught early.

Quick self-check

  • Answer "what breaks if X fails?" in under 15 minutes?
  • Cross-functional teams share one definition of core KPIs?
  • Incidents include downstream dashboard impact without Slack archaeology?
  • Changes include cross-layer impact checks before merge?
  • Named owners for certified metrics and critical pipelines?

0–2: Start Week 1 connect · 3–4: Certify metrics · 5: Add change gates

04

Choose your path

Every org is different. Pick the track closest to your context.

Airflow-centric stack

Orchestration: Airflow · Warehouse: Snowflake/BigQuery

Priority: Register first: revenue and identity DAGs

dbt Cloud + managed warehouse

Hybrid orchestration across tools

Priority: Register first: cross-tool job metadata

Multi-region / multi-tenant

Separate pools per business unit

Priority: Register first: tenant isolation + blast-radius per BU

05

Who owns what

Assign decision rights up front. Metric fights are governance problems.

RoleOwnsOn Metroflow
Data platformAirflow, pools, warehouse opsOwn DAG registration. Blast-radius on failure. Comms template.
Analytics engineerdbt models downstream of DAGsLinked in graph. Not pulled into every page.
BI leadDashboard freshnessNotified when explores will stale.
SRE / infraCapacity, rate limitsPool recommendations from dependency critical path.
06

30 · 60 · 90 day rollout

A program with gates, not just a connector checklist.

Days 1–30

Connect orchestration

  • Register top 20 DAGs in Metroflow
  • Link DAGs to dbt models and tables
  • First on-call blast-radius query
Gate: One DAG traced to exec dashboard
Days 31–60

On-call runbooks

  • 3am template in PagerDuty/Slack
  • Agent-assisted log + lineage context
  • Weekly stale-dashboard report
Gate: MTTR under 2 hours on pilot DAGs
Days 61–90

Proactive ops

  • Pre-deploy impact for infra changes
  • Pool slot optimization from graph
  • Zero-AE-escalation target on tier-1 DAGs
Gate: 60% fewer AE escalations
07

Incident runbook

When extract_charges is in trouble. Follow this timeline.

T+0 · Detect
DAG retry alert

Airflow retry 3/5 or Snowflake rate limit. Open Metroflow.

T+5 min · Blast radius
Run impact query
"Why is extract_charges retrying and what dashboards are affected?"
T+15 min · Communicate
Notify stakeholders

Tag BI and finance with affected dashboard list.

T+30 min · Fix
Apply fix

Scale pool, backoff, or fix upstream. Agent cites rate-limit context.

T+60 min · Verify
Confirm freshness

Dashboards updating. Close incident with lineage proof.

Slack template

[INCIDENT] extract_charges retrying (3/5) Root: Snowflake warehouse rate limit · pool analytics_prod saturated Impact: staging.charges stale · fct_revenue delayed Affected: Revenue Dashboard, Finance daily close Owner: @platform-oncall · ETA 45m Metroflow blast-radius: [paste link]
08

Metric certification pack

Copy into your governance doc. One definition. One owner. Full lineage.

DAG success rate (SLA)

Track first
Formula
% of scheduled DAG runs completed within SLA window.
Scope
Tier-1 DAGs: revenue, identity, product events.
Owner
Data platform + SRE

Dashboard freshness lag

Track second
Formula
Max minutes between last successful DAG run and dashboard refresh.
Threshold
Alert if > 120 min on certified dashboards.
Owner
Platform + BI lead
09

Daily workflows

Four situations you will hit every week.

🚨

2am on-call page

  1. Open graph

    Search DAG or task name.

  2. Blast radius

    List stale dashboards.

  3. Comms + fix

    Slack template, then remediate.

📊

Weekly platform review

  1. Tier-1 SLA

    Any DAGs near breach?

  2. Pool utilization

    Contention on critical path?

  3. Stale report

    Dashboards that missed SLA.

🔧

Before infra change

  1. Impact query

    Warehouse resize or pool change.

  2. Notify owners

    Downstream DAG and BI contacts.

  3. Rollback plan

    Document in change ticket.

📥

New DAG onboarding

  1. Register metadata

    Tasks, schedules, pools.

  2. Link downstream

    Models and explores.

  3. Add to on-call runbook

    Blast-radius query saved.

10

Copy-paste queries

Company Brain or lineage search. Context included.

On-call
"Why is extract_charges retrying and what dashboards are affected?"
Capacity
"If Snowflake throttles tonight, which dashboards go stale by 8am?"
DAG debug
"What dbt models and tables does daily_revenue depend on?"
Critical path
"Which tier-1 DAGs feed the board revenue dashboard?"
Pool planning
"What runs in analytics_prod pool between 2am and 6am?"
Post-incident
"List all explores that were stale during last night's outage."
11

Glossary

Plain English. "Why it matters" tells you when to care.

DAG
Directed acyclic graph of scheduled data tasks.
Why: Your primary on-call object in Metroflow.
Pool slot
Concurrency limit for Airflow task execution.
Why: Contention causes cascading delays.
Blast radius
Downstream assets affected by a failure.
Why: Turns log grep into actionable comms.
Metadata crawl
Metroflow reads job names and schemas, not data rows.
Why: Safe to run in prod.
Freshness lag
Time between successful run and consumer update.
Why: What BI feels when you are still fixing.
Company Brain
AI with cited answers from your metadata graph.
Why: On-call queries in plain English.
12

Outcomes checklist

Measure if the program is working.

Success metric90-day target
Escalations to analytics engineers60% reduction
Mean time to resolve (tier-1 DAGs)< 2 hours
Dashboards stale without proactive commsZero
DAGs with downstream lineage linked100% tier-1

Ready to put this playbook to work?

Week 1: connect your stack and run your first blast-radius query. Week 4: certify your first KPI. Week 8: operationalize the runbook.

Try the live demo Self-host Talk to us