Core concepts

How the pieces fit together

Metroflow is not a passive inventory of tables. Connectors pull metadata only (manifests, DAG definitions, warehouse schemas, BI explores) into a crawler that normalizes assets and relationships. The result is a semantic graph stored in Metroflow's database and exposed through the workspace UI, REST API, and agent runtime.

Semantic graph

The semantic graph is Metroflow's central data model. Nodes represent assets: warehouse tables and columns, dbt models and sources, Airflow tasks and DAGs, Kafka topics, Looker explores, glossary terms, and certified metrics. Edges represent typed relationships: depends_on, materializes, owned_by, tagged_with, defines_metric, and others.

Unlike a flat catalog spreadsheet, the graph preserves context across layers. A single metric node might link upstream to the dbt model that computes it, the Snowflake view it reads, the Airflow DAG that refreshes it, and the dashboard tile that displays it. Traversing two hops answers questions that traditionally require Slack threads and SQL archaeology.

The graph API (REST reference) lets you build internal tools on the same foundation agents use. Filter by owner, PII tag, or freshness; export subgraphs for compliance reviews; or embed lineage widgets in your own portals.

Metadata crawl

A crawl is one connector sync cycle. The crawler authenticates with your source system, fetches the latest metadata snapshot, maps it to Metroflow's canonical schema, and computes a diff against the previous version. Added, changed, and removed assets propagate to the graph; stale nodes are marked or pruned according to your retention policy.

Crawls are scheduled (hourly, daily, or on webhook) or triggered manually from the workspace or API. Large estates sync incrementally: dbt manifest hashes skip full re-parses when nothing changed; Snowflake crawls scope to altered tables when the warehouse exposes change tracking.

Connector credentials are encrypted at rest. In self-hosted deployments, secrets never leave your VPC. See Connectors for per-tool setup and Deployment for secret management patterns.

Lineage

Lineage in Metroflow is derived from connector metadata, not inferred guesses. dbt ref() and source() relationships become edges; Airflow task dependencies link orchestration to warehouse loads; BI tools contribute explore-to-table mappings where exposed.

The workspace lineage graph supports column-level detail when sources provide it (dbt column lineage, warehouse foreign keys). Use blast-radius mode before renames or deprecations: select a node and preview downstream dashboards, exports, and models that would break.

Lineage is environment-aware. Production and staging graphs can be toggled in one click so engineers validate changes against the graph their users actually depend on.

Certified metrics

Certified metrics are first-class graph nodes, not footnotes in a wiki. Each metric carries a canonical definition, owner, approval status, upstream SQL or dbt model, and reconciliation history. When finance and product both reference "MRR," Metroflow shows whether they point at the same computation or divergent forks.

Metrics link to glossary terms and governance tags (PII, revenue-recognized, exec-facing). Stewards certify definitions through the metrics registry; changes require approval and leave an audit trail. Company Brain uses certified definitions when answering business questions, reducing the "which dashboard is right?" problem.

Reconciliation workflows compare metric values across sources where connectors expose aggregates or where you attach validation queries. Discrepancies surface on the metric card with links to the offending pipelines.

Company Brain

Company Brain is Metroflow's infrastructure-aware AI layer. Before generating an answer, Brain retrieves relevant graph nodes (models, metrics, recent deploys, sync status) and grounds the response in that context. Answers include citations: clickable links to catalog entries, lineage subgraphs, and metric definitions.

Brain bridges technical and business language. A question like "Why did NRR drop last week?" might traverse certified metric definitions, check Airflow run history for delayed DAGs, and summarize findings in plain English with evidence, not invented table names.

Brain memory is scoped to your workspace and respects RBAC. Users see only assets their role allows; sensitive columns excluded from crawls never enter retrieval. See AI & agents for query examples and safety details.

Agents

Specialized agents perform focused tasks on top of the graph: draft documentation from lineage gaps, propose dbt tests for untested columns, map Airflow failures to upstream root causes, and generate lineage summaries for PRs. Agents propose outputs; humans approve. No silent writes to production systems.

All agents share the same metadata grounding as Company Brain. The document agent reads model descriptions and upstream sources; the debug agent correlates task logs with graph topology; the lineage agent exports subgraphs for compliance packets. Agent queries are also available via REST API.

Configure LLM provider and model through environment variables at deploy time (Deployment → Environment variables). Metroflow supports OpenAI-compatible endpoints, Anthropic, and local models for air-gapped installs.

Glossary

Quick reference for terms you'll see across the docs and workspace.

How Metroflow differs from traditional catalogs

Traditional data catalogs excel at inventory and search but often lag behind production, treat business context as an afterthought, and bolt on AI without stack awareness. Metroflow is designed around three differences:

Live graph, not static inventory

Catalogs populated by manual tagging or quarterly scans go stale the day after a deploy. Metroflow's incremental crawls keep lineage and ownership aligned with what actually runs. Editors and agents autocomplete from the latest sync, not a wiki snapshot.

Technical + business on one layer

Many catalogs separate "data dictionary" from "lineage tool" from "metrics layer." Metroflow merges them in the semantic graph so a metric, its SQL, its Airflow schedule, and its dashboard consumers share one node neighborhood.

Grounded AI, not generic chat

Generic LLM assistants hallucinate table names. Metroflow agents retrieve graph context first, cite every claim, and respect RBAC. Specialized agents produce PR-ready artifacts instead of free-form prose you'll rewrite anyway.