Connectors

How connectors work

Each connector is a read-only integration with a specific system. On sync, the crawler authenticates, fetches the latest metadata representation (manifest JSON, DAG bag, INFORMATION_SCHEMA rows, schema registry subjects), maps fields to Metroflow's canonical asset types, and writes nodes and edges to the graph database.

dbt

The dbt connector ingests project manifests: models, sources, tests, exposures, metrics, and column lineage when enabled in dbt 1.6+. Supports dbt Core (artifact upload or S3/GCS path), dbt Cloud (API token), and CI artifact webhooks.

1. Generate or locate artifacts

   Run dbt docs generate in CI to produce manifest.json and catalog.json. Metroflow needs at minimum the manifest; catalog enriches column types and stats.

2. Create connector in workspace

   Settings → Connectors → Add → dbt. Choose Core (path), Cloud (account + token), or Webhook (push on deploy).

3. Configure path or credentials

   For Core, point at an S3/GCS prefix or local mount the crawler can read. For Cloud, supply service token with Metadata Only scope.

4. Map environments

   Tag prod vs staging manifests so lineage toggles match your deploy branches (main → production, develop → staging).

5. Run first sync

   Confirm model count matches dbt project size. Open any model card and verify ref() lineage renders.

Webhook mode posts artifacts on merge to main. Configure your CI job to POST the manifest to /api/v1/connectors/{id}/ingest with a connector-scoped token.

Airflow

The Airflow connector reads DAG metadata, task dependencies, schedules, and owners via the stable REST API (Airflow 2.x). Optional log metadata helps the debug agent correlate failures without storing raw log bodies.

1. Enable REST API

   Confirm [api] auth_backend allows your Metroflow service account. For RBAC-enabled Airflow, create a user with Viewer on DAGs.

2. Add connector

   Settings → Connectors → Add → Airflow. Supply base URL (e.g. https://airflow.internal:8080).

3. Authenticate

   Use basic auth, API token, or mutual TLS depending on your deployment. Store credentials in the connector secret field, encrypted at rest.

4. Filter DAGs (optional)

   Include/exclude patterns reduce noise: ^analytics_.* for data team DAGs only.

5. Sync and link

   After sync, open a task node and confirm downstream links to dbt or Snowflake assets when task IDs or tags match naming conventions.

Snowflake

The Snowflake connector queries ACCOUNT_USAGE and INFORMATION_SCHEMA views for databases, schemas, tables, columns, views, dynamic tables, and query-history-derived lineage where available. Requires a dedicated read-only role.

1. Create Metroflow role and user

   Grant IMPORTED PRIVILEGES on SNOWFLAKE database for ACCOUNT_USAGE. Grant USAGE on schemas you want crawled.

2. Add connector

   Settings → Connectors → Add → Snowflake. Enter account identifier, warehouse, role, and service user.

3. Scope databases

   List databases to include; exclude SANDBOX_% schemas if desired. Large estates benefit from scoped crawls.

4. Enable lineage (optional)

   Turn on ACCESS_HISTORY parsing for automatic table-to-table edges. Requires Enterprise edition and added grants.

5. Verify against dbt

   dbt models should materializes to Snowflake tables. Mismatches often indicate custom database/schema mappings in profiles.yml.

Kafka

The Kafka connector registers topics and schemas from Confluent Schema Registry or compatible APIs. It maps Avro, Protobuf, and JSON schemas to column-level topic metadata, useful for streaming lineage into dbt staging models and Flink jobs.

1. Reach registry and bootstrap brokers

   Crawler pods need network access to schema registry HTTPS endpoint and optionally broker metadata APIs.

2. Add connector

   Settings → Connectors → Add → Kafka. Choose Schema Registry URL and authentication (API key, mTLS, or SASL).

3. Topic filters

   Include production namespaces: prod\.events\..*. Exclude internal _confluent.* topics.

4. Link to consumers

   Tag dbt sources or Flink job names matching topic names; Metroflow suggests edges when naming aligns.

5. Schedule sync

   Streaming schemas change frequently; hourly sync is typical for active registries.

Sync schedules

Each connector supports cron-style schedules, interval triggers, or webhook-only (no poll). Default recommendations:

Trigger manual sync: workspace connector card → Sync now, or POST /api/v1/connectors/{id}/sync. Rate limits apply per connector to protect source APIs.

Troubleshooting

Sync failed: authentication

Check connector test from the UI; re-validates credentials without full crawl. Rotate secrets if password policies expired. For Snowflake, confirm role is default for service user.

Partial graph / missing models

Verify include filters aren't too narrow. For dbt, confirm manifest path matches latest CI upload. Compare asset count in sync summary vs source system.

Lineage gaps between systems

Cross-connector edges require identifier alignment. dbt custom schema/database macros may prevent automatic Snowflake links; add explicit alias metadata or naming conventions document in glossary.

Slow crawls

Increase METROFLOW_CRAWLER_CONCURRENCY or add crawler replicas. Scope Snowflake to changed databases. Enable manifest hash short-circuit for dbt.

Timeout / rate limit from source

Backoff is automatic. Spread connector schedules to avoid top-of-hour piles. For Airflow, reduce DAG count via include patterns.

Full integration list

Metroflow ships 40+ connectors across warehouses, orchestrators, BI, ingestion, and DevOps tools. Browse searchable cards with logos and capability tags on the integrations page.

View all integrations →