feat(azure-data-factory): add Azure Data Factory connector

[askumar27]

feat(ingestion): Add Azure Data Factory connector

📋 Summary

Add a new metadata ingestion connector for Azure Data Factory (ADF) that extracts pipelines, activities, datasets, lineage, and execution history into DataHub.

🎯 Motivation

Azure Data Factory is a widely-used cloud ETL/ELT service for data integration and orchestration. Organizations using ADF need visibility into their data pipelines, lineage, and execution history within DataHub to:

• Discover and understand data pipelines across their Azure infrastructure
• Track data lineage from source to destination systems
• Monitor pipeline execution history and status
• Enable governance and compliance for data workflows

This connector fills a gap in DataHub's Azure ecosystem coverage, complementing existing Azure connectors (Azure AD, Azure Blob Storage).

🔧 Changes Overview

New Features

• Azure Data Factory Source Connector - Full metadata extraction from ADF

  • Data Factories → DataHub Containers
  • Pipelines → DataHub DataFlows
  • Activities → DataHub DataJobs (Copy, Data Flow, Lookup, ExecutePipeline, etc.)
  • Datasets → DataHub Datasets (mapped by linked service type)
  • Pipeline Runs → DataHub DataProcessInstances (optional)

• Table-Level Lineage Extraction

  • Copy Activity input/output dataset mapping
  • Data Flow source/sink extraction
  • Lookup Activity dataset references
  • Platform mapping for 40+ linked service types (Snowflake, SQL, BigQuery, S3, Synapse, etc.)

• Pipeline-to-Pipeline Lineage

  • ExecutePipeline activities create lineage to child pipeline's first activity
  • Enables tracing orchestration hierarchies across nested pipelines
  • Direction: ExecutePipeline → ChildFirstActivity

• Data Flow Script Extraction

  • Stores ADF Data Flow transformation scripts in dataTransformLogic aspect
  • Enables viewing transformation logic in DataHub UI

• Execution History (Optional)

  • Pipeline run status, duration, and timestamps
  • Trigger information
  • Run parameters
  • Configurable lookback period (1-90 days)

• Unified Azure Authentication Module

  • Reusable AzureCredentialConfig class for future Azure connectors
  • Supports: Service Principal, Managed Identity, Azure CLI, DefaultAzureCredential

• Stateful Ingestion

  • Stale entity removal support
  • Tracks previously ingested entities

Dependencies

• azure-identity>=1.21.0 - Azure authentication
• azure-mgmt-datafactory>=9.0.0 - ADF Management SDK

🏗️ Architecture/Design Notes

SDK V2 Implementation

This connector uses DataHub SDK V2 (datahub.sdk.Container, datahub.sdk.DataFlow, datahub.sdk.DataJob), following the pattern established by modern connectors. Benefits:

• Type-safe entity creation
• Automatic aspect emission (dataPlatformInstance, status, browsePathsV2)
• Proper browse path hierarchy via parent_container

Entity Hierarchy

Azure Data Factory (Platform)
└── Data Factory (Container)
    └── Pipeline (DataFlow)
        └── Activity (DataJob)
            └── Pipeline Run (DataProcessInstance) [optional]

URN Strategy

Pipeline URNs include factory name for uniqueness across multiple factories:

urn:li:dataFlow:(azure-data-factory,{factory_name}.{pipeline_name},{env})
urn:li:dataJob:(urn:li:dataFlow:(azure-data-factory,{factory_name}.{pipeline_name},{env}),{activity_name})

This ensures uniqueness when multiple factories have identically-named pipelines.

Lineage Types

Lineage Type	Source	Target	Mechanism
Dataset → DataJob	Copy Activity inputs	DataJob inlets	dataJobInputOutput.inputDatasets
DataJob → Dataset	Copy Activity outputs	DataJob outlets	dataJobInputOutput.outputDatasets
DataJob → DataJob	ExecutePipeline	Child's first activity	dataJobInputOutput.inputDatajobs
Data Flow Sources	ExecuteDataFlow	Source datasets	Parsed from Data Flow definition
Data Flow Sinks	ExecuteDataFlow	Sink datasets	Parsed from Data Flow definition

Reusable Azure Auth

The AzureCredentialConfig class in azure/azure_auth.py is designed to be reused by future Azure connectors (e.g., Azure Synapse, Azure Purview).

🧪 Testing

Unit Tests (35 tests)

• Configuration validation
• Linked service platform mapping (40+ service types)
• Activity subtype mapping (20+ activity types)
• Table/file name extraction logic
• Run status mapping
• Resource group extraction

Integration Tests (16 tests, 10 golden files)

Test Scenario	Description	Golden File
Basic ingestion	Factories, pipelines, activities	adf_basic_golden.json
Execution history	Pipeline runs and status	adf_with_runs_golden.json
Platform instance	Multi-instance configuration	adf_platform_instance_golden.json
Nested pipelines	ExecutePipeline with children	adf_nested_golden.json
ForEach loops	Iterative processing	adf_foreach_golden.json
Branching	If-Condition/Switch activities	adf_branching_golden.json
Data Flows	Mapping Data Flow lineage	adf_dataflow_golden.json
Multi-source ETL	SQL→Blob→Synapse chain	adf_multisource_golden.json
Diverse activities	9+ activity types	adf_diverse_golden.json
Mixed dependencies	Pipeline + Dataset lineage	adf_mixed_deps_golden.json

Test Coverage

• All tests passing: 51/51 (35 unit + 16 integration)
• Golden files validate complete MCP output
• Mocked Azure API responses based on real SDK structures

📊 Impact Assessment

Aspect	Assessment
Affected Components	New connector only - no changes to existing code
Breaking Changes	None
Performance Impact	N/A (new feature)
Risk Level	Low - Isolated new connector with comprehensive tests

🚀 Deployment Notes

Installation

pip install 'acryl-datahub[azure-data-factory]'

Required Configuration

source:
  type: azure-data-factory
  config:
    subscription_id: ${AZURE_SUBSCRIPTION_ID}
    credential:
      authentication_method: service_principal
      client_id: ${AZURE_CLIENT_ID}
      client_secret: ${AZURE_CLIENT_SECRET}
      tenant_id: ${AZURE_TENANT_ID}

Azure Permissions Required

• Reader role on Data Factory resources (minimum)
• Data Factory Contributor for execution history

Platform Instance Usage

Use platform_instance when you have multiple ADF deployments that need to be distinguished:

# Scenario: Same pipeline names across dev/prod factories
source:
  type: azure-data-factory
  config:
    subscription_id: ${AZURE_SUBSCRIPTION_ID}
    platform_instance: "production"  # Distinguishes from dev instance

📁 Files Changed

Path	Description
setup.py	Plugin registration and dependencies
source/azure/azure_auth.py	Reusable Azure authentication config
source/azure_data_factory/adf_source.py	Main connector implementation
source/azure_data_factory/adf_config.py	Configuration classes
source/azure_data_factory/adf_client.py	Azure API client wrapper
source/azure_data_factory/adf_models.py	Pydantic models for API responses
source/azure_data_factory/adf_report.py	Ingestion report metrics
docs/sources/azure_data_factory/	User documentation
tests/unit/azure_data_factory/	Unit tests (35 tests)
tests/integration/azure_data_factory/	Integration tests (16 tests, 10 golden files)

📚 Documentation

• User Guide - Prerequisites, capabilities, configuration
• Quick Reference - Setup and config table
• Example Recipe

✅ Checklist

• Code follows DataHub SDK V2 patterns
• Unit tests added (35 tests)
• Integration tests with golden files (16 tests, 10 golden files)
• Documentation created
• Linting passes (ruff, mypy)
• No breaking changes to existing code
• Pipeline-to-pipeline lineage implemented
• Dataset lineage implemented
• Data Flow script extraction implemented
• Mixed dependencies test validates both lineage types

🔗 References

• Azure Data Factory Documentation
• DataHub SDK V2 Examples
• Connectors Accelerator Guidelines

[codecov]

Codecov Report

❌ Patch coverage is 85.04028% with 130 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
.../ingestion/source/azure_data_factory/adf_source.py	85.57%	60 Missing ⚠️
.../ingestion/source/azure_data_factory/adf_client.py	54.62%	49 Missing ⚠️
.../ingestion/source/azure_data_factory/adf_report.py	85.91%	10 Missing ⚠️
...n/src/datahub/ingestion/source/azure/azure_auth.py	80.85%	9 Missing ⚠️
.../ingestion/source/azure_data_factory/adf_models.py	99.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[alwaysmeticulous]

✅ Meticulous spotted 0 visual differences across 982 screens tested: view results.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Expected differences? Click here. Last updated for commit c6943ee. This comment will update as new commits are pushed.}

[codecov]

Bundle Report

Changes will increase total bundle size by 92.83kB (0.32%) ⬆️. This is within the configured threshold ✅

Detailed changes

Bundle name	Size	Change
datahub-react-web-esm	28.85MB	92.83kB (0.32%) ⬆️

Affected Assets, Files, and Routes:

view changes for bundle: datahub-react-web-esm

Assets Changed:

Asset Name	Size Change	Total Size	Change (%)
assets/index-*.js	92.83kB	19.23MB	0.49%

Files in assets/index-*.js:

• ./src/app/ingest/source/builder/constants.ts → Total Size: 7.08kB

• ./src/app/ingestV2/source/builder/constants.ts → Total Size: 6.22kB

• ./src/images/azuredatafactorylogo.svg → Total Size: 1.8kB

[sgomezvillamor]

From what I see, only service principal config validation is tested but not the auth itself.
Is there any way we could properly test all authentication mechanisms?

[askumar27]

Correct - we test configuration validation only (required fields, valid combinations). Testing actual authentication would require:

1. Live Azure credentials - will be added to connector tests
2. Mock Azure AD token endpoints (complex) - can give it a shot

[sgomezvillamor]

maybe this is easier to cover in connector-tests

[askumar27]

yes, that's the plan :)

[sgomezvillamor]

is there any public doc with all existing platform keys that we can reference here?

[askumar27]

Update code comments - added link to metadata-service/configuration/src/main/resources/bootstrap_mcps/data-platforms.yaml which is the canonical source of platform identifiers.

[sgomezvillamor]

I meant public doc for ADF linked services list 😅

[askumar27]

Not sure if there is one 🤔

[sgomezvillamor]

Metric and log look really good

As an extra bonus, I just missed:

• some more fine-grained api call tracking (counters and time)
• some info/debug logs along the code
• lineage_edges_extracted could be splited for different types of lineage

[askumar27]

• Added api_call_counts_by_type: Dict[str, int] for granular tracking
• Added total_api_response_time_seconds: float for timing
• Split lineage into dataset_lineage_extracted, pipeline_lineage_extracted, dataflow_lineage_extracted

[sgomezvillamor]

super. thanks!

what about

some info/debug logs along the code

yesterday I just found a couple of info logs

[askumar27]

Added some more :) trying to make sure there are no duplicate info from the CLI report