Skip to content
praveenanil.dev
← Back to blog

Kafka vs. Airflow: Stop Forcing Streams into Pipelines (and Pipelines into Streams)

2026-07-109 min readAzure, Data Engineering, Kafka, Airflow

You’ve probably seen this go wrong: someone shoves a streaming problem into Airflow “because we have Airflow,” or forces Kafka to coordinate a batch pipeline “because it’s already moving events.” The result is brittle DAGs that never stop running, or Kafka topics full of half-orchestrated state. In this post I’ll show you a cleaner way to think about it, with practical Azure-flavored examples you can run today.

The one-sentence difference

  • Kafka is a high-throughput, low-latency distributed log for event streams.
  • Airflow is a scheduler and orchestrator for discrete, dependency-driven tasks.

If you only keep that in your head, you’ll already make better design calls.

Why the confusion happens

Both tools move data. Both can trigger work. Both can retried-on-failure. So teams assume they’re interchangeable. They’re not. The difference is time and control:

  • Streams (Kafka) assume unbounded, continuous input where consumers pull records and manage their own state/checkpointing.
  • Pipelines (Airflow) assume bounded work units with clear start/finish boundaries and centralized orchestration.

When you blend these ideas, you’ll either:

  • Try to make Airflow “watch” a stream forever (spoiler: it hates that).
  • Or try to make Kafka “decide” what to run next (also a smell).

When Kafka is the right tool

Use Kafka (or a Kafka-compatible service) when:

  • You publish events as they happen: telemetry, CDC from databases, clickstreams, IoT signals.
  • You need to broadcast the same event to multiple consumers independently.
  • You need durable, replayable history so new consumers can catch up.
  • Latency matters (sub-second to a few seconds).
  • Throughput spikes are normal and backpressure must be absorbed.

On Azure:

  • Azure Event Hubs (Standard or Dedicated) exposes a Kafka-compatible endpoint. It’s a great default for managed Kafka protocol without running brokers yourself.
  • Confluent Cloud on Azure is a strong option when you need the full Kafka ecosystem as a managed service.

Trade-offs:

  • You handle consumer scale, partitions, offsets, and schema evolution.
  • Stateful stream processing (joins, windows) is powerful but adds operational complexity.
  • Pricing depends on ingress/egress, throughput units, and retention. Check the pricing page.

When Airflow is the right tool

Use Airflow when:

  • You have a directed acyclic graph (DAG) of tasks with dependencies.
  • Your work is periodic or ad hoc: daily batch, hourly summarization, backfills, data loads.
  • You need explicit retries, SLAs, and visibility into task status.
  • You orchestrate heterogenous systems: data warehouse loads, Spark jobs, notebooks, ML retraining, file movements.

On Azure:

  • I typically run Airflow on AKS or Container Apps. You can also use Astronomer or other managed Airflow providers on Azure.
  • Airflow is great to control Spark jobs (AKS/Synapse/Databricks), copy data (ADLS Gen2), and trigger downstream systems.

Trade-offs:

  • Not for hot paths. It’s a scheduler, not a message bus.
  • Long-lived, never-ending tasks are a smell. Airflow wants finite work.

Anti-patterns that burn teams

  1. Long-running Airflow sensors for “streaming”
  • Symptom: a DAG that never ends, using a sensor to poll Kafka endlessly.
  • Problem: Airflow’s scheduler and executor are designed for bounded tasks. Perma-running tasks consume slots, skew metrics, and complicate retries.
  1. Kafka as an orchestrator
  • Symptom: publishing “do-work” commands into Kafka to choose and sequence batch steps.
  • Problem: sequence, branching, and dependency resolution belong in an orchestrator. Kafka is a log, not a brain.
  1. Idempotency by hope
  • Symptom: exactly-once illusions in batch orchestrators or stream consumers without proper keys and checkpoints.
  • Problem: design for at-least-once, build idempotency into your tasks, and keep checkpoints out of cron-like tools.

The pattern that works: decouple ingestion from orchestration

  • Use Kafka for the hot path: receive events, buffer, fan-out to stream processors, and land raw data in a data lake (Bronze).
  • Use Airflow to orchestrate batch jobs on landed data: compaction, upserts into Delta tables, quality checks, feature builds, and BI loads.

On Azure that often looks like:

  • Event producers -> Event Hubs (Kafka endpoint) -> Stream processor (Flink/Spark Structured Streaming/Azure Stream Analytics) -> ADLS Gen2 Bronze.
  • Airflow DAGs kick off Spark/Databricks jobs to transform Bronze -> Silver -> Gold, run expectations, publish to a warehouse, notify downstreams.

Hands-on: minimal Azure setup

I’ll show you a thin slice you can run in a dev subscription:

  • Provision Event Hubs with Kafka endpoint.
  • Send and read events with a Kafka client.
  • Land events into ADLS Gen2 (simple consumer).
  • Orchestrate a daily batch compaction with Airflow.

1) Create a resource group, storage, and Event Hubs

I’m using Standard tier for Kafka protocol support.

# Variables
LOCATION=eastus
RG=rg-streams-pipelines-demo
STG=stgstreamspipes$RANDOM
EHNS=ehns-streams-pipelines-$RANDOM
EH=orders
 
# Create resource group
az group create -n $RG -l $LOCATION
 
# Storage account with hierarchical namespace (ADLS Gen2)
az storage account create \
  -g $RG -n $STG -l $LOCATION \
  --sku Standard_LRS \
  --kind StorageV2 \
  --hierarchical-namespace true
 
# Get a scoped SAS for simple local testing (avoid in prod)
EXPIRY=$(date -u -d "+1 day" +"%Y-%m-%dT%H:%MZ")
SAS=$(az storage account generate-sas \
  --permissions rwlacup \
  --account-name $STG \
  --services b \
  --resource-types sco \
  --expiry $EXPIRY -o tsv)
 
echo "Blob SAS: ?$SAS" > .blob_sas
 
# Event Hubs namespace (Standard supports Kafka)
az eventhubs namespace create -g $RG -n $EHNS -l $LOCATION --sku Standard
 
# Event Hub (topic)
az eventhubs eventhub create -g $RG --namespace-name $EHNS -n $EH

Grab connection strings and endpoints:

EH_CONN=$(az eventhubs namespace authorization-rule keys list \
  --resource-group $RG \
  --namespace-name $EHNS \
  --name RootManageSharedAccessKey \
  --query primaryConnectionString -o tsv)
 
EH_FQDN="${EHNS}.servicebus.windows.net:9093"
 
echo $EH_CONN > .eh_conn
echo $EH_FQDN > .eh_fqdn

Notes:

  • Kafka endpoint is available on Standard and Dedicated. Basic does not support Kafka protocol.
  • The SAS here is just to demo a local consumer writing to ADLS. For production, use service principals, managed identity, or workload identity.

2) Produce and consume with Kafka protocol

Below is a minimal Python producer using confluent-kafka. It authenticates to Event Hubs with SASL over TLS. The password is the Event Hubs connection string and the username is the literal $ConnectionString.

Install deps:

python -m venv .venv && source .venv/bin/activate
pip install confluent-kafka azure-storage-blob

Producer:

# file: producer.py
import os, json, time, random
from confluent_kafka import Producer
 
topic = os.environ.get("EH_TOPIC", "orders")
bootstrap = os.environ["EH_FQDN"]  # e.g. "ehns-xyz.servicebus.windows.net:9093"
conn = os.environ["EH_CONN"]       # Event Hubs connection string
 
conf = {
    "bootstrap.servers": bootstrap,
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "PLAIN",
    "sasl.username": "$ConnectionString",
    "sasl.password": conn,
    "client.id": "producer-demo",
    "linger.ms": 5,
}
 
p = Producer(conf)
 
def delivery(err, msg):
    if err:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()} [{msg.partition()}] @ {msg.offset()}")
 
i = 0
while True:
    i += 1
    event = {
        "order_id": i,
        "sku": f"SKU-{random.randint(100,999)}",
        "qty": random.randint(1,5),
        "ts": time.time()
    }
    p.produce(topic=topic, value=json.dumps(event).encode("utf-8"), callback=delivery)
    p.poll(0)
    time.sleep(0.5)

Run it:

export EH_TOPIC=$EH
export EH_FQDN=$(cat .eh_fqdn)
export EH_CONN=$(cat .eh_conn)
python producer.py

Consumer that writes newline-delimited JSON to ADLS Gen2:

# file: consumer_to_adls.py
import os, time, json
from datetime import datetime, timezone
from confluent_kafka import Consumer
from azure.storage.blob import BlobServiceClient
 
eh_topic = os.environ.get("EH_TOPIC", "orders")
bootstrap = os.environ["EH_FQDN"]
conn = os.environ["EH_CONN"]
 
account = os.environ["STG_ACCOUNT"]
sas = os.environ["STG_SAS"]  # "?sv=..."
container = os.environ.get("STG_CONTAINER", "bronze")
 
# Ensure container exists
bsc = BlobServiceClient(account_url=f"https://{account}.blob.core.windows.net", credential=sas)
try:
    bsc.create_container(container)
except Exception:
    pass
 
consumer = Consumer({
    "bootstrap.servers": bootstrap,
    "group.id": "adls-writer",
    "auto.offset.reset": "earliest",
    "security.protocol": "SASL_SSL",
    "sasl.mechanisms": "PLAIN",
    "sasl.username": "$ConnectionString",
    "sasl.password": conn,
    "enable.auto.commit": False
})
 
consumer.subscribe([eh_topic])
 
buffer = []
BATCH = 100
FLUSH_SECS = 10
last_flush = time.time()
 
def flush():
    global buffer, last_flush
    if not buffer:
        return
    utc = datetime.now(timezone.utc)
    path = f"events/eh={eh_topic}/dt={utc.strftime('%Y-%m-%d')}/hr={utc.strftime('%H')}/{int(utc.timestamp())}.json"
    blob = bsc.get_container_client(container).get_blob_client(path)
    data = ("\n".join(buffer)).encode("utf-8")
    blob.upload_blob(data)
    print(f"Wrote {len(buffer)} records to {path}")
    buffer = []
    last_flush = time.time()
 
try:
    while True:
        msg = consumer.poll(1.0)
        if msg is None:
            # periodic flush
            if time.time() - last_flush > FLUSH_SECS:
                flush()
            continue
        if msg.error():
            print(f"Error: {msg.error()}")
            continue
        buffer.append(msg.value().decode("utf-8"))
        if len(buffer) >= BATCH:
            flush()
            consumer.commit(asynchronous=False)
except KeyboardInterrupt:
    pass
finally:
    flush()
    consumer.commit(asynchronous=False)
    consumer.close()

Run it:

export EH_TOPIC=$EH
export EH_FQDN=$(cat .eh_fqdn)
export EH_CONN=$(cat .eh_conn)
export STG_ACCOUNT=$STG
export STG_SAS=$(cat .blob_sas)
python consumer_to_adls.py

You now have a true stream ingestion path landing raw JSON files partitioned by day/hour in ADLS Gen2.

3) Orchestrate a daily batch with Airflow

Now we’ll keep Airflow where it shines: a bounded daily job to compact raw files and publish a curated set. I’ll show a minimal DAG that:

  • Lists yesterday’s landed files.
  • Concatenates them into a single file.
  • Uploads the result to a “silver” container.
  • Notifies a downstream system (placeholder).

You can run this in any Airflow deployment. If you need a quick local environment, the official Airflow Docker Compose quickstart works well.

DAG:

# file: dags/daily_compact.py
from datetime import datetime, timedelta
import os, subprocess, glob
from airflow import DAG
from airflow.operators.python import PythonOperator
 
# Expect environment variables in the Airflow worker
STG_ACCOUNT = os.environ["STG_ACCOUNT"]
STG_SAS     = os.environ["STG_SAS"]
BRONZE      = os.environ.get("BRONZE_CONTAINER", "bronze")
SILVER      = os.environ.get("SILVER_CONTAINER", "silver")
EH_TOPIC    = os.environ.get("EH_TOPIC", "orders")
 
def _list_and_download(**context):
    # Yesterday in UTC
    ds = (datetime.utcnow() - timedelta(days=1)).strftime("%Y-%m-%d")
    # Hourly partitions
    files = []
    for hr in [f"{h:02d}" for h in range(0,24)]:
        prefix = f"https://{STG_ACCOUNT}.blob.core.windows.net/{BRONZE}/events/eh={EH_TOPIC}/dt={ds}/hr={hr}/"
        # Use Azure CLI or azcopy; here we'll use azcopy for speed
        # Requires azcopy available in the worker image
        cmd = [
            "azcopy", "copy",
            prefix + STG_SAS,
            f"/tmp/bronze/{ds}/{hr}/",
            "--recursive=true"
        ]
        subprocess.run(cmd, check=False)
        found = glob.glob(f"/tmp/bronze/{ds}/{hr}/*.json")
        files.extend(found)
    context["ti"].xcom_push(key="files", value=files)
 
def _compact(**context):
    files = context["ti"].xcom_pull(key="files")
    ds = (datetime.utcnow() - timedelta(days=1)).strftime("%Y-%m-%d")
    os.makedirs(f"/tmp/silver/{ds}", exist_ok=True)
    out = f"/tmp/silver/{ds}/orders_compacted_{ds}.json"
    with open(out, "wb") as w:
        for f in files:
            with open(f, "rb") as r:
                w.write(r.read())
                w.write(b"\n")
    context["ti"].xcom_push(key="out", value=out)
 
def _upload(**context):
    out = context["ti"].xcom_pull(key="out")
    ds = (datetime.utcnow() - timedelta(days=1)).strftime("%Y-%m-%d")
    dest = f"https://{STG_ACCOUNT}.blob.core.windows.net/{SILVER}/eh={EH_TOPIC}/dt={ds}/orders_compacted_{ds}.json"
    cmd = ["azcopy", "copy", out, dest + STG_SAS]
    subprocess.run(cmd, check=True)
 
default_args = {
    "owner": "data-eng",
    "retries": 2,
    "retry_delay": timedelta(minutes=5)
}
 
with DAG(
    dag_id="daily_orders_compact",
    start_date=datetime(2024,1,1),
    schedule_interval="0 2 * * *",  # 2 AM UTC daily
    catchup=False,
    default_args=default_args,
    tags=["batch","silver"]
):
    t1 = PythonOperator(task_id="download_bronze", python_callable=_list_and_download, provide_context=True)
    t2 = PythonOperator(task_id="compact", python_callable=_compact, provide_context=True)
    t3 = PythonOperator(task_id="upload_silver", python_callable=_upload, provide_context=True)
 
    t1 >> t2 >> t3

Notes:

  • This DAG is bounded: every run handles “yesterday” and finishes.
  • Airflow manages retries and observability. If compaction fails, you retry just that run.
  • In production, replace azcopy with first-class libs or operators, and write to open formats (e.g., Delta/Parquet) with Spark jobs that Airflow triggers.

4) Optional: provision with Bicep

If you prefer infra-as-code, here’s a Bicep file that deploys the storage account and Event Hubs namespace/hub. Adjust names to your standards.

// file: main.bicep
param location string = 'eastus'
param rgName string = 'rg-streams-pipelines-demo'
param stgName string
param ehnsName string
param ehName string = 'orders'
 
resource stg 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: stgName
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    isHnsEnabled: true
    allowBlobPublicAccess: false
    minimumTlsVersion: 'TLS1_2'
    supportsHttpsTrafficOnly: true
  }
}
 
resource ehns 'Microsoft.EventHub/namespaces@2024-01-01' = {
  name: ehnsName
  location: location
  sku: {
    name: 'Standard'
    tier: 'Standard'
    capacity: 1
  }
  properties: {
    isAutoInflateEnabled: true
    maximumThroughputUnits: 4
  }
}
 
resource eh 'Microsoft.EventHub/namespaces/eventhubs@2024-01-01' = {
  name: '${ehns.name}/${ehName}'
  properties: {
    messageRetentionInDays: 3
    partitionCount: 4
  }
}
 
resource auth 'Microsoft.EventHub/namespaces/authorizationRules@2024-01-01' = {
  name: '${ehns.name}/RootManageSharedAccessKey'
  properties: {
    rights: [
      'Listen'
      'Send'
      'Manage'
    ]
  }
}
 
output eventHubsFqdn string = '${ehnsName}.servicebus.windows.net:9093'

Deploy:

az group create -n $RG -l $LOCATION
az deployment group create -g $RG -f main.bicep \
  -p stgName=$STG ehnsName=$EHNS ehName=$EH

How Kafka and Airflow work together without stepping on each other

  • Kafka owns the event backbone. It handles spikes, maintains ordering within partitions, and lets multiple teams build consumers at their own pace. Your stream processors do near-real-time enrichments and land raw/ready data.
  • Airflow owns the batch lifecycle. It sequences compaction, vacuuming, z-ordering, CDC merges into tables, validations, training, and downstream loads. Each run is self-contained, idempotent, and observable.

This decoupling scales organizationally, too. Streaming teams can evolve schemas and partitioning. Analytics teams can add DAGs without touching the ingestion path.

What about event-driven batch?

Sometimes you want batch to follow stream landing more tightly than “daily at 2 AM.”

Two solid options:

  • Event-driven trigger to orchestrator: A lightweight function or webhook calls an Airflow DAG run when new partitions arrive. Keep the DAG bounded to the new partition set.
  • Micro-batch in the stream layer: Use streaming engines to trigger every few minutes and write compacted outputs continuously. Airflow then runs coarse-grained maintenance tasks less frequently.

Avoid tailing Kafka directly from a DAG as a “sensor forever.”

Operational tips I wish I learned earlier

  • Throughput units vs partitions: In Event Hubs, provision enough throughput units and partitions to meet peak rates and parallelism. Benchmark with real producers/consumers before production. Check the pricing page for cost trade-offs.
  • Retention is your friend: Keep enough retention to replay from schema changes or consumer outages, but not so much that storage costs surprise you. Again, check the pricing page.
  • Schema management: Use a schema registry (Confluent Schema Registry or alternatives). Enforce forward/backward compatibility and evolve intentionally.
  • Idempotency everywhere: Stream consumers and batch tasks should be safe to retry. Use deterministic keys, upserts (merge), and write-once paths per run.
  • Observability: For streams, track consumer lag and processing delay. For Airflow, set SLAs and alert on late tasks. Logs are not metrics.
  • Security posture: Prefer managed identity and private endpoints on Azure. For local tests I used SAS and connection strings; switch to managed identities in production.

Common “but can’t I just…” questions

  • “Can’t I just use Airflow to poll Kafka and process events?” You can, but you’ll fight the scheduler, lose elasticity, and blur the line between orchestration and processing. Use a stream processor that scales with partitions and time.
  • “Can’t Kafka trigger my batch job?” It can emit an event. Let a small function translate that into a bounded DAG run with clear parameters (e.g., which partition/hour to process). Don’t bake orchestration rules into the bus.
  • “We only need events for one consumer; Airflow is simpler.” If volume is tiny and latency tolerance is high, you might not need Kafka. A queue or just batch landing + Airflow is fine. Don’t over-engineer.
  • “We have a lakehouse; where does it fit?” Stream land to Bronze quickly, curate in Silver with batch or structured streaming, and publish Gold for analytics. Airflow is great for the Silver/Gold orchestration; streams keep Bronze flowing.

Cost and sizing sanity

  • Event Hubs capacity and retention drive cost; choose Standard vs Dedicated carefully. Check the pricing page and model your expected ingress/egress and retention.
  • Airflow cost is mostly the compute you provision (AKS/Container Apps) and any workers you scale out. Size for concurrency you actually need; use autoscaling executors if appropriate. Check the pricing pages for your chosen compute and storage.

I deliberately didn’t include made-up numbers here. Always check the pricing page for accurate, current prices.

A realistic small-team architecture on Azure

  • Event producers (apps, CDC) -> Event Hubs (Kafka endpoint), topic-per-domain, partitions sized to throughput.
  • Streaming processor (Spark Structured Streaming on Databricks or AKS, or Azure Stream Analytics) writes to ADLS Gen2 Bronze with partitioning by date and hour.
  • Airflow on AKS orchestrates:
    • Daily compaction and optimize steps.
    • Quality checks with Great Expectations or native validations.
    • Merge/upsert into curated tables.
    • Downstream syncs to your warehouse or serving layers.
  • Optional: A function triggers the Airflow DAG with parameters when a new hour lands, for near-real-time freshness without long-running sensors.

This keeps each tool in its lane and gives you clear SLIs: event lag for streaming, DAG duration for batch.

Wrapping up

If you remember one thing, remember this: Kafka moves events; Airflow moves decisions. Kafka is the backbone of your real-time fabric; Airflow is the conductor of your batch orchestra. When you stop forcing streams into pipelines (and pipelines into streams), designs get simpler, costs get clearer, and on-call gets quieter.

In this post I gave you a from-the-ground-up split of responsibilities, plus runnable Azure examples: Kafka-compatible ingestion on Event Hubs, a simple consumer landing to ADLS Gen2, and a clean, bounded Airflow DAG for daily compaction. Use them as a scaffold. Start small, measure lag and runtimes, and scale only where you need to.

Praveen Anil

Praveen Anil

Infrastructure Lead · Azure & AI · About me