Open the editor

Pipe-and-filter architecture

Independent processing steps connected by pipes, each doing one transformation.

Pipe-and-filter architecture diagram: Source, Filters, Pipes, Sink, Error output

Pipe-and-filter breaks processing into a chain of independent filters connected by pipes. Each filter reads input, does one transformation and writes output; it knows nothing about the filters before or after it. The pipes carry data between them, as streams, queues or files.

Unix shell pipelines are the classic example (cat | grep | sort). The same idea runs ETL jobs, compilers, media encoders, CI pipelines and stream processors.

The parts of the pattern

Source
Where data enters: files, a queue, an API or a database change stream.
Filters
Self-contained steps such as parse, validate and enrich. Each can be tested and reused alone.
Pipes
The connections between filters: in-memory streams, message queues or intermediate files.
Sink
Where the finished data ends up, such as a warehouse table.
Error output
Records a filter cannot handle go to a separate output instead of stopping the pipeline.

How it works

  1. The source emits raw CSV rows into the first pipe.
  2. The parse filter turns rows into typed records.
  3. The validate filter passes good records on and sends bad ones to the rejected store.
  4. The enrich filter adds derived fields, and the sink writes the result.

When to use it, and when not to

Good fit

  • Data processing that is naturally a sequence of steps: ETL, media, document conversion, build pipelines.
  • Steps that should be reordered, reused or scaled separately.
  • Batch or streaming workloads where throughput matters more than interactivity.

Think twice

  • Interactive request and response flows that need shared state between steps.
  • Steps that must all succeed or fail together in one transaction.
  • Very fine-grained filters where the cost of moving data between them dominates.

Edit this diagram

Click Open the diagram in the editor to get your own copy, saved in your browser. Rename the boxes to match your system, swap the generic boxes for technology icons from the palette, then export a PNG, SVG or PDF, or share a link. No sign-up.