Understanding Transformer Pipelines

In this lesson, we learn what a Transformer pipeline is, what situations they are used in, and why we need it to create successful streaming pipelines for data ingestion. The Transformer Engine is an execution engine that runs data processing pipelines on Apache Spark, taking advantage of Spark’s performance benefits.

We’ll give you a full walkthrough of building your first Transformer pipeline with advanced processing. In the next lesson, we’ll walk through the architecture of the Transformer Engine and more.