Mastering Streaming Data Pipelines with Kappa Architecture
These days, experience with streaming data is a common requirement in most data engineering job postings. It seems that every business has a need, or at least an appetite, for streaming data. So, what’s all the fuss about? How do we build pipelines that support this type of data flow?
To illustrate the various concepts, we will build a pipeline that processes stock prices in near-real time and share some latency metrics. The code to deploy the pipeline and review the data is available on github.
