Blog Big Data (page 1/2)

Big Data pipelines are designed to process a tons of data, however it is still common to need to execute some task once, either before or after (for batches) the flow was processed. Let see how to do it with Apache Beam.

Apache Beam 2.9.0 has released last week. So if you were using a previous version you are likely tempted to upgrade. This is what I did, an what was expected to be a single line change in a pom has surprisingly been more work. Let’s see what changed!

Apache Beam coder abstraction enables you to switch between implementations without rewriting your pipeline. But how to select your coder? Performance and disk spaces are likely the most important criterias, let’s see how to measure them.

We previously saw that Beam Row structure allows to write generic transforms but that using its serialization can be a bad bet. To illustrate how to switch between one format to another, we will show in this post how to convert a Row to an IndexedRecord

Handing data you don’t know at compile time is a common concern of processing libraries. Apache Beam can’t ignore that since it allows to build portable pipelines for Big Data engines. Let’s see how they started to solve that concern!