Containers for Data Engineers

Starter's Guide

Throughout my career, I’ve seen many evolutions of how we, as data engineers, “ship” pipelines to production. Some of the common methods are through:

• CICD Pipelines in Gitlab/Github

• Emailing the script to the DBA to run (No one will ever admit this though 😆)

• Saving bash jobs to a shared drive and using cron to run them (or windows task scheduler)

• Airflow+S…