ETL Batch pipeline with Cloud Storage, Scala Dataproc Serverless, BigQuery orchestrated by Airflow
This video shows a complete use case with an ETL Batch Pipeline deployed on Google Cloud.
This pipeline stages are :
Extract : Cloud Storage
Transform : Spark Scala and Dataproc Serverless
Load : BigQuery
Everything is orchestrated by Cloud Composer and Apache Airflow.
The CI CD part is done with Cloud Build.
#googlecloud #Airflow #CloudComposer #ETL #CloudStorage #BigQuery#Spark #Scala #Dataproc #Serverless
▸ Github : https://github.com/tosun-si/teams-lea...
▸ X : https://x.com/MazlumTosun3/status/169...
Feel free to subscribe to the channel and click on the bell 🔔 to receive notifications for the next videos.
📲 Follow me on social networks :
▸ Articles : / mazlum.tosun
▸ X : / mazlumtosun3
▸ LinkedIn : / mazlum-tosun-900b1812
▸ WhatsApp : https://whatsapp.com/channel/0029VaCj...
