Serverless ELT with GCS, BigQuery and Cloud Workflows
This video shows a complete example with a Serverless ELT batch pipeline with Cloud Storage, BigQuery and Cloud Workflows.
Extract : Cloud Storage
Load : Load raw files from GCS BigQuery
Transform : Transform raw to domain data with a SQL query and load the result to a final table
Backup processed files : Workflows invokes Cloud Build to move processed files from the input to the destination bucket. The advantage is to have access to the gcloud cli, because it's not possible natively with the storage API to move several files between buckets.
The deployment is done with Cloud Build.
Firstly the use case is execute with gcloud commands and bash scripts :
Deploy the workflow
Run the workflow with runtime arguments
Create a scheduler and cron job with Cloud Scheduler to launch the workflow with runtime arguments
Then the use case is execute with Terraform :
Deploy the workflow
Create a scheduler and cron job with Cloud Scheduler to launch the workflow with runtime arguments
#googlecloud #ELT #Pipeline #Serverless #CloudWorkflows #CloudStorage #BigQuery #CloudBuild #Terraform #CloudScheduler #sql
▸ Github :
https://github.com/tosun-si/teams-lea...
▸ Slides : https://docs.google.com/presentation/...
▸ LinkedIn : https://www.linkedin.com/posts/mazlum...
▸ X : https://x.com/MazlumTosun3/status/173...
Feel free to subscribe to the channel and click on the bell 🔔 to receive notifications for the next videos.
📲 Follow me on social networks :
▸ Articles : / mazlum.tosun
▸ X : / mazlumtosun3
▸ LinkedIn : / mazlum-tosun-900b1812
▸ WhatsApp : https://whatsapp.com/channel/0029VaCj...
