Error handling and monitoring with Airflow and Cloud Composer
This video shows of how we can apply error handling and monitoring efficiently with Airflow and Cloud Composer in Google Cloud
💡 The principle is to use an interceptor and failure callback in our DAGs.
What is the advantage of this practice ❓ ❓
✅ The logic is 𝐜𝐞𝐧𝐭𝐫𝐚𝐥𝐢𝐳𝐞𝐝 𝐢𝐧 𝐨𝐧𝐞 𝐩𝐥𝐚𝐜𝐞 and we prevent the repetition of this technical code in different places in our DAGs. This code repetition is 𝐞𝐫𝐫𝐨𝐫 𝐩𝐫𝐨𝐧𝐞 𝐚𝐧𝐝 𝐩𝐨𝐥𝐥𝐮𝐭𝐞 𝐭𝐡𝐞 𝐫𝐞𝐬𝐭 𝐨𝐟 𝐭𝐡𝐞 𝐜𝐨𝐝𝐞 dedicated to orchestration
✅ We use 𝐬𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧 𝐨𝐟 𝐜𝐨𝐧𝐜𝐞𝐫𝐧, because this responsibility is delegated outside of the DAGs, written once and centralized in one place
✅ We can easily use a 𝐥𝐨𝐠 𝐩𝐚𝐭𝐭𝐞𝐫𝐧 𝐚𝐧𝐝 𝐢𝐧𝐭𝐞𝐫𝐜𝐞𝐩𝐭 𝐢𝐭 via Cloud Monitoring, a Log Based Metric and an Alerting Policy
✅ The errors are stored in a BigQuery table to be analyzed more easily with SQL queries
✅ The errors are proposed in a Looker Studio dashboard to allow partners in the project to visualize the data 📊
🔛 🍃 This use case uses the following actions :
➡ Create the 𝐋𝐨𝐠 𝐁𝐚𝐬𝐞𝐝 𝐌𝐞𝐭𝐫𝐢𝐜 with Terraform and Cloud Build to filter and 𝐢𝐧𝐭𝐞𝐫𝐜𝐞𝐩𝐭 𝐭𝐡𝐞 𝐥𝐨𝐠 written by the DAGs interceptor
➡ Create the 𝐀𝐥𝐞𝐫𝐭𝐢𝐧𝐠 𝐏𝐨𝐥𝐢𝐜𝐲 based on the previous Log Based metric, with Terraform and Cloud Build
➡ Deploy the 𝐃𝐀𝐆𝐬 𝐢𝐧 𝐂𝐥𝐨𝐮𝐝 𝐂𝐨𝐦𝐩𝐨𝐬𝐞𝐫 via Cloud Build
➡ Simulate 𝟐 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝐞𝐫𝐫𝐨𝐫𝐬 in two DAGs
➡ Check the 𝐞𝐦𝐚𝐢𝐥𝐬 sent by the alerting policy in real time
➡ Visualize the result in the 𝚍𝚊𝚐_𝚏𝚊𝚒𝚕𝚞𝚛𝚎 table in BigQuery
➡ Visualize the result in a 𝙇𝙤𝙤𝙠𝙚𝙧 𝙎𝙩𝙪𝙙𝙞𝙤 𝘿𝙖𝙨𝙝𝙗𝙤𝙖𝙧𝙙, that targets on the 𝚍𝚊𝚐_𝚏𝚊𝚒𝚕𝚞𝚛𝚎 table. We use a partition by day on the ingestion date to optimize the cost.
#GoogleCloud #Airflow #CloudComposer #errorhandling #monitoring
▸ Github :
https://github.com/tosun-si/airflow-c...
▸ Slides : https://docs.google.com/presentation/...
Feel free to subscribe to the channel and click on the bell 🔔 to receive notifications for the next videos.
📲 Follow me on social networks :
▸ Articles : / mazlum.tosun
▸ X : / mazlumtosun3
▸ LinkedIn : / mazlum-tosun-900b1812
▸ WhatsApp : https://whatsapp.com/channel/0029VaCj...
