Setting Up Airflow 101 - Google Cloud Composer Vs Astronomer Vs MWAA
Over the last 3 months, I have taken on two different migrations that involved taking companies from manually managing Airflow VMs to going over to using Cloud Composer and MWAA (Managed Workflows For Apache Airflow).
These are two great options when it comes to starting your first Airflow project. They help reduce a lot of issues such as scaling workloads across workers and managing all the other various components required to run Airflow at scale.
Of course, both Cloud Composer and MWAA are services provided by two of the larger cloud providers. But there is a third option called Astronomer.
In this article, I want to discuss some of the benefits and limitations of Cloud Composer and MWAA as well as discuss why Astronomer might be a better fit for many companies.
If you enjoyed this video, check out some of my other top videos.
What Skills Do Data Engineers Need?
• What Skills Do Data Engineers Need To...
Data Engineering Project Ideas
• 5 Data Sources for Your Data Engineer...
If you want to learn more about machine learning, check out DataCamps Machine Learining Course
https://bit.ly/3BeLEml
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
https://seattledataguy.substack.com/
Or check out my blog
https://www.theseattledataguy.com/
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.
