[ad_1]
Apache Airflow is an open-source challenge for scheduling and managing workflows, written in Python.
Kaxil Naik, director of Airflow engineering at Astronomer and one of many core committers of Airflow, informed SD Occasions: “It’s used to automate your each day jobs or each day duties, and duties may be so simple as operating a Python script or it may be as sophisticated as bringing in all the information from 500 completely different information warehouses and manipulating it.”
It was created at Airbnb in 2014 and is about to rejoice its 10 yr anniversary later this yr. It joined the Apache Software program Basis in March 2016 on the Incubation stage and was made a top-level challenge in 2019.
Airflow was initially designed for simply ETL use circumstances, however has over time developed so as to add options that make it helpful for all features associated to information engineering.
“It has continued to be the chief on this house, as a result of we’ve maintained stability between innovation and stability. Due to this nearly 10 years of Airflow in the identical house, we’ve added so many options that enable Airflow to be very dependable and steady,” he mentioned.
The newest launch, 2.9, got here out earlier this week and added new options like the power to mix dataset and time-based schedules, customized names for Dynamic Activity Mapping, and the power to group activity logs.
The challenge may be discovered on GitHub right here.
[ad_2]
Supply hyperlink
