10 Posts

Airflow

Apache Airflow for Data Science - How to Work with Variables

Apache Airflow for Data Science - How to Work with Variables

Hardcoding values in your Airflow DAGs is a bad practice. Learn how to use Airflow variables …

Apache Airflow for Data Science - How to Download Files from Amazon S3

Apache Airflow for Data Science - How to Download Files from Amazon S3

Learn how to download files from Amazon S3 (AWS) to your local machine with Apache Airflow and …

Apache Airflow for Data Science - How to Upload Files to Amazon S3

Apache Airflow for Data Science - How to Upload Files to Amazon S3

Learn how to setup an Amazon S3 (AWS) Bucket and how to upload files from local disk with …

Apache Airflow for Data Science - How to Work with REST APIs

Apache Airflow for Data Science - How to Work with REST APIs

Learn to work with REST APIs in Apache Airflow by utilizing HttpSensor and HttpOperator Airflow …

Apache Airflow for Data Science - How to Communicate Between Tasks with Airflow XComs

Apache Airflow for Data Science - How to Communicate Between Tasks with Airflow XComs

Learn to send and receive data between Airflow tasks with XComs, and when you shouldn’t …

Apache Airflow for Data Science - How to Run Tasks in Parallel

Apache Airflow for Data Science - How to Run Tasks in Parallel

Build a Data Pipeline (DAG) in Apache Airflow that makes four GET API requests in Parallel.

Apache Airflow for Data Science - How to Migrate Airflow Metadata DB to Postgres and Enable Parallel Execution

Apache Airflow for Data Science - How to Migrate Airflow Metadata DB to Postgres and Enable Parallel Execution

Apache Airflow doesn’t run tasks in parallel by default - but there’s an easy fix. …

Apache Airflow for Data Science - How to Work With Databases (Postgres)

Apache Airflow for Data Science - How to Work With Databases (Postgres)

Learn how to extract, transform, and load data with Airflow and Postgres database by coding a …

Apache Airflow for Data Science - How to Write Your First DAG in 10 Minutes

Apache Airflow for Data Science - How to Write Your First DAG in 10 Minutes

Apache Airflow is a common tool used by Data Engineers. Learn how to write your first data …

Apache Airflow for Data Science - How to Install Airflow Locally

Apache Airflow for Data Science - How to Install Airflow Locally

Want to learn Apache Airflow as a Data Engineer? Start by installing it locally. Go from zero …