Running Airflow with Docker on Developer Environment

What is Airflow:

Apache Airflow™ is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow provides extensible Python framework to build workflow which can be integrated with any technology. Web UI provided by airflow helps to manage workflows very effectively.

Airflow can be deployed in multiple ways from a single process on your laptop to a distributed setup to support even the biggest workflows.

Installing Airflow:

Here I will be explaining airflow standalone setup for developer.

What you need:

• Docker
• Python [with PIP and Virtual environment ]

So I assume now you have docker and python setup ready, lets get going.

1. Create new Dockerfile and copy the below code to it.

FROM apache/airflow:2.5.1-python3.9

COPY requirements.txt /requirements.txt

RUN pip install --no-cache-dir -r /requirements.txt

USER root

RUN apt-get update && apt-get install -y \
    wget

COPY start.sh /start.sh
RUN chmod +x /start.sh
USER airflow
ENTRYPOINT ["/bin/bash","/start.sh"]

Lets decode line by line

FROM apache/airflow:2.5.1-python3.9 : Refer airlfow base image

COPY requirements.txt /requirements.txt : copy the requirement.txt file this is helpful when we need to install required package for DAGs

RUN pip install --no-cache-dir -r /requirements.txt : install the packages

USER root : switch to root use

RUN apt-get update && apt-get install -y wget : update the apt-get and install wget

COPY start.sh /start.sh : start.sh has airflow comamnds we will be executing copy the file

RUN chmod +x /start.sh : change permission of start.sh file so we can execute

USER airflow: switch to airflow user

ENTRYPOINT ["/bin/bash","/start.sh"] : declare entry point commands.

2. Create start.sh file

#!/bin/bash
airflow standalone

The airflow standalone command initializes the database, creates a user, and starts all components. standalone command execute multiple commands like

airflow db migrate

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

airflow webserver --port 8080

airflow scheduler

3. Build the docker image

docker build . -t airflow-local

4. Run the image

docker run -p 8080:8080 -v /c/local/clone/path/airflow-examples/dags:/opt/airflow/dags  -d airflow-local

# Here replace local path of airflow DAG's repository, so that if you made changes to dags in the folder we don't have to restart docker

5. After successful run, this will initialize airflow database, webserver and scheduler. Airflow UI will be up and running at http://localhost:8080/home

6. Go to the running container to note password generated by airflow, we need it to login in the airlfow webserver

7. visit http://localhost:8080/home and on login page enter username and password you noted from logs.

After successful login, you should be able to see the home screen.

You can find all code at https://github.com/harshalpagar/airflow-examples.

Happy Learning :-)

Thank You.