Running Airflow with Docker on Developer Environment

3 min readSep 4, 2023

What is Airflow:

Apache Airflow™ is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows. Airflow provides extensible Python framework to build workflow which can be integrated with any technology. Web UI provided by airflow helps to manage workflows very effectively.

Airflow can be deployed in multiple ways from a single process on your laptop to a distributed setup to support even the biggest workflows.

Installing Airflow:

Here I will be explaining airflow standalone setup for developer.

What you need:

Docker
Python [with PIP and Virtual environment ]

So I assume now you have docker and python setup ready, lets get going.

Create new Dockerfile and copy the below code to it.

FROM apache/airflow:2.5.1-python3.9

COPY requirements.txt /requirements.txt

RUN pip install --no-cache-dir -r /requirements.txt

USER root

RUN apt-get update && apt-get install -y \
    wget

COPY start.sh /start.sh
RUN chmod +x /start.sh
USER airflow
ENTRYPOINT ["/bin/bash","/start.sh"]

Lets decode line by line

FROM apache/airflow:2.5.1-python3.9 : Refer airlfow base image
COPY requirements.txt /requirements.txt : copy the requirement.txt file this is helpful when we need to install required package for DAGs
RUN pip install --no-cache-dir -r /requirements.txt : install the packages
USER root : switch to root use
RUN apt-get update && apt-get install -y wget : update the apt-get and install wget
COPY start.sh /start.sh : start.sh has airflow comamnds we will be executing copy the file
RUN chmod +x /start.sh : change permission of start.sh file so we can execute
USER airflow: switch to airflow user
ENTRYPOINT ["/bin/bash","/start.sh"] : declare entry point commands.

2. Create start.sh file

#!/bin/bash
airflow standalone

The airflow standalone command initializes the database, creates a user, and starts all components. standalone command execute multiple commands like

airflow db migrate

airflow users create \
    --username admin \
    --firstname Peter \
    --lastname Parker \
    --role Admin \
    --email spiderman@superhero.org

airflow webserver --port 8080

airflow scheduler

3. Build the docker image

docker build . -t airflow-local

4. Run the image

docker run -p 8080:8080 -v /c/local/clone/path/airflow-examples/dags:/opt/airflow/dags  -d airflow-local

# Here replace local path of airflow DAG's repository, so that if you made changes to dags in the folder we don't have to restart docker

5. After successful run, this will initialize airflow database, webserver and scheduler. Airflow UI will be up and running at http://localhost:8080/home

6. Go to the running container to note password generated by airflow, we need it to login in the airlfow webserver