docker compose down --volumes --rmi all
Using custom images
When you want to run Airflow locally, you might want to use an extended image, containing some additional dependencies - for
example you might add new python packages, or upgrade airflow providers to a later version. This can be done very easily
by specifying build: . in your docker-compose.yaml and placing a custom Dockerfile alongside your
docker-compose.yaml. Then you can use docker compose build command
to build your image (you need to do it only once). You can also add the --build flag to your docker compose commands
to rebuild the images on-the-fly when you run other docker compose commands.
Examples of how you can extend the image with custom providers, python packages,
apt packages and more can be found in Building the image.
Special case - adding dependencies via requirements.txt file
Usual case for custom images, is when you want to add a set of requirements to it - usually stored in
requirements.txt file. For development, you might be tempted to add it dynamically when you are
starting the original airflow image, but this has a number of side effects (for example your containers
will start much slower - each additional dependency will further delay your containers start up time).
Also it is completely unnecessary, because docker compose has the development workflow built-in.
You can - following the previous chapter, automatically build and use your custom image when you
iterate with docker compose locally. Specifically when you want to add your own requirement file,
you should do those steps:
Comment out the image: ... line and remove comment from the build: . line in the
docker-compose.yaml file. The relevant part of the docker-compose file of yours should look similar
to (use correct image tag):
#image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.5.3}
build: .
Create Dockerfile in the same folder your docker-compose.yaml file is with content similar to:
FROM apache/airflow:2.5.3
ADD requirements.txt .
RUN pip install -r requirements.txt
Place requirements.txt file in the same directory.
Run docker compose build to build the image, or add --build flag to docker compose up or
docker compose run commands to build the image automatically as needed.
Networking
In general, if you want to use Airflow locally, your DAGs may try to connect to servers which are running on the host. In order to achieve that, an extra configuration must be added in docker-compose.yaml. For example, on Linux the configuration must be in the section services: airflow-worker adding extra_hosts: - "host.docker.internal:host-gateway"; and use host.docker.internal instead of localhost. This configuration vary in different platforms. Please check the Docker documentation for Windows and Mac for further information.
FAQ: Frequently asked questions
ModuleNotFoundError: No module named 'XYZ'
The Docker Compose file uses the latest Airflow image (apache/airflow). If you need to install a new Python library or system library, you can customize and extend it.
What's Next?
From this point, you can head to the Tutorials section for further examples or the How-to Guides section if you're ready to get your hands dirty.
Environment variables supported by Docker Compose
Do not confuse the variable names here with the build arguments set when image is built. The
AIRFLOW_UID build arg defaults to 50000 when the image is built, so it is
"baked" into the image. On the other hand, the environment variables below can be set when the container
is running, using - for example - result of id -u command, which allows to use the dynamic host
runtime user id which is unknown at the time of building the image.
AIRFLOW_UID
UID of the user to run Airflow containers as.
Override if you want to use use non-default Airflow
UID (for example when you map folders from host,
it should be set to result of id -u call.
When it is changed, a user with the UID is
created with default name inside the container
and home of the use is set to /airflow/home/
in order to share Python libraries installed there.
This is in order to achieve the OpenShift
compatibility. See more in the
Arbitrary Docker User
50000
Before Airflow 2.2, the Docker Compose also had AIRFLOW_GID parameter, but it did not provide any additional
functionality - only added confusion - so it has been removed.
Those additional variables are useful in case you are trying out/testing Airflow installation via Docker Compose.
They are not intended to be used in production, but they make the environment faster to bootstrap for first time
users with the most common customizations.
_AIRFLOW_WWW_USER_USERNAME
Username for the administrator UI account.
If this value is specified, admin UI user gets
created automatically. This is only useful when
you want to run Airflow for a test-drive and
want to start a container with embedded development
database.
airflow
_AIRFLOW_WWW_USER_PASSWORD
Password for the administrator UI account.
Only used when _AIRFLOW_WWW_USER_USERNAME set.
airflow
_PIP_ADDITIONAL_REQUIREMENTS
If not empty, airflow containers will attempt to
install requirements specified in the variable.
example: lxml==4.6.3 charset-normalizer==1.4.1.
Available in Airflow image 2.1.1 and above.
Cleaning up
Using custom images
Special case - adding dependencies via requirements.txt file
Networking
FAQ: Frequently asked questions
