TIL: Background Processes and Docker containers
Recently I have set up Apache Airflow in Docker containers. Apart from the container with Apache Airflow backend database server (running PostgreSQL), there are two containers running Apache Airflow Webserver and Apache Airflow Scheduler.
The first problem that was encountered is the inability to view task logs through Web UI (while Apache Airflow is configured with LocalExecutor). There is a relevant issue on github repo puckel/docker-airflow. The solution was to attach a volume to each container so that in each container Airflow logs directory is pointed at the same place on the host machine.
The second problem is clean up of old logs.