Monitor Changes and Automatically Restart Containers with Docker Compose Watch
In this blog post, we will look at the Docker Compose watch option. This feature is an excellent tool for software developers, especially those working in a Docker environment.
Docker Compose is a great tool for defining and running multi-container Docker applications. It is also very useful tool if you want to setup your local development environment using Docker.
In my previous blog posts, I already covered about setting local development environment using docker compose part -I , part – II
One of the major pain point while using docker compose for testing applications is when ever the source code changes, we need to restart the container manually to deploy the new code. This results in spending more time on restarting container than the coding resulting in low productivity.
Now let’s look at the new option available in docker compose to overcome the manual restart of container when ever there is a change in sour code. This option can significantly enhance your development workflow by providing real-time feedback on your changes.
Docker Compose Watch Option
Starting with Docker Compose version 2.22.0, bundled with Docker Desktop 4.24, comes with new watch command option.
This allows you to monitor changes to your source code and based on your requirement you can do following actions.
sync
– syncs the source code from local system to containerrebuild
– automatically restart containers
Using Docker Compose Watch
I will explain how to use docker watch option with the help of spring boot project.
Using rebuild action
Let’s first look at the rebuild
action
Let’s first write Dockerfile
FROM eclipse-temurin:17.0.4_8-jdk-jammy
COPY ./target/*.jar /app.jar
RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot
USER sbuser
ENTRYPOINT ["java", "-jar", "/app.jar"]
Code language: Java (java)
Let’s write docker-compose file
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./target/springboot-resource-file-0.0.1-SNAPSHOT.jar
ports:
- 8080:8080
- 8000:8000
environment:
JAVA_OPTS:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Code language: Java (java)
In above compose file we are watching the springboot-resource-file-0.0.1-SNAPSHOT.jar for changes.
You can start the container and watch for changes with following command.
docker-compose watch
Code language: Java (java)
In below log you can see that container started and waiting for changes to springboot-resource-file-0.0.1-SNAPSHOT.jar
file
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.8s
watching [D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar]
Code language: Java (java)
Now If you change the source code and open another terminal run maven command to build the jar file
mvn package
Code language: Java (java)
Now if you look at the docker watch command log, when it recognizes change to jar file, it will rebuild docker image with new jar file and restarts the container
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 2/2
✔ Network springboot-resource-file_default Created 0.1s
✔ Container springboot-resource-file-springboot-app-1 Started 0.1s
watching [D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar]
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar
[+] Building 32.8s (11/11) FINISHED docker-container:multiarch
=> [springboot-app internal] booting buildkit 1.5s
=> => starting container buildx_buildkit_multiarch0 1.5s
=> [springboot-app internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 252B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 2.3s
=> [springboot-app auth] library/eclipse-temurin:pull token for registry-1.docker.io 0.0s
=> [springboot-app internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [springboot-app internal] load build context 1.9s
=> => transferring context: 19.04MB 1.9s
=> CACHED [springboot-app 1/3] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.0s
=> [springboot-app 2/3] COPY ./target/*.jar /app.jar 0.5s
=> [springboot-app 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.8s
=> [springboot-app] exporting to docker image format 24.6s
=> => exporting layers 1.6s
=> => exporting manifest sha256:44c931bec1e0d576241cd07e057980647523714cfbeb4b080b1e39b0b647178d 0.0s
=> => exporting config sha256:49a3bdb345025df95e65891ffd891aa8b91ff3a1ae15e3900aea53854e7273a2 0.0s
=> => sending tarball 22.9s
=> [springboot-app springboot-app] importing to docker 0.5s
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started
Code language: Java (java)
Note
2. Please do not use
mvn clean package
command. clean
options clears all the files from target folder and it takes some time to create jar file, mean while docker compose watch command tries to rebuild container as there changes to jar file but it will error out as it can not find the jar fileYou can also watch folder for changes
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./target
ports:
- 8080:8080
- 8000:8000
environment:
JAVA_OPTS:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Code language: Java (java)
Let’s start watch command
docker-compose watch
Code language: Java (java)
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.7s
watching [D:\IdeaProjects\springboot-resource-file\target]
Code language: Java (java)
Now build the package again with maven command
mvn package
Code language: Java (java)
From below console log, you can observe that watch option detected file changes and restarted containers multiple times.
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.8s
watching [D:\IdeaProjects\springboot-resource-file\target]
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\checkstyle-result.xml
[+] Building 27.9s (10/10) FINISHED docker-container:multiarch
=> [springboot-app internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 252B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 2.4s
=> [springboot-app auth] library/eclipse-temurin:pull token for registry-1.docker.io 0.0s
=> [springboot-app internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.1s
=> [springboot-app internal] load build context 0.1s
=> => transferring context: 100B 0.1s
=> [springboot-app 1/3] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> CACHED [springboot-app 2/3] COPY ./target/*.jar /app.jar 0.0s
=> CACHED [springboot-app 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.0s
=> [springboot-app] exporting to docker image format 24.9s
=> => exporting layers 0.0s
=> => exporting manifest sha256:44c931bec1e0d576241cd07e057980647523714cfbeb4b080b1e39b0b647178d 0.0s
=> => exporting config sha256:49a3bdb345025df95e65891ffd891aa8b91ff3a1ae15e3900aea53854e7273a2 0.0s
=> => sending tarball 24.8s
=> [springboot-app springboot-app] importing to docker 0.1s
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.8s
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\checkstyle-result.xml
[+] Building 27.5s (9/9) FINISHED docker-container:multiarch
=> [springboot-app internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 252B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 1.0s
=> [springboot-app internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [springboot-app internal] load build context 0.1s
=> => transferring context: 100B 0.1s
=> [springboot-app 1/3] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> CACHED [springboot-app 2/3] COPY ./target/*.jar /app.jar 0.0s
=> CACHED [springboot-app 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.0s
=> [springboot-app] exporting to docker image format 25.9s
=> => exporting layers 0.0s
=> => exporting manifest sha256:44c931bec1e0d576241cd07e057980647523714cfbeb4b080b1e39b0b647178d 0.0s
=> => exporting config sha256:49a3bdb345025df95e65891ffd891aa8b91ff3a1ae15e3900aea53854e7273a2 0.0s
=> => sending tarball 25.8s
=> [springboot-app springboot-app] importing to docker 0.1s
[+] Running 1/0
✔ Container springboot-resource-file-springboot-app-1 Running 0.0s
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\surefire-reports\2023-11-08T00-38-03_844.dumpstream
[+] Building 24.5s (9/9) FINISHED docker-container:multiarch
=> [springboot-app internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 252B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 0.9s
=> [springboot-app internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [springboot-app internal] load build context 0.1s
=> => transferring context: 100B 0.0s
=> [springboot-app 1/3] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.0s
=> CACHED [springboot-app 2/3] COPY ./target/*.jar /app.jar 0.0s
=> CACHED [springboot-app 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.0s
=> [springboot-app] exporting to docker image format 22.9s
=> => exporting layers 0.0s
=> => exporting manifest sha256:44c931bec1e0d576241cd07e057980647523714cfbeb4b080b1e39b0b647178d 0.0s
=> => exporting config sha256:49a3bdb345025df95e65891ffd891aa8b91ff3a1ae15e3900aea53854e7273a2 0.0s
=> => sending tarball 22.9s
=> [springboot-app springboot-app] importing to docker 0.2s
[+] Running 1/0
✔ Container springboot-resource-file-springboot-app-1 Running 0.0s
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\surefire-reports\2023-11-08T00-38-03_844.dumpstream
[+] Building 29.0s (9/9) FINISHED docker-container:multiarch
=> [springboot-app internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 252B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 1.0s
=> [springboot-app internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [springboot-app 1/3] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> [springboot-app internal] load build context 0.1s
=> => transferring context: 100B 0.0s
=> CACHED [springboot-app 2/3] COPY ./target/*.jar /app.jar 0.0s
=> CACHED [springboot-app 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.0s
=> [springboot-app] exporting to docker image format 27.4s
=> => exporting layers 0.0s
=> => exporting manifest sha256:44c931bec1e0d576241cd07e057980647523714cfbeb4b080b1e39b0b647178d 0.0s
=> => exporting config sha256:49a3bdb345025df95e65891ffd891aa8b91ff3a1ae15e3900aea53854e7273a2 0.0s
=> => sending tarball 27.4s
=> [springboot-app springboot-app] importing to docker 0.2s
[+] Running 1/0
✔ Container springboot-resource-file-springboot-app-1 Running 0.0s
Code language: Java (java)
Using Multi Stage Dockerfile
If you are using multistage dockerfile like below
FROM eclipse-temurin:17.0.4_8-jdk-jammy as build
ENV DOCKER_BUILDKIT=1
COPY mvnw ./
COPY .mvn .mvn
COPY pom.xml ./
COPY src src
RUN chmod a+rx mvnw
#RUN ./mvnw clean package -DskipTests
RUN --mount=type=cache,target=/root/.m2,rw ./mvnw package -DskipTests
FROM eclipse-temurin:17.0.4_8-jdk-jammy
COPY --from=build "./target/*.jar" /app.jar
RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot
USER sbuser
ENTRYPOINT ["java", "-jar", "/app.jar"]
Code language: Java (java)
You can watch for the changes in src folder and build the container
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./src/
ports:
- 8080:8080
- 8000:8000
environment:
JAVA_OPTS:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Code language: Java (java)
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 2/2
✔ Network springboot-qrcode-generator_default Created 0.1s
✔ Container springboot-qrcode-generator-springboot-app-1 Started 0.1s
watching [D:\IdeaProjects\springboot-qrcode-generator\src]
Code language: Java (java)
When change source code of any file
docker compose detects changes and starts containers again with updated jars.
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 2/2
✔ Network springboot-qrcode-generator_default Created 0.1s
✔ Container springboot-qrcode-generator-springboot-app-1 Started 0.1s
watching [D:\IdeaProjects\springboot-qrcode-generator\src]
[+] Building 43.8s (17/17) FINISHED docker-container:multiarch
=> [springboot-app internal] booting buildkit 1.0s
=> => starting container buildx_buildkit_multiarch0 1.0s
=> [springboot-app internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 534B 0.0s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 2.3s
=> [springboot-app auth] library/eclipse-temurin:pull token for registry-1.docker.io 0.0s
=> [springboot-app internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> CACHED [springboot-app build 1/7] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.0s
=> [springboot-app internal] load build context 0.1s
=> => transferring context: 4.73kB 0.0s
=> CACHED [springboot-app build 2/7] COPY mvnw ./ 0.0s
=> CACHED [springboot-app build 3/7] COPY .mvn .mvn 0.0s
=> CACHED [springboot-app build 4/7] COPY pom.xml ./ 0.0s
=> CACHED [springboot-app build 5/7] COPY src src 0.0s
=> [springboot-app build 6/7] RUN chmod a+rx mvnw 0.3s
=> [springboot-app build 7/7] RUN --mount=type=cache,target=/root/.m2,rw ./mvnw package -DskipTests 11.5s
=> [springboot-app stage-1 2/3] COPY --from=build ./target/*.jar /app.jar 0.3s
=> [springboot-app stage-1 3/3] RUN addgroup --system springboot && adduser --system sbuser && adduser sbuser springboot 0.7s
=> [springboot-app] exporting to docker image format 26.2s
=> => exporting layers 1.7s
=> => exporting manifest sha256:3c7b07e0fff0cd804923ab24a351c1d2e2c595541c39c95534d28a9533d6da2a 0.0s
=> => exporting config sha256:e729512cb9497920443727afdca92007d066a30a8cc59d5f0152fe12afc25082 0.0s
=> => sending tarball 24.4s
=> [springboot-app springboot-app] importing to docker 0.5s
[+] Running 1/1
✔ Container springboot-qrcode-generator-springboot-app-1 Started
Code language: Java (java)
Using sync action
Let’s look at the sync action of watch option, which copies the code from local system to container.
Sync action useful when copying the source code or properties file to container is enough to effect the changes without restarting the containers.
Sync is option is mostly used in when you containerize the applications based on JavaScript frameworks like React and Angular.
For Spring Boot projects, sync is useful when you use external properties file.
For example, you can watch for changes to directory and sync the changed files to container.
In the following docker compose file we are watching for changes in resource directory and copying the changed file /temp directory inside the container.
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./target/springboot-resource-file-0.0.1-SNAPSHOT.jar
- action: sync
path: ./src/main/resources
target: /tmp
ports:
- 8080:8080
- 8000:8000
environment:
JAVA_OPTS:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Code language: Java (java)
Start containers with watch option
docker-comppose watcch
Code language: Java (java)
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.8s
watching [D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar D:\IdeaProjects\springboot-resource-file\src\main\resources]
Code language: Java (java)
Now change content of application.properties file
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 0.8s
watching [D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar D:\IdeaProjects\springboot-resource-file\src\main\resources]
Syncing springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\src\main\resources\application.properties
Code language: Java (java)
Similarly you can watch for changes for single file for changes
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./target/springboot-resource-file-0.0.1-SNAPSHOT.jar
- action: sync
path: ./src/main/resources/application.properties
target: /tmp
ports:
- 8080:8080
- 8000:8000
environment:
JAVA_OPTS:
-Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n
Code language: Java (java)
Note
You can’t sync the files to root structure of the container just as copy files to container as in Dockerfile ( e.g. COPY src src
TroubleShooting
ERROR [springboot-app 2/2] COPY ./target/*.jar /app.jar
Rebuilding springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar
[+] Building 3.9s (7/7) FINISHED docker-container:multiarch
=> [springboot-app internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 253B 0.1s
=> [springboot-app internal] load metadata for docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy 2.9s
=> [springboot-app auth] library/eclipse-temurin:pull token for registry-1.docker.io 0.0s
=> [springboot-app internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> CACHED [springboot-app 1/2] FROM docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> => resolve docker.io/library/eclipse-temurin:17.0.4_8-jdk-jammy@sha256:94c7c78ff834c578719f892f1e19e0686a4688d093ad74e728e9af2203bd35b8 0.1s
=> [springboot-app internal] load build context 0.4s
=> => transferring context: 2B 0.3s
=> ERROR [springboot-app 2/2] COPY ./target/*.jar /app.jar 0.2s
------
> [springboot-app 2/2] COPY ./target/*.jar /app.jar:
------
Application failed to start after update
Code language: Java (java)
While docker compose watching the file/folder for changes, if you run mvn clean package
it will clear all files from target folder and tries to re-create all the build related files including the jar. Mean while docker compose tries to rebuild the container but can not find the jar as it was deleted by clear command and not yet created.
tar: application.properties: Cannot open: Permission denied
[+] Building 0.0s (0/0) docker-container:multiarch
[+] Running 1/1
✔ Container springboot-resource-file-springboot-app-1 Started 1.4s
watching [D:\IdeaProjects\springboot-resource-file\target\springboot-resource-file-0.0.1-SNAPSHOT.jar D:\IdeaProjects\springboot-resource-file\src\main\resources]
Syncing springboot-app after changes were detected:
- D:\IdeaProjects\springboot-resource-file\src\main\resources\application.properties
<tar: application.properties: Cannot open: Permission denied
8tar: Exiting with failure status due to previous errors
time="2023-11-12T00:46:49+13:00" level=warning msg="Error handling changed files for service springboot-app: 1 error occurred:\n\t* copying files to dc72b43ae493ad409ad49c84730cca5f7b2569c8368ce1e3e3d23189732ae834: exit code 2\n\n"
Code language: Java (java)
This error occurs when you are watching for sync action in docker compose file and do not specify the target option or target option does not point to existing directory structure in container.
You can not sync to container directly, you have specify the existing directory from the container
services:
springboot-app:
build:
context: .
dockerfile: ./Dockerfile
develop:
watch:
- action: rebuild
path: ./target/springboot-resource-file-0.0.1-SNAPSHOT.jar
- action: sync
path: ./src/main/resources/application.properties
target: .
Code language: Java (java)
cannot take exclusive lock for project “xxx”: process with PID yyyy is still running
After exiting docker compose watch command, if you try to run the watch command again you will see this error.
You have delete the lock files manually to free the watcher up for that project.
They should be located in the following directory for your respective OS
windows: %LOCALAPPDATA%
unix: /run/user/UID
macos: ~/Library/Application Support
plan 9: /tmp
The lock file names follows the below convention:docker-compose.<project-name>.pid