Description of problem:
I'm trying to use the "systemctl enable" feature of systemd to bring up a container running a web service on system startup. After rebooting the host, the container starts up but doesn't seem to be responding to my web requests. After restarting the service it responds as expected again. Started and interrupted requests to that web service produce TCP CLOSE-WAIT connections.
Version-Release number of selected component (if applicable):
podman version 1.6.4
systemd 239
4.18.0-193.el8.x86_64
Red Hat Enterprise Linux release 8.2 (Ootpa)
How reproducible: easily
Steps to Reproduce:
# I create a container running httpd.
[root@standard system]# podman run -dit --name web1 -p 7000:8080 --volume /root/httpd/:/var/www/html/ registry.access.redhat.com/ubi8/httpd-24
ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00
# I curl it from the host and it responds.
[root@standard system]# curl localhost:7000/test1.txt
test file 1
# I create a service file [1] and enable it
[root@standard system]# pwd
/etc/systemd/system
[root@standard system]# podman generate systemd --files --name web1
/etc/systemd/system/container-web1.service
[root@standard system]# systemctl daemon-reload
[root@standard system]# systemctl is-enabled container-web1.service
enabled
# I reboot the machine, expecting the container to start and serve the website. The systemd service starts fine [2] but the httpd service doesn't respond.
[root@standard ~]# reboot
# curling this endpoint hangs. I waited up to 3 minutes and it stays like that, the command doesn't return the prompt.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
# While I'm waiting for the prompt to return, this is the output of ss.
[root@standard ~]# ss -n | grep 7000
tcp ESTAB 0 0 127.0.0.1:60552 127.0.0.1:7000
tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60552
# I stop the curl command with ctrl c and issue the same command again.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000
tcp ESTAB 0 0 127.0.0.1:60556 127.0.0.1:7000
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552
tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60556
# One more time.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp ESTAB 0 0 127.0.0.1:60560 127.0.0.1:7000
tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60560
tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000
tcp FIN-WAIT-2 0 0 127.0.0.1:60556 127.0.0.1:7000
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60556
# I interrupt curl with ctrl c again and do not run any more curl commands.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp FIN-WAIT-2 0 0 127.0.0.1:60560 127.0.0.1:7000
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60560
tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000
tcp FIN-WAIT-2 0 0 127.0.0.1:60556 127.0.0.1:7000
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552
tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60556
# I can restart the service and then it works as expected - I get responses to my requests.
[root@standard ~]# systemctl restart container-web1.service
[root@standard ~]# curl localhost:7000/test1.txt
test file 1
Actual results:
HTTP requests to a container that has started on system startup as a result of its service file being enabled seemingly don't trigger an HTTP response.
Expected results:
HTTP requests to a container that has started on system startup as a result of its service file being enabled produce an HTTP response.
Additional info:
[1]
[root@standard system]# systemctl cat container-web1.service
# /etc/systemd/system/container-web1.service
# container-web1.service
# autogenerated by Podman 1.6.4
# Mon May 30 16:42:10 CEST 2022
[Unit]
Description=Podman container-web1.service
Documentation=man:podman-generate-systemd(1)
[Service]
Restart=on-failure
ExecStart=/usr/bin/podman start web1
ExecStop=/usr/bin/podman stop -t 10 web1
KillMode=none
Type=forking
PIDFile=/var/run/containers/storage/overlay-containers/ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00/userdata/conmon.pid
[Install]
WantedBy=multi-user.target
[2]
[root@standard ~]# systemctl status container-web1.service --no-pager
● container-web1.service - Podman container-web1.service
Loaded: loaded (/etc/systemd/system/container-web1.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-05-30 16:54:53 CEST; 7min ago
Docs: man:podman-generate-systemd(1)
Process: 855 ExecStart=/usr/bin/podman start web1 (code=exited, status=0/SUCCESS)
Main PID: 1076 (conmon)
Tasks: 0 (limit: 11468)
Memory: 70.4M
CGroup: /system.slice/container-web1.service
‣ 1076 /usr/bin/conmon --api-version 1 -s -c ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 -u ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 …
May 30 16:54:51 standard systemd[1]: Starting Podman container-web1.service...
May 30 16:54:52 standard podman[855]: 2022-05-30 16:54:52.281898948 +0200 CEST m=+0.884322776 system refresh
May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.891709417 +0200 CEST m=+2.494133252 container init ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 …est, name=web1)
May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.933636308 +0200 CEST m=+2.536060139 container start ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00…est, name=web1)
May 30 16:54:53 standard podman[855]: web1
May 30 16:54:53 standard systemd[1]: Started Podman container-web1.service.
Hint: Some lines were ellipsized, use -l to show in full.
This issue was resolved upstream in podman 1.8.1 and the fix was included in RHEL 8.2 as part of podman-1.9.3-2.module+el8.2.1+6867+366c07d6. The podman-1.6.4 release is no longer maintained. For more information about container-tools module content availability please see https://access.redhat.com/support/policy/updates/containertools
The solution is to update to podman-1.9.3+ (preferably podman-4.0.2) and re-generate the container's systemd service. The new systemd unit file will contain the correct options to ensure your service/container is started after the network is available on your host.
Description of problem: I'm trying to use the "systemctl enable" feature of systemd to bring up a container running a web service on system startup. After rebooting the host, the container starts up but doesn't seem to be responding to my web requests. After restarting the service it responds as expected again. Started and interrupted requests to that web service produce TCP CLOSE-WAIT connections. Version-Release number of selected component (if applicable): podman version 1.6.4 systemd 239 4.18.0-193.el8.x86_64 Red Hat Enterprise Linux release 8.2 (Ootpa) How reproducible: easily Steps to Reproduce: # I create a container running httpd. [root@standard system]# podman run -dit --name web1 -p 7000:8080 --volume /root/httpd/:/var/www/html/ registry.access.redhat.com/ubi8/httpd-24 ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 # I curl it from the host and it responds. [root@standard system]# curl localhost:7000/test1.txt test file 1 # I create a service file [1] and enable it [root@standard system]# pwd /etc/systemd/system [root@standard system]# podman generate systemd --files --name web1 /etc/systemd/system/container-web1.service [root@standard system]# systemctl daemon-reload [root@standard system]# systemctl is-enabled container-web1.service enabled # I reboot the machine, expecting the container to start and serve the website. The systemd service starts fine [2] but the httpd service doesn't respond. [root@standard ~]# reboot # curling this endpoint hangs. I waited up to 3 minutes and it stays like that, the command doesn't return the prompt. [root@standard ~]# curl localhost:7000/test1.txt # in separate window # While I'm waiting for the prompt to return, this is the output of ss. [root@standard ~]# ss -n | grep 7000 tcp ESTAB 0 0 127.0.0.1:60552 127.0.0.1:7000 tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60552 # I stop the curl command with ctrl c and issue the same command again. [root@standard ~]# curl localhost:7000/test1.txt # in separate window [root@standard ~]# ss -n | grep 7000 tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000 tcp ESTAB 0 0 127.0.0.1:60556 127.0.0.1:7000 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552 tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60556 # One more time. [root@standard ~]# curl localhost:7000/test1.txt # in separate window [root@standard ~]# ss -n | grep 7000 tcp ESTAB 0 0 127.0.0.1:60560 127.0.0.1:7000 tcp ESTAB 87 0 127.0.0.1:7000 127.0.0.1:60560 tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000 tcp FIN-WAIT-2 0 0 127.0.0.1:60556 127.0.0.1:7000 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60556 # I interrupt curl with ctrl c again and do not run any more curl commands. [root@standard ~]# curl localhost:7000/test1.txt # in separate window [root@standard ~]# ss -n | grep 7000 tcp FIN-WAIT-2 0 0 127.0.0.1:60560 127.0.0.1:7000 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60560 tcp FIN-WAIT-2 0 0 127.0.0.1:60552 127.0.0.1:7000 tcp FIN-WAIT-2 0 0 127.0.0.1:60556 127.0.0.1:7000 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60552 tcp CLOSE-WAIT 88 0 127.0.0.1:7000 127.0.0.1:60556 # I can restart the service and then it works as expected - I get responses to my requests. [root@standard ~]# systemctl restart container-web1.service [root@standard ~]# curl localhost:7000/test1.txt test file 1 Actual results: HTTP requests to a container that has started on system startup as a result of its service file being enabled seemingly don't trigger an HTTP response. Expected results: HTTP requests to a container that has started on system startup as a result of its service file being enabled produce an HTTP response. Additional info: [1] [root@standard system]# systemctl cat container-web1.service # /etc/systemd/system/container-web1.service # container-web1.service # autogenerated by Podman 1.6.4 # Mon May 30 16:42:10 CEST 2022 [Unit] Description=Podman container-web1.service Documentation=man:podman-generate-systemd(1) [Service] Restart=on-failure ExecStart=/usr/bin/podman start web1 ExecStop=/usr/bin/podman stop -t 10 web1 KillMode=none Type=forking PIDFile=/var/run/containers/storage/overlay-containers/ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00/userdata/conmon.pid [Install] WantedBy=multi-user.target [2] [root@standard ~]# systemctl status container-web1.service --no-pager ● container-web1.service - Podman container-web1.service Loaded: loaded (/etc/systemd/system/container-web1.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2022-05-30 16:54:53 CEST; 7min ago Docs: man:podman-generate-systemd(1) Process: 855 ExecStart=/usr/bin/podman start web1 (code=exited, status=0/SUCCESS) Main PID: 1076 (conmon) Tasks: 0 (limit: 11468) Memory: 70.4M CGroup: /system.slice/container-web1.service ‣ 1076 /usr/bin/conmon --api-version 1 -s -c ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 -u ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 … May 30 16:54:51 standard systemd[1]: Starting Podman container-web1.service... May 30 16:54:52 standard podman[855]: 2022-05-30 16:54:52.281898948 +0200 CEST m=+0.884322776 system refresh May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.891709417 +0200 CEST m=+2.494133252 container init ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 …est, name=web1) May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.933636308 +0200 CEST m=+2.536060139 container start ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00…est, name=web1) May 30 16:54:53 standard podman[855]: web1 May 30 16:54:53 standard systemd[1]: Started Podman container-web1.service. Hint: Some lines were ellipsized, use -l to show in full.