RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2091721 - systemd managed container doesn't start serving web traffic despite of starting on system startup.
Summary: systemd managed container doesn't start serving web traffic despite of starti...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.2
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-30 20:54 UTC by janfishergg
Modified: 2022-06-13 21:34 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 2096449 (view as bug list)
Environment:
Last Closed: 2022-06-13 21:34:27 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-123694 0 None None None 2022-05-30 21:03:04 UTC

Description janfishergg 2022-05-30 20:54:07 UTC 
Description of problem:
I'm trying to use the "systemctl enable" feature of systemd to bring up a container running a web service on system startup. After rebooting the host, the container starts up but doesn't seem to be responding to my web requests. After restarting the service it responds as expected again. Started and interrupted requests to that web service produce TCP CLOSE-WAIT connections.

Version-Release number of selected component (if applicable):
podman version 1.6.4
systemd 239
4.18.0-193.el8.x86_64
Red Hat Enterprise Linux release 8.2 (Ootpa)


How reproducible: easily


Steps to Reproduce:
# I create a container running httpd.
[root@standard system]# podman run -dit --name web1 -p 7000:8080 --volume /root/httpd/:/var/www/html/ registry.access.redhat.com/ubi8/httpd-24
ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00

# I curl it from the host and it responds.
[root@standard system]# curl localhost:7000/test1.txt
test file 1

# I create a service file [1] and enable it
[root@standard system]# pwd
/etc/systemd/system
[root@standard system]# podman generate systemd --files --name web1
/etc/systemd/system/container-web1.service
[root@standard system]# systemctl daemon-reload
[root@standard system]# systemctl is-enabled container-web1.service 
enabled



# I reboot the machine, expecting the container to start and serve the website. The systemd service starts fine [2] but the httpd service doesn't respond.
[root@standard ~]# reboot

# curling this endpoint hangs. I waited up to 3 minutes and it stays like that, the command doesn't return the prompt.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window

# While I'm waiting for the prompt to return, this is the output of ss.
[root@standard ~]# ss -n | grep 7000
tcp                ESTAB               0                    0                                                                127.0.0.1:60552     127.0.0.1:7000 
tcp                ESTAB               87                   0                                                                127.0.0.1:7000      127.0.0.1:60552

# I stop the curl command with ctrl c and issue the same command again.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60552     127.0.0.1:7000 
tcp               ESTAB                  0                   0                                                               127.0.0.1:60556     127.0.0.1:7000 
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60552
tcp               ESTAB                  87                  0                                                               127.0.0.1:7000      127.0.0.1:60556

# One more time.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp               ESTAB                  0                   0                                                               127.0.0.1:60560     127.0.0.1:7000 
tcp               ESTAB                  87                  0                                                               127.0.0.1:7000      127.0.0.1:60560
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60552     127.0.0.1:7000 
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60556     127.0.0.1:7000 
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60552
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60556

# I interrupt curl with ctrl c again and do not run any more curl commands.
[root@standard ~]# curl localhost:7000/test1.txt # in separate window
[root@standard ~]# ss -n | grep 7000
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60560     127.0.0.1:7000 
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60560
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60552     127.0.0.1:7000 
tcp               FIN-WAIT-2             0                   0                                                               127.0.0.1:60556     127.0.0.1:7000 
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60552
tcp               CLOSE-WAIT             88                  0                                                               127.0.0.1:7000      127.0.0.1:60556

# I can restart the service and then it works as expected - I get responses to my requests.
[root@standard ~]# systemctl restart container-web1.service
[root@standard ~]# curl localhost:7000/test1.txt
test file 1

Actual results:
HTTP requests to a container that has started on system startup as a result of its service file being enabled seemingly don't trigger an HTTP response.

Expected results:
HTTP requests to a container that has started on system startup as a result of its service file being enabled produce an HTTP response.

Additional info:
[1]
[root@standard system]# systemctl cat container-web1.service 
# /etc/systemd/system/container-web1.service
# container-web1.service
# autogenerated by Podman 1.6.4
# Mon May 30 16:42:10 CEST 2022

[Unit]
Description=Podman container-web1.service
Documentation=man:podman-generate-systemd(1)

[Service]
Restart=on-failure
ExecStart=/usr/bin/podman start web1
ExecStop=/usr/bin/podman stop -t 10 web1
KillMode=none
Type=forking
PIDFile=/var/run/containers/storage/overlay-containers/ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00/userdata/conmon.pid

[Install]
WantedBy=multi-user.target


[2]
[root@standard ~]# systemctl status container-web1.service --no-pager 
● container-web1.service - Podman container-web1.service
   Loaded: loaded (/etc/systemd/system/container-web1.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-05-30 16:54:53 CEST; 7min ago
     Docs: man:podman-generate-systemd(1)
  Process: 855 ExecStart=/usr/bin/podman start web1 (code=exited, status=0/SUCCESS)
 Main PID: 1076 (conmon)
    Tasks: 0 (limit: 11468)
   Memory: 70.4M
   CGroup: /system.slice/container-web1.service
           ‣ 1076 /usr/bin/conmon --api-version 1 -s -c ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 -u ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 …

May 30 16:54:51 standard systemd[1]: Starting Podman container-web1.service...
May 30 16:54:52 standard podman[855]: 2022-05-30 16:54:52.281898948 +0200 CEST m=+0.884322776 system refresh
May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.891709417 +0200 CEST m=+2.494133252 container init ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00 …est, name=web1)
May 30 16:54:53 standard podman[855]: 2022-05-30 16:54:53.933636308 +0200 CEST m=+2.536060139 container start ad7aa8adf5ebd7d41ec97c2645c1829afe0309484e2effcb3e66d8523de66a00…est, name=web1)
May 30 16:54:53 standard podman[855]: web1
May 30 16:54:53 standard systemd[1]: Started Podman container-web1.service.
Hint: Some lines were ellipsized, use -l to show in full.
Comment 10 Derrick Ornelas 2022-06-13 21:34:27 UTC 
This issue was resolved upstream in podman 1.8.1 and the fix was included in RHEL 8.2 as part of podman-1.9.3-2.module+el8.2.1+6867+366c07d6. The podman-1.6.4 release is no longer maintained. For more information about container-tools module content availability please see https://access.redhat.com/support/policy/updates/containertools

The solution is to update to podman-1.9.3+ (preferably podman-4.0.2) and re-generate the container's systemd service.  The new systemd unit file will contain the correct options to ensure your service/container is started after the network is available on your host.
Note You need to log in before you can comment on or make changes to this bug.