Bug 1853455

Summary: podman ignores infra_command option from containers.conf
Product: Red Hat Enterprise Linux 8 Reporter: Derrick Ornelas <dornelas>
Component: podmanAssignee: Jindrich Novy <jnovy>
Status: CLOSED ERRATA QA Contact: Yuhui Jiang <yujiang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 8.2CC: bbaude, dornelas, dwalsh, jligon, jnovy, kanderso, lsm5, mheon, smccarty, tsweeney, ypu, yujiang
Target Milestone: rcKeywords: Triaged
Target Release: 8.3   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: podman-3.0 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 15:32:55 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1186913, 1765476, 1823899    

Description Derrick Ornelas 2020-07-02 17:58:47 UTC
Description of problem:

Podman appears to ignore the infra_command option when it is explicitly set in containers.conf


Version-Release number of selected component (if applicable):

podman-1.9.3-2.module+el8.2.1+6867+366c07d6


How reproducible: 100%


Steps to Reproduce:
1.  Copy default /usr/share/containers/containers.conf file to /etc/containers/containers.conf

  # cp /usr/share/containers/containers.conf /etc/containers/containers.conf


2.  Edit /etc/containers/containers.conf and set infra_image and infra_command as follows:

  infra_command = "/usr/bin/date"
  infra_image = "registry.access.redhat.com/ubi8-minimal:latest"


3.  Create a pod

  # podman pod create --name testpod


Actual results:

New infra container is configured to run image's default command instead of specified infra_command


Expected results:

New infra container is configured to run with specified infra_command


Additional info:


I was attempting to test using '/usr/bin/sleep infinity' to create an ad-hoc "pause" container, but I noticed that it wasn't using sleep at all for Cmd.  I then tested with a single binary/command.  Here's the full output from my reproducer:


# cp /usr/share/containers/containers.conf /etc/containers/containers.conf

# diff /usr/share/containers/containers.conf /etc/containers/containers.conf
293a294
> infra_command = "/usr/bin/date"
300a302
> infra_image = "registry.access.redhat.com/ubi8-minimal:latest"


# podman pod create --name testpod
57f86ca8a384e089de2405aad5a1b2edf225972b0a080ce22798660e365be9af

 podman ps -a
CONTAINER ID  IMAGE                                           COMMAND    CREATED         STATUS   PORTS  NAMES
e2b15a61cf1d  registry.access.redhat.com/ubi8-minimal:latest  /bin/bash  15 seconds ago  Created         57f86ca8a384-infra


# podman inspect 57f86ca8a384-infra | jq '.[].Config'
{
  "Hostname": "testpod",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": false,
  "AttachStderr": false,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "TERM=xterm",
    "container=oci"
  ],
  "Cmd": [
    "/bin/bash"
  ],
  "Image": "registry.access.redhat.com/ubi8-minimal:latest",
  "Volumes": null,
  "WorkingDir": "/",
  "Entrypoint": "",
  "OnBuild": null,
  "Labels": null,
  "Annotations": null,
  "StopSignal": 0
}

Comment 1 Daniel Walsh 2020-07-02 19:04:48 UTC
Works in podman 2.0/

Comment 2 Derrick Ornelas 2020-07-09 15:39:47 UTC
Was an upstream patch added recently to fix this?  This still doesn't appear to work with podman 2.0.2

# rpm -q podman containers-common
podman-2.0.2-1.module+el8.3.0+7303+7fef20f0.x86_64
containers-common-1.1.0-1.module+el8.3.0+7097+8d4f8cb4.x86_64


# diff /usr/share/containers/containers.conf /etc/containers/containers.conf 
293a294,295
> infra_command = "/usr/bin/date"
> infra_image = "registry.access.redhat.com/ubi8-minimal:latest"


# podman pod create --name testpod
f0d432545473f9149ef459741dc680d6b73238fe22d7007d4dda7944ac9b223d

# podman ps -a
CONTAINER ID  IMAGE                                           COMMAND    CREATED        STATUS   PORTS   NAMES
1dce7b722b94  registry.access.redhat.com/ubi8-minimal:latest  /bin/bash  6 seconds ago  Created          f0d432545473-infra


# podman inspect f0d432545473-infra | jq '.[].Config'
{
  "Hostname": "testpod",
  "Domainname": "",
  "User": "",
  "AttachStdin": false,
  "AttachStdout": false,
  "AttachStderr": false,
  "Tty": false,
  "OpenStdin": false,
  "StdinOnce": false,
  "Env": [
    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "TERM=xterm",
    "container=oci"
  ],
  "Cmd": [
    "/bin/bash"
  ],
  "Image": "registry.access.redhat.com/ubi8-minimal:latest",
  "Volumes": null,
  "WorkingDir": "/",
  "Entrypoint": "",
  "OnBuild": null,
  "Labels": null,
  "Annotations": null,
  "StopSignal": 15
}


# podman --log-level=debug pod start testpod
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level=debug pod start testpod) 
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/etc/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] host enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [nproc=4194304:4194304]  [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false  private k8s-file -1 bridge false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {false systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /var/run/libpod/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm   false 2048 runc map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false   [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/lib/containers/storage/libpod 10 /var/run/libpod /var/lib/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}} 
DEBU[0000] Reading configuration file "/etc/containers/containers.conf" 
DEBU[0000] Merged system config "/etc/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] host enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [nproc=4194304:4194304]  [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false  private k8s-file -1 bridge false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {false systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /var/run/libpod/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /usr/bin/date registry.access.redhat.com/ubi8-minimal:latest /usr/libexec/podman/catatonit shm   false 2048 runc map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false   [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/lib/containers/storage/libpod 10 /var/run/libpod /var/lib/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is being used 
DEBU[0000] cached value indicated that native-diff is not being used 
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
WARN[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 7              
DEBU[0000] Strongconnecting node 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 
DEBU[0000] Pushed 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 onto stack 
DEBU[0000] Finishing node 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87. Popped 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 off stack 
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/D46G2QQFUTHTZ4Q6DUBGLRV5XN:/var/lib/containers/storage/overlay/l/TC3FPGOLKWBBL4VLKNQVNTLIEQ,upperdir=/var/lib/containers/storage/overlay/be04e0021267140db5e2629c7119e9232c7c234d4f10deb101ed5569e53dbd21/diff,workdir=/var/lib/containers/storage/overlay/be04e0021267140db5e2629c7119e9232c7c234d4f10deb101ed5569e53dbd21/work,context="system_u:object_r:container_file_t:s0:c456,c667" 
DEBU[0000] mounted container "1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87" at "/var/lib/containers/storage/overlay/be04e0021267140db5e2629c7119e9232c7c234d4f10deb101ed5569e53dbd21/merged" 
DEBU[0000] Created root filesystem for container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 at /var/lib/containers/storage/overlay/be04e0021267140db5e2629c7119e9232c7c234d4f10deb101ed5569e53dbd21/merged 
DEBU[0000] Made network namespace at /var/run/netns/cni-5adf7490-14b3-6921-736a-fbbbb1718776 for container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 
INFO[0000] About to add CNI network lo (type=loopback)  
INFO[0000] Got pod network &{Name:testpod Namespace:testpod ID:1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 NetNS:/var/run/netns/cni-5adf7490-14b3-6921-736a-fbbbb1718776 Networks:[] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: &{0.4.0 [{Name:cni-podman0 Mac:c2:ef:d2:4c:86:27 Sandbox:} {Name:vetha18f425f Mac:f6:d0:00:4a:ab:0a Sandbox:} {Name:eth0 Mac:7e:80:8d:91:c6:8f Sandbox:/var/run/netns/cni-5adf7490-14b3-6921-736a-fbbbb1718776}] [{Version:4 Interface:0xc00035c928 Address:{IP:10.88.0.2 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}] {[]  [] []}} 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
WARN[0000] User mount overriding libpod mount at "/dev/shm" 
DEBU[0000] Setting CGroups for container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 to machine-libpod_pod_f0d432545473f9149ef459741dc680d6b73238fe22d7007d4dda7944ac9b223d.slice:libpod:1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 at /var/lib/containers/storage/overlay-containers/1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 -u 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87/userdata -p /var/run/containers/storage/overlay-containers/1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87/userdata/pidfile -n f0d432545473-infra --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket -s -l k8s-file:/var/lib/containers/storage/overlay-containers/1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87/userdata/conmon.pid]"
INFO[0000] Running conmon under slice machine-libpod_pod_f0d432545473f9149ef459741dc680d6b73238fe22d7007d4dda7944ac9b223d.slice and unitName libpod-conmon-1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87.scope 
DEBU[0000] Received: 41465                              
INFO[0000] Got Conmon PID as 41453                      
DEBU[0000] Created container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 in OCI runtime 
DEBU[0000] Starting container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 with command [/bin/sh -c /bin/bash] 
DEBU[0000] Started container 1dce7b722b94a5075e5d5a19608a3908999ad5e559c43bb41338aec510807b87 
f0d432545473f9149ef459741dc680d6b73238fe22d7007d4dda7944ac9b223d
DEBU[0000] Called start.PersistentPostRunE(podman --log-level=debug pod start testpod)

Comment 3 Tom Sweeney 2020-07-15 22:03:17 UTC
Dan any thoughts on Derrick's question in this comment: https://bugzilla.redhat.com/show_bug.cgi?id=1853455#c2?

Comment 4 Daniel Walsh 2020-07-16 18:45:19 UTC
Matt, this is still broken as far as I can see.

It looks like the code expects InfraCommand and InfraImage are to be treated as global options, but the CLI treats them as Pod Options.

I would have thought they were Pod Options, and was trying to get it to work, but this really needs you to look at it, to figure out 
is intended.

Comment 5 Matthew Heon 2020-07-16 19:03:36 UTC
It looks like we also have a related issue upstream - https://github.com/containers/podman/issues/6969

I think we may have broken infra command and image as part of the 2.0 migration - will take a look.

Comment 6 Daniel Walsh 2020-07-17 10:25:13 UTC
Yes I agree, we definitely broke it, I am not sure how to fix it.

Comment 7 Daniel Walsh 2020-09-13 10:58:50 UTC
Looks like the issue related to this has moved https://github.com/containers/podman/issues/7167

Comment 8 Daniel Walsh 2020-09-15 20:08:31 UTC
Fixed in https://github.com/containers/podman/pull/7621

Comment 9 Tom Sweeney 2020-09-16 17:09:43 UTC
Assigning to Jindrich for packaging needs once the PR noted in the prior comment is merged.

Comment 22 Tom Sweeney 2020-11-16 21:55:50 UTC
Setting this back to Assigned based on test status and follow up investigation.

Comment 24 Daniel Walsh 2020-11-20 15:09:47 UTC
Yes this will be fixed in rhel8.4.

Comment 27 Daniel Walsh 2021-01-28 11:56:16 UTC
Fixed in podman 3.0

Comment 40 errata-xmlrpc 2021-05-18 15:32:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: container-tools:rhel8 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:1796