Description of problem: Version-Release number of selected component (if applicable): sh# rpm -q podman crun kernel NetworkManager podman-1.5.2-0.42.dev.git099549b.fc32.x86_64 crun-0.8-1.fc31.x86_64 kernel-5.3.0-0.rc6.git2.1.fc32.x86_64 NetworkManager-1.20.0-3.fc32.x86_64 How reproducible: Deterministic Steps to Reproduce: 1. dnf install -y podman 2. podman run --rm fedora:30 cat /etc/os-release Actual results: sh# podman run --rm fedora:30 cat /etc/os-release Trying to pull docker.io/library/fedora:30... Getting image source signatures Copying blob sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62 Copying config sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999 Writing manifest to image destination Storing signatures time="2019-09-03T03:22:16-04:00" level=error msg="Error adding network: cannot convert: no valid IP addresses" time="2019-09-03T03:22:16-04:00" level=error msg="Error while adding to cni lo network: cannot convert: no valid IP addresses" Error: error configuring network namespace for container f62f08385626aa23bbc3b051b5af839f951d4656aafedd5418e84de090ee6e15: cannot convert: no valid IP addresses Expected results: sh# podman run --rm fedora:30 cat /etc/os-release NAME=Fedora VERSION="30 (Container Image)" ID=fedora VERSION_ID=30 VERSION_CODENAME="" PLATFORM_ID="platform:f30" PRETTY_NAME="Fedora 30 (Container Image)" ANSI_COLOR="0;34" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:30" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=30 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=30 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Container Image" VARIANT_ID=container Additional info:
I tried --privileged, croupsV1, croupsV2 but I haven't found a workaround yet. debug output: sh-5.0# mount -l sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1988808k,nr_inodes=497202,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) /dev/mapper/fedora_kvm--07--guest11-root on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15263) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel) /dev/vda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400868k,mode=700) tmpfs on /run/netns type tmpfs (rw,nosuid,nodev,seclabel,mode=755) sh-5.0# podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/library/fedora 30 e9ed59d2baf7 7 days ago 254 MB sh-5.0# podman --log-level=debug run --rm fedora:30 cat /etc/os-release DEBU[0000] using conmon: "/usr/libexec/podman/conmon" DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Set libpod namespace to "" DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] cached value indicated that overlay is supported DEBU[0000] cached value indicated that metacopy is being used DEBU[0000] cached value indicated that native-diff is not being used WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true DEBU[0000] Initializing event backend journald DEBU[0000] using runtime "/usr/bin/crun" WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]docker.io/library/fedora:30" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" DEBU[0000] Got mounts: [] DEBU[0000] Got volumes: [] DEBU[0000] No hostname set; container's hostname will default to runtime default DEBU[0000] Using bridge netmode DEBU[0000] created OCI spec and options for new container DEBU[0000] Allocated lock 0 for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" DEBU[0000] created container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has work directory "/var/lib/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320bb94005fd834ad526de8654c1a081/userdata" DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has run directory "/var/run/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320b194005fd834ad526de8654c1a081/userdata" DEBU[0000] New container created "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has CgroupParent "machine.slice/libpod-cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1.scope" DEBU[0000] Not attaching to stdin DEBU[0000] Made network namespace at /var/run/netns/cni-8452e019-4627-7e29-1340-33b7d0ce2a4a for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 INFO[0000] Got pod network &{Name:gracious_jennings Namespace:gracious_jennings ID:cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 NetNS:/var/run/netns/cni-8452e019-4627--1340-33b7d0ce2a4a Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} INFO[0000] About to add CNI network cni-loopback (type=loopback) DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/RBF7GCCMW2773TNPW5A3JNTEG7,upperdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ef9f759dfad20732a929c1ed7f85e7d024706d34bb/diff,workdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e7d024706d34bb/work,context="system_u:object_rtainer_file_t:s0:c280,c797" DEBU[0000] mounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" at "/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e4706d34bb/merged" DEBU[0000] Created root filesystem for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 at /var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad2a929c1ed7f85e7d024706d34bb/merged ERRO[0000] Error adding network: cannot convert: no valid IP addresses ERRO[0000] Error while adding to cni lo network: cannot convert: no valid IP addresses DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] unmounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 DEBU[0000] Network is already cleaned up, skipping... DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... ERRO[0000] error configuring network namespace for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081: cannot convert: no valid IP addresses
Old cgroups and moby-engine works without any problem. sh# docker run --rm fedora:30 cat /etc/os-release Unable to find image 'fedora:30' locally 30: Pulling from library/fedora 5a915a173fbc: Pull complete Digest: sha256:d8d53450cae00985f9dad54a3520944c59e64aa8f01d3be61988404e11c15973 Status: Downloaded newer image for fedora:30 NAME=Fedora VERSION="30 (Container Image)" ID=fedora VERSION_ID=30 VERSION_CODENAME="" PLATFORM_ID="platform:f30" PRETTY_NAME="Fedora 30 (Container Image)" ANSI_COLOR="0;34" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:30" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=30 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=30 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Container Image" VARIANT_ID=container
Can you get the output of `ifconfig` after a container is run? Also, `rpm -qa containernetworking-plugins`? Does this node only have IPv6 addresses?
(In reply to Matthew Heon from comment #3) > Can you get the output of `ifconfig` after a container is run? Also, `rpm > -qa containernetworking-plugins`? > sh-5.0# rpm -q containernetworking-plugins containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64 > Does this node only have IPv6 addresses? No, it has also IPv4.
hmm, it work with containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31.x86_64 nvr in f31 is higher than in rawhide even though version in rawhide was built 9 days later. But I assume git hash should help to distinguish which commit is newer.
I see git485be65 as 0.8.1 - not 0.8.2 Might be that the new 0.8.2 release is broken?
(In reply to Matthew Heon from comment #6) > I see git485be65 as 0.8.1 - not 0.8.2 > I can see following versions in koji https://koji.fedoraproject.org/koji/buildinfo?buildID=1358195 containernetworking-plugins-0.8.1-7.1.dev.git485be65.fc32 containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31 containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32 You would need to ask Lokesh about details or check dist-git :-) > Might be that the new 0.8.2 release is broken? I would say so. Do you need some help with reproducing? Or you already managed to reproduce it yourself.
Definitely looks like 485be65 is 0.8.1-7 and not 0.8.2... I wonder if the newer tag was to try and supercede a broken build?
I'm now seeing this on my rawhide setup, containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64, podman version seems not to matter, at least among the subset (1.5.2-0.42.dev.git099549b.fc32, 1.5.2-0.46.dev.git1d8a940.fc32)
Technically a bug in loopback, but could be fixed in the base CNI types package too. See https://github.com/containernetworking/plugins/issues/381#issuecomment-527979684 for root cause analysis.
It seems to be fixed in rawhide, some time between 2019-09-11T16:53:54Z and 2019-09-12T08:58:41Z. Podman gating tests run at t0 failed with the usual "no valid IP" error; tests run at t1 (and three subsequent runs) passed.
Fixed in containernetworking-plugins-0.8.2-0.3.dev.git23d5525.fc32.x86_64 [1,2] [1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-320a481c5b [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1375969