Bug 1748214

I tried --privileged, croupsV1, croupsV2 but I haven't found a workaround yet.

debug output:
sh-5.0# mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1988808k,nr_inodes=497202,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/fedora_kvm--07--guest11-root on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15263)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel)
/dev/vda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400868k,mode=700)
tmpfs on /run/netns type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

sh-5.0# podman images
REPOSITORY                 TAG   IMAGE ID       CREATED      SIZE
docker.io/library/fedora   30    e9ed59d2baf7   7 days ago   254 MB

sh-5.0# podman --log-level=debug run --rm fedora:30 cat /etc/os-release
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is being used 
DEBU[0000] cached value indicated that native-diff is not being used 
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]docker.io/library/fedora:30" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] Got mounts: []                               
DEBU[0000] Got volumes: []                              
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Using bridge netmode                         
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] created container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has work directory "/var/lib/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320bb94005fd834ad526de8654c1a081/userdata" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has run directory "/var/run/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320b194005fd834ad526de8654c1a081/userdata" 
DEBU[0000] New container created "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has CgroupParent "machine.slice/libpod-cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1.scope" 
DEBU[0000] Not attaching to stdin
DEBU[0000] Made network namespace at /var/run/netns/cni-8452e019-4627-7e29-1340-33b7d0ce2a4a for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
INFO[0000] Got pod network &{Name:gracious_jennings Namespace:gracious_jennings ID:cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 NetNS:/var/run/netns/cni-8452e019-4627--1340-33b7d0ce2a4a Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/RBF7GCCMW2773TNPW5A3JNTEG7,upperdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ef9f759dfad20732a929c1ed7f85e7d024706d34bb/diff,workdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e7d024706d34bb/work,context="system_u:object_rtainer_file_t:s0:c280,c797" 
DEBU[0000] mounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" at "/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e4706d34bb/merged" 
DEBU[0000] Created root filesystem for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 at /var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad2a929c1ed7f85e7d024706d34bb/merged 
ERRO[0000] Error adding network: cannot convert: no valid IP addresses 
ERRO[0000] Error while adding to cni lo network: cannot convert: no valid IP addresses 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
ERRO[0000] error configuring network namespace for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081: cannot convert: no valid IP addresses

Old cgroups and moby-engine works without any problem.

sh# docker run --rm fedora:30 cat /etc/os-release
Unable to find image 'fedora:30' locally
30: Pulling from library/fedora
5a915a173fbc: Pull complete 
Digest: sha256:d8d53450cae00985f9dad54a3520944c59e64aa8f01d3be61988404e11c15973
Status: Downloaded newer image for fedora:30
NAME=Fedora
VERSION="30 (Container Image)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

Can you get the output of `ifconfig` after a container is run? Also, `rpm -qa containernetworking-plugins`?

Does this node only have IPv6 addresses?

(In reply to Matthew Heon from comment #3)
> Can you get the output of `ifconfig` after a container is run? Also, `rpm
> -qa containernetworking-plugins`?
> 

sh-5.0# rpm -q containernetworking-plugins
containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64


> Does this node only have IPv6 addresses?

No, it has also IPv4.

hmm,
it work with containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31.x86_64

nvr in f31 is higher than in rawhide even though version in rawhide was built 9 days later.
But I assume git hash should help to distinguish which commit is newer.

I see git485be65 as 0.8.1 - not 0.8.2

Might be that the new 0.8.2 release is broken?

(In reply to Matthew Heon from comment #6)
> I see git485be65 as 0.8.1 - not 0.8.2
> 

I can see following versions in koji
https://koji.fedoraproject.org/koji/buildinfo?buildID=1358195


containernetworking-plugins-0.8.1-7.1.dev.git485be65.fc32
containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31

containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32

You would need to ask Lokesh about details or check dist-git :-)

> Might be that the new 0.8.2 release is broken?
I would say so.

Do you need some help with reproducing?
Or you already managed to reproduce it yourself.

Definitely looks like 485be65 is 0.8.1-7 and not 0.8.2... I wonder if the newer tag was to try and supercede a broken build?

I'm now seeing this on my rawhide setup, containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64, podman version seems not to matter, at least among the subset (1.5.2-0.42.dev.git099549b.fc32, 1.5.2-0.46.dev.git1d8a940.fc32)

Technically a bug in loopback, but could be fixed in the base CNI types package too.  See https://github.com/containernetworking/plugins/issues/381#issuecomment-527979684 for root cause analysis.

It seems to be fixed in rawhide, some time between 2019-09-11T16:53:54Z and 2019-09-12T08:58:41Z. Podman gating tests run at t0 failed with the usual "no valid IP" error; tests run at t1 (and three subsequent runs) passed.

Fixed in containernetworking-plugins-0.8.2-0.3.dev.git23d5525.fc32.x86_64 [1,2]

  [1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-320a481c5b
  [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1375969