Bug 1748214 - Error adding network: cannot convert: no valid IP addresses
Summary: Error adding network: cannot convert: no valid IP addresses
Keywords:
Status: CLOSED RAWHIDE
Alias: None
Product: Fedora
Classification: Fedora
Component: podman
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Matthew Heon
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-09-03 07:48 UTC by Lukas Slebodnik
Modified: 2019-09-13 10:25 UTC (History)
9 users (show)

Fixed In Version: containernetworking-plugins-0.8.2-0.3.dev.git23d5525.fc32
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-09-13 10:25:10 UTC
Type: Bug


Attachments (Terms of Use)

Description Lukas Slebodnik 2019-09-03 07:48:01 UTC
Description of problem:



Version-Release number of selected component (if applicable):
sh# rpm -q podman crun kernel NetworkManager
podman-1.5.2-0.42.dev.git099549b.fc32.x86_64
crun-0.8-1.fc31.x86_64
kernel-5.3.0-0.rc6.git2.1.fc32.x86_64
NetworkManager-1.20.0-3.fc32.x86_64

How reproducible:
Deterministic

Steps to Reproduce:
1. dnf install -y podman
2. podman run --rm fedora:30 cat /etc/os-release

Actual results:
sh# podman run --rm fedora:30 cat /etc/os-release
Trying to pull docker.io/library/fedora:30...
Getting image source signatures
Copying blob sha256:5a915a173fbc36dc8e1410afdd9de2b08f71efb226f8eb1ebcdc00a1acbced62
Copying config sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999
Writing manifest to image destination
Storing signatures
time="2019-09-03T03:22:16-04:00" level=error msg="Error adding network: cannot convert: no valid IP addresses"
time="2019-09-03T03:22:16-04:00" level=error msg="Error while adding to cni lo network: cannot convert: no valid IP addresses"
Error: error configuring network namespace for container f62f08385626aa23bbc3b051b5af839f951d4656aafedd5418e84de090ee6e15: cannot convert: no valid IP addresses

Expected results:
sh# podman run --rm fedora:30 cat /etc/os-release
NAME=Fedora
VERSION="30 (Container Image)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

Additional info:

Comment 1 Lukas Slebodnik 2019-09-03 07:51:22 UTC
I tried --privileged, croupsV1, croupsV2 but I haven't found a workaround yet.

debug output:
sh-5.0# mount -l
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=1988808k,nr_inodes=497202,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
/dev/mapper/fedora_kvm--07--guest11-root on / type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15263)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel)
/dev/vda1 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=400868k,mode=700)
tmpfs on /run/netns type tmpfs (rw,nosuid,nodev,seclabel,mode=755)

sh-5.0# podman images
REPOSITORY                 TAG   IMAGE ID       CREATED      SIZE
docker.io/library/fedora   30    e9ed59d2baf7   7 days ago   254 MB

sh-5.0# podman --log-level=debug run --rm fedora:30 cat /etc/os-release
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is being used 
DEBU[0000] cached value indicated that native-diff is not being used 
WARN[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime runc: no valid executable found for OCI runtime runc: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]docker.io/library/fedora:30" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] Got mounts: []                               
DEBU[0000] Got volumes: []                              
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Using bridge netmode                         
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea019c637999" 
DEBU[0000] exporting opaque data as blob "sha256:e9ed59d2baf72308f3a811ebc49ff3f4e0175abf40bf636bea0160759c637999" 
DEBU[0000] created container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has work directory "/var/lib/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320bb94005fd834ad526de8654c1a081/userdata" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has run directory "/var/run/containers/storage/overlay-containers/cd2faa2b2e18caf59a87e5da9c93320b194005fd834ad526de8654c1a081/userdata" 
DEBU[0000] New container created "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" has CgroupParent "machine.slice/libpod-cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1.scope" 
DEBU[0000] Not attaching to stdin
DEBU[0000] Made network namespace at /var/run/netns/cni-8452e019-4627-7e29-1340-33b7d0ce2a4a for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
INFO[0000] Got pod network &{Name:gracious_jennings Namespace:gracious_jennings ID:cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 NetNS:/var/run/netns/cni-8452e019-4627--1340-33b7d0ce2a4a Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
DEBU[0000] overlay: mount_data=nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/RBF7GCCMW2773TNPW5A3JNTEG7,upperdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ef9f759dfad20732a929c1ed7f85e7d024706d34bb/diff,workdir=/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e7d024706d34bb/work,context="system_u:object_rtainer_file_t:s0:c280,c797" 
DEBU[0000] mounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" at "/var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad20732a929c1ed7f85e4706d34bb/merged" 
DEBU[0000] Created root filesystem for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 at /var/lib/containers/storage/overlay/bee39ae5dd59c0e1a6ea3a1f9f759dfad2a929c1ed7f85e7d024706d34bb/merged 
ERRO[0000] Error adding network: cannot convert: no valid IP addresses 
ERRO[0000] Error while adding to cni lo network: cannot convert: no valid IP addresses 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081" 
DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
DEBU[0000] Cleaning up container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
DEBU[0000] Container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081 storage is already unmounted, skipping... 
ERRO[0000] error configuring network namespace for container cd2faa2b2e18caf59a87e5da9c93320b1867b94005fd834ad526de8654c1a081: cannot convert: no valid IP addresses

Comment 2 Lukas Slebodnik 2019-09-03 08:14:44 UTC
Old cgroups and moby-engine works without any problem.

sh# docker run --rm fedora:30 cat /etc/os-release
Unable to find image 'fedora:30' locally
30: Pulling from library/fedora
5a915a173fbc: Pull complete 
Digest: sha256:d8d53450cae00985f9dad54a3520944c59e64aa8f01d3be61988404e11c15973
Status: Downloaded newer image for fedora:30
NAME=Fedora
VERSION="30 (Container Image)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Container Image)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

Comment 3 Matthew Heon 2019-09-03 13:25:52 UTC
Can you get the output of `ifconfig` after a container is run? Also, `rpm -qa containernetworking-plugins`?

Does this node only have IPv6 addresses?

Comment 4 Lukas Slebodnik 2019-09-03 14:21:14 UTC
(In reply to Matthew Heon from comment #3)
> Can you get the output of `ifconfig` after a container is run? Also, `rpm
> -qa containernetworking-plugins`?
> 

sh-5.0# rpm -q containernetworking-plugins
containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64


> Does this node only have IPv6 addresses?

No, it has also IPv4.

Comment 5 Lukas Slebodnik 2019-09-03 14:24:13 UTC
hmm,
it work with containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31.x86_64

nvr in f31 is higher than in rawhide even though version in rawhide was built 9 days later.
But I assume git hash should help to distinguish which commit is newer.

Comment 6 Matthew Heon 2019-09-03 14:35:01 UTC
I see git485be65 as 0.8.1 - not 0.8.2

Might be that the new 0.8.2 release is broken?

Comment 7 Lukas Slebodnik 2019-09-03 14:39:48 UTC
(In reply to Matthew Heon from comment #6)
> I see git485be65 as 0.8.1 - not 0.8.2
> 

I can see following versions in koji
https://koji.fedoraproject.org/koji/buildinfo?buildID=1358195


containernetworking-plugins-0.8.1-7.1.dev.git485be65.fc32
containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31

containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32

You would need to ask Lokesh about details or check dist-git :-)

> Might be that the new 0.8.2 release is broken?
I would say so.

Do you need some help with reproducing?
Or you already managed to reproduce it yourself.

Comment 8 Matthew Heon 2019-09-03 15:21:26 UTC
Definitely looks like 485be65 is 0.8.1-7 and not 0.8.2... I wonder if the newer tag was to try and supercede a broken build?

Comment 10 Ed Santiago 2019-09-04 13:09:26 UTC
I'm now seeing this on my rawhide setup, containernetworking-plugins-0.8.2-0.1.dev.git7e68430.fc32.x86_64, podman version seems not to matter, at least among the subset (1.5.2-0.42.dev.git099549b.fc32, 1.5.2-0.46.dev.git1d8a940.fc32)

Comment 11 Dan Williams 2019-09-04 16:28:41 UTC
Technically a bug in loopback, but could be fixed in the base CNI types package too.  See https://github.com/containernetworking/plugins/issues/381#issuecomment-527979684 for root cause analysis.

Comment 12 Ed Santiago 2019-09-12 19:27:49 UTC
It seems to be fixed in rawhide, some time between 2019-09-11T16:53:54Z and 2019-09-12T08:58:41Z. Podman gating tests run at t0 failed with the usual "no valid IP" error; tests run at t1 (and three subsequent runs) passed.

Comment 13 Ed Santiago 2019-09-12 19:53:04 UTC
Fixed in containernetworking-plugins-0.8.2-0.3.dev.git23d5525.fc32.x86_64 [1,2]

  [1] https://bodhi.fedoraproject.org/updates/FEDORA-2019-320a481c5b
  [2] https://koji.fedoraproject.org/koji/buildinfo?buildID=1375969


Note You need to log in before you can comment on or make changes to this bug.