Created attachment 1804869 [details] libvirt.log 1:cgroup 1:qemu Description of problem: When we spin a VM on a Centos9 host, using either ubi:8 or ubi:9 imaged nova_libvirt container, libvirt is apparently using the wrong namespace [1]. Workaround is to enable cgroupsv1 on Centos9: grubby --update-kernel=/boot/vmlinuz-$(uname -r) \ --args="systemd.unified_cgroup_hierarchy=0" podman inspect nova_libvirt [2] Version-Release number of selected component (if applicable): libvirt-libs-7.5.0-1.el9.x86_64 libvirt-daemon-7.5.0-1.el9.x86_64 libvirt-daemon-driver-nwfilter-7.5.0-1.el9.x86_64 libvirt-daemon-driver-qemu-7.5.0-1.el9.x86_64 libvirt-daemon-driver-secret-7.5.0-1.el9.x86_64 libvirt-daemon-driver-storage-core-7.5.0-1.el9.x86_64 libvirt-daemon-driver-nodedev-7.5.0-1.el9.x86_64 libvirt-client-7.5.0-1.el9.x86_64 libvirt-daemon-config-nwfilter-7.5.0-1.el9.x86_64 How reproducible: All the time Steps to Reproduce: 1. Deploy tripleo-standalone with centos9 [a] 2. logs of the traceback in /var/log/containers/nova/nova-compute.log [a] https://docs.engineering.redhat.com/display/PRODCHAIN/Getting+Started+with+CentOS9+and+RDO Actual results: Unable to spin a VM with cgroupsv2 [1] Expected results: We shouldn't have to enable cgroupsv1 Additional info: [1] ~~~ 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [req-b0625165-ddfa-44ea-b0b0-aaae60144b5e 1ad7f11d2d69417396216fd1a8198e31 2c48ec91e20d4f1b83bef4a08e76338c - default default] [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] Failed to build and run instance: libvirt.libvirtError: unable to open '/sys/fs/cgroup/machine/qemu-2-instance-00000002.libvirt-qemu/': No such file or directory 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] Traceback (most recent call last): 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] self.driver.spawn(context, instance, image_meta, 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4188, in spawn 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] self._create_guest_with_network( 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7246, in _create_guest_with_network 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] self._cleanup( 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] self.force_reraise() 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] raise self.value 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7215, in _create_guest_with_network 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] guest = self._create_guest( 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7155, in _create_guest 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] guest.launch(pause=pause) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 162, in launch 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] LOG.exception('Error launching a defined domain with XML: %s', 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] self.force_reraise() 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] raise self.value 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 159, in launch 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] return self._domain.createWithFlags(flags) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] result = proxy_call(self._autowrap, f, *args, **kwargs) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] rv = execute(f, *args, **kwargs) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] six.reraise(c, e, tb) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] raise value 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] rv = meth(*args, **kwargs) 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1385, in createWithFlags 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] raise libvirtError('virDomainCreateWithFlags() failed') 2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] libvirt.libvirtError: unable to open '/sys/fs/cgroup/machine/qemu-2-instance-00000002.libvirt-qemu/': No such file or directory ~~~ [2] ~~~ [ { "Id": "77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7", "Created": "2021-07-23T13:14:19.395704989Z", "Path": "dumb-init", "Args": [ "--single-child", "--", "kolla_start" ], "State": { "OciVersion": "1.0.2-dev", "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 3209, "ConmonPid": 3206, "ExitCode": 0, "Error": "", "StartedAt": "2021-07-23T13:49:47.787263025Z", "FinishedAt": "0001-01-01T00:00:00Z", "Healthcheck": { "Status": "unhealthy", "FailingStreak": 4, "Log": [ { "Start": "2021-07-23T13:49:48.705574555Z", "End": "2021-07-23T13:49:49.134419538Z", "ExitCode": 1, "Output": "" }, { "Start": "2021-07-23T13:50:19.723073637Z", "End": "2021-07-23T13:50:19.888211284Z", "ExitCode": 1, "Output": "" }, { "Start": "2021-07-23T13:50:51.063414185Z", "End": "2021-07-23T13:50:51.138384572Z", "ExitCode": 1, "Output": "" }, { "Start": "2021-07-23T13:51:21.86194552Z", "End": "2021-07-23T13:51:21.926286773Z", "ExitCode": 1, "Output": "" } ] } }, "Image": "1f43b8f165c383258b4cb5e7fb06745eb70c38c72d2abe939dd6b656ee1d2608", "ImageName": "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20", "Rootfs": "", "Pod": "", "ResolvConfPath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/resolv.conf", "HostnamePath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/hostname", "HostsPath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/hosts", "StaticDir": "/var/lib/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata", "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/config.json", "OCIRuntime": "crun", "ConmonPidFile": "/var/run/nova_libvirt.pid", "PidFile": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/pidfile", "Name": "nova_libvirt", "RestartCount": 0, "Driver": "overlay", "MountLabel": "system_u:object_r:container_share_t:s0", "ProcessLabel": "system_u:system_r:spc_t:s0", "AppArmorProfile": "", "EffectiveCaps": [ "CAP_AUDIT_CONTROL", "CAP_AUDIT_READ", "CAP_AUDIT_WRITE", "CAP_BLOCK_SUSPEND", "CAP_BPF", "CAP_CHECKPOINT_RESTORE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_DAC_READ_SEARCH", "CAP_FOWNER", "CAP_FSETID", "CAP_IPC_LOCK", "CAP_IPC_OWNER", "CAP_KILL", "CAP_LEASE", "CAP_LINUX_IMMUTABLE", "CAP_MAC_ADMIN", "CAP_MAC_OVERRIDE", "CAP_MKNOD", "CAP_NET_ADMIN", "CAP_NET_BIND_SERVICE", "CAP_NET_BROADCAST", "CAP_NET_RAW", "CAP_PERFMON", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYSLOG", "CAP_SYS_ADMIN", "CAP_SYS_BOOT", "CAP_SYS_CHROOT", "CAP_SYS_MODULE", "CAP_SYS_NICE", "CAP_SYS_PACCT", "CAP_SYS_PTRACE", "CAP_SYS_RAWIO", "CAP_SYS_RESOURCE", "CAP_SYS_TIME", "CAP_SYS_TTY_CONFIG", "CAP_WAKE_ALARM" ], "BoundingCaps": [ "CAP_AUDIT_CONTROL", "CAP_AUDIT_READ", "CAP_AUDIT_WRITE", "CAP_BLOCK_SUSPEND", "CAP_BPF", "CAP_CHECKPOINT_RESTORE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_DAC_READ_SEARCH", "CAP_FOWNER", "CAP_FSETID", "CAP_IPC_LOCK", "CAP_IPC_OWNER", "CAP_KILL", "CAP_LEASE", "CAP_LINUX_IMMUTABLE", "CAP_MAC_ADMIN", "CAP_MAC_OVERRIDE", "CAP_MKNOD", "CAP_NET_ADMIN", "CAP_NET_BIND_SERVICE", "CAP_NET_BROADCAST", "CAP_NET_RAW", "CAP_PERFMON", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYSLOG", "CAP_SYS_ADMIN", "CAP_SYS_BOOT", "CAP_SYS_CHROOT", "CAP_SYS_MODULE", "CAP_SYS_NICE", "CAP_SYS_PACCT", "CAP_SYS_PTRACE", "CAP_SYS_RAWIO", "CAP_SYS_RESOURCE", "CAP_SYS_TIME", "CAP_SYS_TTY_CONFIG", "CAP_WAKE_ALARM" ], "ExecIDs": [], "GraphDriver": { "Name": "overlay", "Data": { "LowerDir": "/var/lib/containers/storage/overlay/168cfe1653ff8dee1c2f7241814b86456128a6f42cc4c7356223253868bb89eb/diff:/var/lib/containers/storage/overlay/124358034bc0b94e94a7db6c1a6f5e21e3b84cca56138ed1b75e82fa0903b59e/diff:/var/lib/containers/storage/overlay/23bf7e38c5fc0091f7629b646a3d19625d940b3c4cccf60e48d229c30f11716f/diff:/var/lib/containers/storage/overlay/3fd3345cad5976aaffa2b15de92fd7f4cea971b281c259ce1699989e8140002d/diff:/var/lib/containers/storage/overlay/463a5bdd7f34a606bb773a02a5fcacca9bd2ff4b17eb19bb1fbce12f75f2ca1a/diff:/var/lib/containers/storage/overlay/8f504e330821ef83a3eee0f194b603b7c49eaf53b89b601f6af7c753bd757a0a/diff:/var/lib/containers/storage/overlay/70c843edc9283e026538a181f8034de1c742c2932a3a9be5a8c8100aa4217d45/diff:/var/lib/containers/storage/overlay/821166fb38d952480578060f2276f81b0677d4d2a8699e2572e9d7e2e1451ebf/diff:/var/lib/containers/storage/overlay/d013ec051ce03124d045eb5ae7b18ed48d54e9afacdd7f2c173568870def2a52/diff:/var/lib/containers/storage/overlay/1ba55ed8919a44506837a7953d67610dd9ee2a663df95cc2d4dbb1d7fa3c4c1a/diff", "MergedDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/merged", "UpperDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/diff", "WorkDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/work" } }, "Mounts": [ { "Type": "bind", "Source": "/etc/pki/ca-trust/extracted", "Destination": "/etc/pki/ca-trust/extracted", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/run", "Destination": "/run", "Driver": "", "Mode": "", "Options": [ "nosuid", "nodev", "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/tripleo-config/ceph", "Destination": "/var/lib/kolla/config_files/src-ceph", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/cache/libvirt", "Destination": "/var/cache/libvirt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "shared" }, { "Type": "bind", "Source": "/etc/hosts", "Destination": "/etc/hosts", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt", "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/container-config-scripts/nova_libvirt_launcher.sh", "Destination": "/nova_libvirt_launcher.sh", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/pki/tls/certs/ca-bundle.crt", "Destination": "/etc/pki/tls/certs/ca-bundle.crt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/localtime", "Destination": "/etc/localtime", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/dev/log", "Destination": "/dev/log", "Driver": "", "Mode": "", "Options": [ "nosuid", "nodev", "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/sys/fs/selinux", "Destination": "/sys/fs/selinux", "Driver": "", "Mode": "", "Options": [ "noexec", "nosuid", "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/selinux/config", "Destination": "/etc/selinux/config", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/libvirt", "Destination": "/var/lib/libvirt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "shared" }, { "Type": "bind", "Source": "/var/lib/kolla/config_files/nova_libvirt.json", "Destination": "/var/lib/kolla/config_files/config.json", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/run/libvirt", "Destination": "/run/libvirt", "Driver": "", "Mode": "", "Options": [ "nosuid", "nodev", "rbind" ], "RW": true, "Propagation": "shared" }, { "Type": "bind", "Source": "/etc/puppet", "Destination": "/etc/puppet", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/sys/fs/cgroup", "Destination": "/sys/fs/cgroup", "Driver": "", "Mode": "", "Options": [ "noexec", "nosuid", "nodev", "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/log/containers/libvirt", "Destination": "/var/log/libvirt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/pki/ca-trust/source/anchors", "Destination": "/etc/pki/ca-trust/source/anchors", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/dev", "Destination": "/dev", "Driver": "", "Mode": "", "Options": [ "nosuid", "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/log/libvirt/qemu", "Destination": "/var/log/libvirt/qemu", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/pki/tls/cert.pem", "Destination": "/etc/pki/tls/cert.pem", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/libvirt", "Destination": "/etc/libvirt", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/etc/ssh/ssh_known_hosts", "Destination": "/etc/ssh/ssh_known_hosts", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/vhost_sockets", "Destination": "/var/lib/vhost_sockets", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/config-data/puppet-generated/nova_libvirt", "Destination": "/var/lib/kolla/config_files/src", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/lib/modules", "Destination": "/lib/modules", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/nova", "Destination": "/var/lib/nova", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "shared" } ], "Dependencies": [], "NetworkSettings": { "EndpointID": "", "Gateway": "", "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "", "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "" }, "ExitCommand": [ "/usr/bin/podman", "--root", "/var/lib/containers/storage", "--runroot", "/run/containers/storage", "--log-level", "warning", "--cgroup-manager", "systemd", "--tmpdir", "/run/libpod", "--runtime", "crun", "--storage-driver", "overlay", "--storage-opt", "overlay.mountopt=nodev,metacopy=on", "--events-backend", "journald", "container", "cleanup", "77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7" ], "Namespace": "", "IsInfra": false, "Config": { "Hostname": "standalone-0.localdomain", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM=xterm", "container=oci", "LANG=en_US.UTF-8", "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS", "TRIPLEO_CONFIG_HASH=3dbb540d164aaf80e4de40255f0d73f4", "HOME=/root", "HOSTNAME=standalone-0.localdomain" ], "Cmd": [ "kolla_start" ], "Image": "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20", "Volumes": null, "WorkingDir": "/", "Entrypoint": "dumb-init --single-child --", "OnBuild": null, "Labels": { "architecture": "x86_64", "batch": "20201127.1", "build-date": "2021-03-31T09:10:50.108261", "com.redhat.build-host": "cpt-1003.osbs.prod.upshift.rdu2.redhat.com", "com.redhat.component": "openstack-nova-libvirt-container", "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI", "config_data": "{'depends_on': ['tripleo_nova_virtlogd_wrapper.service'], 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': '3dbb540d164aaf80e4de40255f0d73f4'}, 'healthcheck': {'test': '/openstack/healthcheck libvirtd'}, 'image': '192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20', 'net': 'host', 'pid': 'host', 'pids_limit': 65536, 'privileged': True, 'restart': 'always', 'security_opt': ['label=level:s0', 'label=type:spc_t', 'label=filetype:container_share_t'], 'start_order': 1, 'ulimit': ['nofile=131072', 'nproc=126960'], 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/libvirt:/var/log/libvirt:z', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro', '/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro', '/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/run:/run', '/sys/fs/cgroup:/sys/fs/cgroup', '/etc/libvirt:/etc/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/libvirt:/var/lib/libvirt:shared', '/var/cache/libvirt:/var/cache/libvirt:shared', '/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro', '/var/lib/vhost_sockets:/var/lib/vhost_sockets', '/var/lib/nova:/var/lib/nova:shared', '/sys/fs/selinux:/sys/fs/selinux', '/etc/selinux/config:/etc/selinux/config:ro']}", "config_id": "tripleo_step3", "container_name": "nova_libvirt", "description": "Red Hat OpenStack Platform 17.0 nova-libvirt", "distribution-scope": "public", "io.buildah.version": "1.22.0-dev", "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.", "io.k8s.display-name": "Red Hat OpenStack Platform 17.0 nova-libvirt", "io.openshift.expose-services": "", "io.openshift.tags": "rhosp osp openstack osp-17.0", "maintainer": "OpenStack TripleO team", "managed_by": "tripleo_ansible", "name": "rhosp17/openstack-nova-libvirt", "release": "1159", "summary": "Red Hat OpenStack Platform 17.0 nova-libvirt", "tcib_managed": "true", "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi9/images/9.0.0alpha-1159", "vcs-ref": "ca3e3df61059593ee4d3ce5ecb0931fb4ce5b69b", "vcs-type": "git", "vendor": "Red Hat, Inc.", "version": "17.0.0" }, "Annotations": { "io.container.manager": "libpod", "io.kubernetes.cri-o.Created": "2021-07-23T13:14:19.395704989Z", "io.kubernetes.cri-o.TTY": "false", "io.podman.annotations.autoremove": "FALSE", "io.podman.annotations.init": "FALSE", "io.podman.annotations.label": "level:s0,label=type:spc_t,label=filetype:container_share_t", "io.podman.annotations.privileged": "TRUE", "io.podman.annotations.publish-all": "FALSE", "org.opencontainers.image.stopSignal": "15" }, "StopSignal": 15, "Healthcheck": { "Test": [ "CMD-SHELL", "/openstack/healthcheck libvirtd" ], "Interval": 30000000000, "Timeout": 30000000000, "Retries": 3 }, "CreateCommand": [ "podman", "container", "run", "--name", "nova_libvirt", "--conmon-pidfile", "/var/run/nova_libvirt.pid", "--detach=True", "--env", "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS", "--env", "TRIPLEO_CONFIG_HASH=3dbb540d164aaf80e4de40255f0d73f4", "--healthcheck-command", "/openstack/healthcheck libvirtd", "--label", "config_id=tripleo_step3", "--label", "container_name=nova_libvirt", "--label", "managed_by=tripleo_ansible", "--label", "config_data={'depends_on': ['tripleo_nova_virtlogd_wrapper.service'], 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': '3dbb540d164aaf80e4de40255f0d73f4'}, 'healthcheck': {'test': '/openstack/healthcheck libvirtd'}, 'image': '192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20', 'net': 'host', 'pid': 'host', 'pids_limit': 65536, 'privileged': True, 'restart': 'always', 'security_opt': ['label=level:s0', 'label=type:spc_t', 'label=filetype:container_share_t'], 'start_order': 1, 'ulimit': ['nofile=131072', 'nproc=126960'], 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/libvirt:/var/log/libvirt:z', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro', '/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro', '/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/run:/run', '/sys/fs/cgroup:/sys/fs/cgroup', '/etc/libvirt:/etc/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/libvirt:/var/lib/libvirt:shared', '/var/cache/libvirt:/var/cache/libvirt:shared', '/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro', '/var/lib/vhost_sockets:/var/lib/vhost_sockets', '/var/lib/nova:/var/lib/nova:shared', '/sys/fs/selinux:/sys/fs/selinux', '/etc/selinux/config:/etc/selinux/config:ro']}", "--log-driver", "k8s-file", "--log-opt", "path=/var/log/containers/stdouts/nova_libvirt.log", "--network", "host", "--pid", "host", "--pids-limit", "65536", "--privileged=True", "--security-opt", "label=level:s0", "--security-opt", "label=type:spc_t", "--security-opt", "label=filetype:container_share_t", "--ulimit", "nofile=131072", "--ulimit", "nproc=126960", "--volume", "/etc/hosts:/etc/hosts:ro", "--volume", "/etc/localtime:/etc/localtime:ro", "--volume", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "--volume", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "--volume", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "--volume", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "--volume", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "--volume", "/dev/log:/dev/log", "--volume", "/etc/puppet:/etc/puppet:ro", "--volume", "/var/log/containers/libvirt:/var/log/libvirt:z", "--volume", "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro", "--volume", "/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro", "--volume", "/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro", "--volume", "/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro", "--volume", "/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro", "--volume", "/lib/modules:/lib/modules:ro", "--volume", "/dev:/dev", "--volume", "/run:/run", "--volume", "/sys/fs/cgroup:/sys/fs/cgroup", "--volume", "/etc/libvirt:/etc/libvirt", "--volume", "/run/libvirt:/run/libvirt:shared", "--volume", "/var/lib/libvirt:/var/lib/libvirt:shared", "--volume", "/var/cache/libvirt:/var/cache/libvirt:shared", "--volume", "/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro", "--volume", "/var/lib/vhost_sockets:/var/lib/vhost_sockets", "--volume", "/var/lib/nova:/var/lib/nova:shared", "--volume", "/sys/fs/selinux:/sys/fs/selinux", "--volume", "/etc/selinux/config:/etc/selinux/config:ro", "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20" ], "Umask": "0022", "Timeout": 0, "StopTimeout": 10 }, "HostConfig": { "Binds": [ "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro,rprivate,rbind", "/run:/run:rw,rprivate,nosuid,nodev,rbind", "/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro,rprivate,rbind", "/var/cache/libvirt:/var/cache/libvirt:shared,rw,rbind", "/etc/hosts:/etc/hosts:ro,rprivate,rbind", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro,rprivate,rbind", "/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro,rprivate,rbind", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro,rprivate,rbind", "/etc/localtime:/etc/localtime:ro,rprivate,rbind", "/dev/log:/dev/log:rw,rprivate,nosuid,nodev,rbind", "/sys/fs/selinux:/sys/fs/selinux:rw,rprivate,noexec,nosuid,rbind", "/etc/selinux/config:/etc/selinux/config:ro,rprivate,rbind", "/var/lib/libvirt:/var/lib/libvirt:shared,rw,rbind", "/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro,rprivate,rbind", "/run/libvirt:/run/libvirt:shared,rw,nosuid,nodev,rbind", "/etc/puppet:/etc/puppet:ro,rprivate,rbind", "/sys/fs/cgroup:/sys/fs/cgroup:rw,rprivate,noexec,nosuid,nodev,rbind", "/var/log/containers/libvirt:/var/log/libvirt:rw,rprivate,rbind", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro,rprivate,rbind", "/dev:/dev:rw,rprivate,nosuid,rbind", "/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro,rprivate,rbind", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro,rprivate,rbind", "/etc/libvirt:/etc/libvirt:rw,rprivate,rbind", "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro,rprivate,rbind", "/var/lib/vhost_sockets:/var/lib/vhost_sockets:rw,rprivate,rbind", "/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro,rprivate,rbind", "/lib/modules:/lib/modules:ro,rprivate,rbind", "/var/lib/nova:/var/lib/nova:shared,rw,rbind" ], "CgroupManager": "systemd", "CgroupMode": "private", "ContainerIDFile": "", "LogConfig": { "Type": "k8s-file", "Config": null, "Path": "/var/log/containers/stdouts/nova_libvirt.log", "Tag": "", "Size": "0B" }, "NetworkMode": "host", "PortBindings": {}, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": [], "CapDrop": [], "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": [], "GroupAdd": [], "IpcMode": "private", "Cgroup": "", "Cgroups": "default", "Links": null, "OomScoreAdj": 0, "PidMode": "host", "Privileged": true, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "label=level:s0,label=type:spc_t,label=filetype:container_share_t" ], "Tmpfs": {}, "UTSMode": "private", "UsernsMode": "", "ShmSize": 65536000, "Runtime": "oci", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": 0, "OomKillDisable": false, "PidsLimit": 65536, "Ulimits": [ { "Name": "RLIMIT_NOFILE", "Soft": 131072, "Hard": 131072 }, { "Name": "RLIMIT_NPROC", "Soft": 126960, "Hard": 126960 } ], "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "CgroupConf": null } } ] ~~~
So I did some investigating and after I managed to reproduce the issue locally it turned out that this is not libvirt issue at all. When I tried to create cgroup manually inside the container and tried to move manually started QEMU process I got the same error from kernel: "No such file or directory". That meant this has to be related to the fact that we are running inside a container and a bit more reading lead me to this part of "man podman run": --cgroupns=mode Set the cgroup namespace mode for the container. • host: use the host's cgroup namespace inside the container. • container:id: join the namespace of the specified container. • private: create a new cgroup namespace. • ns:path: join the namespace at the specified path. If the host uses cgroups v1, the default is set to host. On cgroups v2, the default is private. When I tried starting the container with "--cgroupns=host" I was again able manually move the QEMU process to different cgroup and libvirt no longer failed with that error as well. Based on these findings moving to Nova as they need to use that option as well.
Thanks Pavel. Testing locally on c9 and upstream with CI on c8
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543