Bug 1985377

Summary: [rhel9][nova] libvirt containerized on ubi:9 or ubi:8 with centos9 host (cgroupsv2) is using wrong namespace
Product: Red Hat OpenStack Reporter: David Vallee Delisle <dvd>
Component: openstack-tripleo-heat-templatesAssignee: OSP Team <rhos-maint>
Status: CLOSED ERRATA QA Contact: Joe H. Rahme <jhakimra>
Severity: medium Docs Contact:
Priority: medium    
Version: 17.0 (Wallaby)CC: alifshit, bdobreli, dasmith, eglynn, jhakimra, jschluet, kchamart, mburns, mschuppe, phrdina, ratailor, sbauza, sgordon, virt-maint, vromanso
Target Milestone: betaKeywords: Patch, Triaged
Target Release: 17.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-tripleo-heat-templates-14.3.1-0.20220607161058.ced328c.el9ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-21 12:16:15 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
libvirt.log 1:cgroup 1:qemu none

Description David Vallee Delisle 2021-07-23 14:16:35 UTC 
Created attachment 1804869 [details]
libvirt.log 1:cgroup 1:qemu

Description of problem:
When we spin a VM on a Centos9 host, using either ubi:8 or ubi:9 imaged nova_libvirt container, libvirt is apparently using the wrong namespace [1].

Workaround is to enable cgroupsv1 on Centos9:
grubby --update-kernel=/boot/vmlinuz-$(uname -r) \
 --args="systemd.unified_cgroup_hierarchy=0"

podman inspect nova_libvirt [2]

Version-Release number of selected component (if applicable):
libvirt-libs-7.5.0-1.el9.x86_64
libvirt-daemon-7.5.0-1.el9.x86_64
libvirt-daemon-driver-nwfilter-7.5.0-1.el9.x86_64
libvirt-daemon-driver-qemu-7.5.0-1.el9.x86_64
libvirt-daemon-driver-secret-7.5.0-1.el9.x86_64
libvirt-daemon-driver-storage-core-7.5.0-1.el9.x86_64
libvirt-daemon-driver-nodedev-7.5.0-1.el9.x86_64
libvirt-client-7.5.0-1.el9.x86_64
libvirt-daemon-config-nwfilter-7.5.0-1.el9.x86_64


How reproducible:
All the time

Steps to Reproduce:
1. Deploy tripleo-standalone with centos9 [a]
2. logs of the traceback in /var/log/containers/nova/nova-compute.log

[a] https://docs.engineering.redhat.com/display/PRODCHAIN/Getting+Started+with+CentOS9+and+RDO

Actual results:
Unable to spin a VM with cgroupsv2 [1]

Expected results:
We shouldn't have to enable cgroupsv1


Additional info:


[1]
~~~
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [req-b0625165-ddfa-44ea-b0b0-aaae60144b5e 1ad7f11d2d69417396216fd1a8198e31 2c48ec91e20d4f1b83bef4a08e76338c - default default] [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] Failed to build and run instance: libvirt.libvirtError: unable to open '/sys/fs/cgroup/machine/qemu-2-instance-00000002.libvirt-qemu/': No such file or directory
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] Traceback (most recent call last):
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2409, in _build_and_run_instance
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     self.driver.spawn(context, instance, image_meta,
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4188, in spawn
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     self._create_guest_with_network(
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7246, in _create_guest_with_network
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     self._cleanup(
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     self.force_reraise()
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     raise self.value
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7215, in _create_guest_with_network
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     guest = self._create_guest(
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7155, in _create_guest
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     guest.launch(pause=pause)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 162, in launch
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     LOG.exception('Error launching a defined domain with XML: %s',
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     self.force_reraise()
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     raise self.value
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     return self._domain.createWithFlags(flags)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     result = proxy_call(self._autowrap, f, *args, **kwargs)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     rv = execute(f, *args, **kwargs)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     six.reraise(c, e, tb)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     raise value
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     rv = meth(*args, **kwargs)
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]   File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1385, in createWithFlags
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f]     raise libvirtError('virDomainCreateWithFlags() failed')
    2021-07-23 13:35:10.995 2 ERROR nova.compute.manager [instance: fd89d0f1-a4fe-45d1-a7c8-240929ab1d9f] libvirt.libvirtError: unable to open '/sys/fs/cgroup/machine/qemu-2-instance-00000002.libvirt-qemu/': No such file or directory
~~~

[2]
~~~
[
    {
        "Id": "77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7",
        "Created": "2021-07-23T13:14:19.395704989Z",
        "Path": "dumb-init",
        "Args": [
            "--single-child",
            "--",
            "kolla_start"
        ],
        "State": {
            "OciVersion": "1.0.2-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 3209,
            "ConmonPid": 3206,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-07-23T13:49:47.787263025Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Healthcheck": {
                "Status": "unhealthy",
                "FailingStreak": 4,
                "Log": [
                    {
                        "Start": "2021-07-23T13:49:48.705574555Z",
                        "End": "2021-07-23T13:49:49.134419538Z",
                        "ExitCode": 1,
                        "Output": ""
                    },
                    {
                        "Start": "2021-07-23T13:50:19.723073637Z",
                        "End": "2021-07-23T13:50:19.888211284Z",
                        "ExitCode": 1,
                        "Output": ""
                    },
                    {
                        "Start": "2021-07-23T13:50:51.063414185Z",
                        "End": "2021-07-23T13:50:51.138384572Z",
                        "ExitCode": 1,
                        "Output": ""
                    },
                    {
                        "Start": "2021-07-23T13:51:21.86194552Z",
                        "End": "2021-07-23T13:51:21.926286773Z",
                        "ExitCode": 1,
                        "Output": ""
                    }
                ]
            }
        },
        "Image": "1f43b8f165c383258b4cb5e7fb06745eb70c38c72d2abe939dd6b656ee1d2608",
        "ImageName": "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20",
        "Rootfs": "",
        "Pod": "",
        "ResolvConfPath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/resolv.conf",
        "HostnamePath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/hostname",
        "HostsPath": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/hosts",
        "StaticDir": "/var/lib/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata",
        "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/config.json",
        "OCIRuntime": "crun",
        "ConmonPidFile": "/var/run/nova_libvirt.pid",
        "PidFile": "/run/containers/storage/overlay-containers/77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7/userdata/pidfile",
        "Name": "nova_libvirt",
        "RestartCount": 0,
        "Driver": "overlay",
        "MountLabel": "system_u:object_r:container_share_t:s0",
        "ProcessLabel": "system_u:system_r:spc_t:s0",
        "AppArmorProfile": "",
        "EffectiveCaps": [
            "CAP_AUDIT_CONTROL",
            "CAP_AUDIT_READ",
            "CAP_AUDIT_WRITE",
            "CAP_BLOCK_SUSPEND",
            "CAP_BPF",
            "CAP_CHECKPOINT_RESTORE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_DAC_READ_SEARCH",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_IPC_LOCK",
            "CAP_IPC_OWNER",
            "CAP_KILL",
            "CAP_LEASE",
            "CAP_LINUX_IMMUTABLE",
            "CAP_MAC_ADMIN",
            "CAP_MAC_OVERRIDE",
            "CAP_MKNOD",
            "CAP_NET_ADMIN",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_BROADCAST",
            "CAP_NET_RAW",
            "CAP_PERFMON",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYSLOG",
            "CAP_SYS_ADMIN",
            "CAP_SYS_BOOT",
            "CAP_SYS_CHROOT",
            "CAP_SYS_MODULE",
            "CAP_SYS_NICE",
            "CAP_SYS_PACCT",
            "CAP_SYS_PTRACE",
            "CAP_SYS_RAWIO",
            "CAP_SYS_RESOURCE",
            "CAP_SYS_TIME",
            "CAP_SYS_TTY_CONFIG",
            "CAP_WAKE_ALARM"
        ],
        "BoundingCaps": [
            "CAP_AUDIT_CONTROL",
            "CAP_AUDIT_READ",
            "CAP_AUDIT_WRITE",
            "CAP_BLOCK_SUSPEND",
            "CAP_BPF",
            "CAP_CHECKPOINT_RESTORE",
            "CAP_CHOWN",
            "CAP_DAC_OVERRIDE",
            "CAP_DAC_READ_SEARCH",
            "CAP_FOWNER",
            "CAP_FSETID",
            "CAP_IPC_LOCK",
            "CAP_IPC_OWNER",
            "CAP_KILL",
            "CAP_LEASE",
            "CAP_LINUX_IMMUTABLE",
            "CAP_MAC_ADMIN",
            "CAP_MAC_OVERRIDE",
            "CAP_MKNOD",
            "CAP_NET_ADMIN",
            "CAP_NET_BIND_SERVICE",
            "CAP_NET_BROADCAST",
            "CAP_NET_RAW",
            "CAP_PERFMON",
            "CAP_SETFCAP",
            "CAP_SETGID",
            "CAP_SETPCAP",
            "CAP_SETUID",
            "CAP_SYSLOG",
            "CAP_SYS_ADMIN",
            "CAP_SYS_BOOT",
            "CAP_SYS_CHROOT",
            "CAP_SYS_MODULE",
            "CAP_SYS_NICE",
            "CAP_SYS_PACCT",
            "CAP_SYS_PTRACE",
            "CAP_SYS_RAWIO",
            "CAP_SYS_RESOURCE",
            "CAP_SYS_TIME",
            "CAP_SYS_TTY_CONFIG",
            "CAP_WAKE_ALARM"
        ],
        "ExecIDs": [],
        "GraphDriver": {
            "Name": "overlay",
            "Data": {
                "LowerDir": "/var/lib/containers/storage/overlay/168cfe1653ff8dee1c2f7241814b86456128a6f42cc4c7356223253868bb89eb/diff:/var/lib/containers/storage/overlay/124358034bc0b94e94a7db6c1a6f5e21e3b84cca56138ed1b75e82fa0903b59e/diff:/var/lib/containers/storage/overlay/23bf7e38c5fc0091f7629b646a3d19625d940b3c4cccf60e48d229c30f11716f/diff:/var/lib/containers/storage/overlay/3fd3345cad5976aaffa2b15de92fd7f4cea971b281c259ce1699989e8140002d/diff:/var/lib/containers/storage/overlay/463a5bdd7f34a606bb773a02a5fcacca9bd2ff4b17eb19bb1fbce12f75f2ca1a/diff:/var/lib/containers/storage/overlay/8f504e330821ef83a3eee0f194b603b7c49eaf53b89b601f6af7c753bd757a0a/diff:/var/lib/containers/storage/overlay/70c843edc9283e026538a181f8034de1c742c2932a3a9be5a8c8100aa4217d45/diff:/var/lib/containers/storage/overlay/821166fb38d952480578060f2276f81b0677d4d2a8699e2572e9d7e2e1451ebf/diff:/var/lib/containers/storage/overlay/d013ec051ce03124d045eb5ae7b18ed48d54e9afacdd7f2c173568870def2a52/diff:/var/lib/containers/storage/overlay/1ba55ed8919a44506837a7953d67610dd9ee2a663df95cc2d4dbb1d7fa3c4c1a/diff",
                "MergedDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/merged",
                "UpperDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/diff",
                "WorkDir": "/var/lib/containers/storage/overlay/731fb94d3aaff95145c9ba5e0dcf449e3e1e791ae9174d420a7223f14822e88d/work"
            }
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/etc/pki/ca-trust/extracted",
                "Destination": "/etc/pki/ca-trust/extracted",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/run",
                "Destination": "/run",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/tripleo-config/ceph",
                "Destination": "/var/lib/kolla/config_files/src-ceph",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/cache/libvirt",
                "Destination": "/var/cache/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/etc/hosts",
                "Destination": "/etc/hosts",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.trust.crt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/container-config-scripts/nova_libvirt_launcher.sh",
                "Destination": "/nova_libvirt_launcher.sh",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/pki/tls/certs/ca-bundle.crt",
                "Destination": "/etc/pki/tls/certs/ca-bundle.crt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/localtime",
                "Destination": "/etc/localtime",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/dev/log",
                "Destination": "/dev/log",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/sys/fs/selinux",
                "Destination": "/sys/fs/selinux",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "noexec",
                    "nosuid",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/selinux/config",
                "Destination": "/etc/selinux/config",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/libvirt",
                "Destination": "/var/lib/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/kolla/config_files/nova_libvirt.json",
                "Destination": "/var/lib/kolla/config_files/config.json",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/run/libvirt",
                "Destination": "/run/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "shared"
            },
            {
                "Type": "bind",
                "Source": "/etc/puppet",
                "Destination": "/etc/puppet",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/sys/fs/cgroup",
                "Destination": "/sys/fs/cgroup",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "noexec",
                    "nosuid",
                    "nodev",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/log/containers/libvirt",
                "Destination": "/var/log/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/pki/ca-trust/source/anchors",
                "Destination": "/etc/pki/ca-trust/source/anchors",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/dev",
                "Destination": "/dev",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "nosuid",
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/log/libvirt/qemu",
                "Destination": "/var/log/libvirt/qemu",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/pki/tls/cert.pem",
                "Destination": "/etc/pki/tls/cert.pem",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/libvirt",
                "Destination": "/etc/libvirt",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/ssh/ssh_known_hosts",
                "Destination": "/etc/ssh/ssh_known_hosts",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/vhost_sockets",
                "Destination": "/var/lib/vhost_sockets",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/config-data/puppet-generated/nova_libvirt",
                "Destination": "/var/lib/kolla/config_files/src",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/lib/modules",
                "Destination": "/lib/modules",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nova",
                "Destination": "/var/lib/nova",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "shared"
            }
        ],
        "Dependencies": [],
        "NetworkSettings": {
            "EndpointID": "",
            "Gateway": "",
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "",
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": ""
        },
        "ExitCommand": [
            "/usr/bin/podman",
            "--root",
            "/var/lib/containers/storage",
            "--runroot",
            "/run/containers/storage",
            "--log-level",
            "warning",
            "--cgroup-manager",
            "systemd",
            "--tmpdir",
            "/run/libpod",
            "--runtime",
            "crun",
            "--storage-driver",
            "overlay",
            "--storage-opt",
            "overlay.mountopt=nodev,metacopy=on",
            "--events-backend",
            "journald",
            "container",
            "cleanup",
            "77831a2fd2d82bc100dd9d809afaf1ea9e5fa4e4be591cf97b7f337bab8b6db7"
        ],
        "Namespace": "",
        "IsInfra": false,
        "Config": {
            "Hostname": "standalone-0.localdomain",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "TERM=xterm",
                "container=oci",
                "LANG=en_US.UTF-8",
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "TRIPLEO_CONFIG_HASH=3dbb540d164aaf80e4de40255f0d73f4",
                "HOME=/root",
                "HOSTNAME=standalone-0.localdomain"
            ],
            "Cmd": [
                "kolla_start"
            ],
            "Image": "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": "dumb-init --single-child --",
            "OnBuild": null,
            "Labels": {
                "architecture": "x86_64",
                "batch": "20201127.1",
                "build-date": "2021-03-31T09:10:50.108261",
                "com.redhat.build-host": "cpt-1003.osbs.prod.upshift.rdu2.redhat.com",
                "com.redhat.component": "openstack-nova-libvirt-container",
                "com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
                "config_data": "{'depends_on': ['tripleo_nova_virtlogd_wrapper.service'], 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': '3dbb540d164aaf80e4de40255f0d73f4'}, 'healthcheck': {'test': '/openstack/healthcheck libvirtd'}, 'image': '192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20', 'net': 'host', 'pid': 'host', 'pids_limit': 65536, 'privileged': True, 'restart': 'always', 'security_opt': ['label=level:s0', 'label=type:spc_t', 'label=filetype:container_share_t'], 'start_order': 1, 'ulimit': ['nofile=131072', 'nproc=126960'], 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/libvirt:/var/log/libvirt:z', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro', '/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro', '/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/run:/run', '/sys/fs/cgroup:/sys/fs/cgroup', '/etc/libvirt:/etc/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/libvirt:/var/lib/libvirt:shared', '/var/cache/libvirt:/var/cache/libvirt:shared', '/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro', '/var/lib/vhost_sockets:/var/lib/vhost_sockets', '/var/lib/nova:/var/lib/nova:shared', '/sys/fs/selinux:/sys/fs/selinux', '/etc/selinux/config:/etc/selinux/config:ro']}",
                "config_id": "tripleo_step3",
                "container_name": "nova_libvirt",
                "description": "Red Hat OpenStack Platform 17.0 nova-libvirt",
                "distribution-scope": "public",
                "io.buildah.version": "1.22.0-dev",
                "io.k8s.description": "The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
                "io.k8s.display-name": "Red Hat OpenStack Platform 17.0 nova-libvirt",
                "io.openshift.expose-services": "",
                "io.openshift.tags": "rhosp osp openstack osp-17.0",
                "maintainer": "OpenStack TripleO team",
                "managed_by": "tripleo_ansible",
                "name": "rhosp17/openstack-nova-libvirt",
                "release": "1159",
                "summary": "Red Hat OpenStack Platform 17.0 nova-libvirt",
                "tcib_managed": "true",
                "url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi9/images/9.0.0alpha-1159",
                "vcs-ref": "ca3e3df61059593ee4d3ce5ecb0931fb4ce5b69b",
                "vcs-type": "git",
                "vendor": "Red Hat, Inc.",
                "version": "17.0.0"
            },
            "Annotations": {
                "io.container.manager": "libpod",
                "io.kubernetes.cri-o.Created": "2021-07-23T13:14:19.395704989Z",
                "io.kubernetes.cri-o.TTY": "false",
                "io.podman.annotations.autoremove": "FALSE",
                "io.podman.annotations.init": "FALSE",
                "io.podman.annotations.label": "level:s0,label=type:spc_t,label=filetype:container_share_t",
                "io.podman.annotations.privileged": "TRUE",
                "io.podman.annotations.publish-all": "FALSE",
                "org.opencontainers.image.stopSignal": "15"
            },
            "StopSignal": 15,
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "/openstack/healthcheck libvirtd"
                ],
                "Interval": 30000000000,
                "Timeout": 30000000000,
                "Retries": 3
            },
            "CreateCommand": [
                "podman",
                "container",
                "run",
                "--name",
                "nova_libvirt",
                "--conmon-pidfile",
                "/var/run/nova_libvirt.pid",
                "--detach=True",
                "--env",
                "KOLLA_CONFIG_STRATEGY=COPY_ALWAYS",
                "--env",
                "TRIPLEO_CONFIG_HASH=3dbb540d164aaf80e4de40255f0d73f4",
                "--healthcheck-command",
                "/openstack/healthcheck libvirtd",
                "--label",
                "config_id=tripleo_step3",
                "--label",
                "container_name=nova_libvirt",
                "--label",
                "managed_by=tripleo_ansible",
                "--label",
                "config_data={'depends_on': ['tripleo_nova_virtlogd_wrapper.service'], 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': '3dbb540d164aaf80e4de40255f0d73f4'}, 'healthcheck': {'test': '/openstack/healthcheck libvirtd'}, 'image': '192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20', 'net': 'host', 'pid': 'host', 'pids_limit': 65536, 'privileged': True, 'restart': 'always', 'security_opt': ['label=level:s0', 'label=type:spc_t', 'label=filetype:container_share_t'], 'start_order': 1, 'ulimit': ['nofile=131072', 'nproc=126960'], 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/libvirt:/var/log/libvirt:z', '/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro', '/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro', '/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro', '/lib/modules:/lib/modules:ro', '/dev:/dev', '/run:/run', '/sys/fs/cgroup:/sys/fs/cgroup', '/etc/libvirt:/etc/libvirt', '/run/libvirt:/run/libvirt:shared', '/var/lib/libvirt:/var/lib/libvirt:shared', '/var/cache/libvirt:/var/cache/libvirt:shared', '/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro', '/var/lib/vhost_sockets:/var/lib/vhost_sockets', '/var/lib/nova:/var/lib/nova:shared', '/sys/fs/selinux:/sys/fs/selinux', '/etc/selinux/config:/etc/selinux/config:ro']}",
                "--log-driver",
                "k8s-file",
                "--log-opt",
                "path=/var/log/containers/stdouts/nova_libvirt.log",
                "--network",
                "host",
                "--pid",
                "host",
                "--pids-limit",
                "65536",
                "--privileged=True",
                "--security-opt",
                "label=level:s0",
                "--security-opt",
                "label=type:spc_t",
                "--security-opt",
                "label=filetype:container_share_t",
                "--ulimit",
                "nofile=131072",
                "--ulimit",
                "nproc=126960",
                "--volume",
                "/etc/hosts:/etc/hosts:ro",
                "--volume",
                "/etc/localtime:/etc/localtime:ro",
                "--volume",
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro",
                "--volume",
                "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro",
                "--volume",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro",
                "--volume",
                "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro",
                "--volume",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro",
                "--volume",
                "/dev/log:/dev/log",
                "--volume",
                "/etc/puppet:/etc/puppet:ro",
                "--volume",
                "/var/log/containers/libvirt:/var/log/libvirt:z",
                "--volume",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro",
                "--volume",
                "/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro",
                "--volume",
                "/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro",
                "--volume",
                "/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro",
                "--volume",
                "/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro",
                "--volume",
                "/lib/modules:/lib/modules:ro",
                "--volume",
                "/dev:/dev",
                "--volume",
                "/run:/run",
                "--volume",
                "/sys/fs/cgroup:/sys/fs/cgroup",
                "--volume",
                "/etc/libvirt:/etc/libvirt",
                "--volume",
                "/run/libvirt:/run/libvirt:shared",
                "--volume",
                "/var/lib/libvirt:/var/lib/libvirt:shared",
                "--volume",
                "/var/cache/libvirt:/var/cache/libvirt:shared",
                "--volume",
                "/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro",
                "--volume",
                "/var/lib/vhost_sockets:/var/lib/vhost_sockets",
                "--volume",
                "/var/lib/nova:/var/lib/nova:shared",
                "--volume",
                "/sys/fs/selinux:/sys/fs/selinux",
                "--volume",
                "/etc/selinux/config:/etc/selinux/config:ro",
                "192.168.24.2:8787/nova-df/openstack-nova-libvirt:rhos-17-rhel-9-2021-07-20"
            ],
            "Umask": "0022",
            "Timeout": 0,
            "StopTimeout": 10
        },
        "HostConfig": {
            "Binds": [
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro,rprivate,rbind",
                "/run:/run:rw,rprivate,nosuid,nodev,rbind",
                "/var/lib/tripleo-config/ceph:/var/lib/kolla/config_files/src-ceph:ro,rprivate,rbind",
                "/var/cache/libvirt:/var/cache/libvirt:shared,rw,rbind",
                "/etc/hosts:/etc/hosts:ro,rprivate,rbind",
                "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro,rprivate,rbind",
                "/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro,rprivate,rbind",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro,rprivate,rbind",
                "/etc/localtime:/etc/localtime:ro,rprivate,rbind",
                "/dev/log:/dev/log:rw,rprivate,nosuid,nodev,rbind",
                "/sys/fs/selinux:/sys/fs/selinux:rw,rprivate,noexec,nosuid,rbind",
                "/etc/selinux/config:/etc/selinux/config:ro,rprivate,rbind",
                "/var/lib/libvirt:/var/lib/libvirt:shared,rw,rbind",
                "/var/lib/kolla/config_files/nova_libvirt.json:/var/lib/kolla/config_files/config.json:ro,rprivate,rbind",
                "/run/libvirt:/run/libvirt:shared,rw,nosuid,nodev,rbind",
                "/etc/puppet:/etc/puppet:ro,rprivate,rbind",
                "/sys/fs/cgroup:/sys/fs/cgroup:rw,rprivate,noexec,nosuid,nodev,rbind",
                "/var/log/containers/libvirt:/var/log/libvirt:rw,rprivate,rbind",
                "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro,rprivate,rbind",
                "/dev:/dev:rw,rprivate,nosuid,rbind",
                "/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro,rprivate,rbind",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro,rprivate,rbind",
                "/etc/libvirt:/etc/libvirt:rw,rprivate,rbind",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro,rprivate,rbind",
                "/var/lib/vhost_sockets:/var/lib/vhost_sockets:rw,rprivate,rbind",
                "/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro,rprivate,rbind",
                "/lib/modules:/lib/modules:ro,rprivate,rbind",
                "/var/lib/nova:/var/lib/nova:shared,rw,rbind"
            ],
            "CgroupManager": "systemd",
            "CgroupMode": "private",
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "k8s-file",
                "Config": null,
                "Path": "/var/log/containers/stdouts/nova_libvirt.log",
                "Tag": "",
                "Size": "0B"
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": [],
            "CapDrop": [],
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [],
            "GroupAdd": [],
            "IpcMode": "private",
            "Cgroup": "",
            "Cgroups": "default",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "host",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "label=level:s0,label=type:spc_t,label=filetype:container_share_t"
            ],
            "Tmpfs": {},
            "UTSMode": "private",
            "UsernsMode": "",
            "ShmSize": 65536000,
            "Runtime": "oci",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": 0,
            "OomKillDisable": false,
            "PidsLimit": 65536,
            "Ulimits": [
                {
                    "Name": "RLIMIT_NOFILE",
                    "Soft": 131072,
                    "Hard": 131072
                },
                {
                    "Name": "RLIMIT_NPROC",
                    "Soft": 126960,
                    "Hard": 126960
                }
            ],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "CgroupConf": null
        }
    }
]
~~~
Comment 1 Pavel Hrdina 2021-07-27 07:28:15 UTC 
So I did some investigating and after I managed to reproduce the issue locally it turned out that this is not libvirt issue at all. When I tried to create cgroup manually inside the container and tried to move manually started QEMU process I got the same error from kernel: "No such file or directory".

That meant this has to be related to the fact that we are running inside a container and a bit more reading lead me to this part of "man podman run":

   --cgroupns=mode
       Set the cgroup namespace mode for the container.

              • host: use the host's cgroup namespace inside the container.

              • container:id: join the namespace of the specified container.

              • private: create a new cgroup namespace.

              • ns:path: join the namespace at the specified path.

       If the host uses cgroups v1, the default is set to host. On cgroups v2, the default is private.

When I tried starting the container with "--cgroupns=host" I was again able manually move the QEMU process to different cgroup and libvirt no longer failed with that error as well.

Based on these findings moving to Nova as they need to use that option as well.
Comment 2 David Vallee Delisle 2021-07-27 12:34:08 UTC 
Thanks Pavel.

Testing locally on c9 and upstream with CI on c8
Comment 13 errata-xmlrpc 2022-09-21 12:16:15 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2022:6543