Description of problem: Unable to spawn a VM with huge pages. The VM spawns when removing the `hw:mem_page_size` property from the flavor. Flavor: ``` +----------------------------+-----------------------------------------------------------------------------------------+ | Field | Value | +----------------------------+-----------------------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 40 | | id | 100 | | name | nfv_qe_base_flavor | | os-flavor-access:is_public | True | | properties | hw:cpu_policy='dedicated', hw:emulator_threads_policy='share', hw:mem_page_size='large' | | ram | 8192 | | rxtx_factor | 1.0 | | swap | | | vcpus | 6 | +----------------------------+-----------------------------------------------------------------------------------------+ ``` Error from nova log: ``` : libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-04T02:21:20.797312Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on: info: QEMU waiting for connection on: disconnected:unix: /var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on 2023-02-04T02:21:21.459234Z qemu-kvm: -chardev socket,id=charnet1,path=/var/lib/vhost_sockets/vhu40962b01-97,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu40962b01-97,server=on 2023-02-04T02:21:22.469403Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/26-instance-0000005c for guest RAM: Permission denied 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest Traceback (most recent call last): 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags) 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs) 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs) 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 129, in execute 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest six.reraise(c, e, tb) 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest raise value 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 83, in tworker 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs) 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1409, in createWithFlags 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed') 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-04T02:21:20.797312Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest 2023-02-04T02:21:21.459234Z qemu-kvm: -chardev socket,id=charnet1,path=/var/lib/vhost_sockets/vhu40962b01-97,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu40962b01-97,server=on 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest 2023-02-04T02:21:22.469403Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/26-instance-0000005c for guest RAM: Permission denied 2023-02-04 02:21:24.572 2 ERROR nova.virt.libvirt.guest 2023-02-04 02:21:24.573 2 ERROR nova.virt.libvirt.driver [req-d5994861-eef6-4131-9991-a70ff4dc0ede da777dd854dc4ad0bffeecdc72bdd6e0 a10fd0c114594dec81e725b9ab51e7a1 - default default] [instance: 3f425565-a9e6-406d-9e72-290d9d22a35e] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-04T02:21:20.797312Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on ``` Error from qemu log: ``` [root@compute-0 log]# cat containers/libvirt/qemu/instance-0000005c.log [20/20] 2023-02-04 02:21:20.745+0000: starting up libvirt version: 8.10.0, package: 2.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2022-12-06-04:27:41, ), qemu version: 7.2.0qemu-kvm-7.2.0-5.el9, kernel: 5.14.0-244.el9.x86_64, hostname: compute-0.l$ caldomain LC_ALL=C \ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \ HOME=/var/lib/libvirt/qemu/domain-26-instance-0000005c \ XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-26-instance-0000005c/.local/share \ XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-26-instance-0000005c/.cache \ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-26-instance-0000005c/.config \ /usr/libexec/qemu-kvm \ -name guest=instance-0000005c,debug-threads=on \ -S \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-26-instance-0000005c/master-key.aes"}' \ -machine pc-q35-rhel9.0.0,usb=off,dump-guest-core=off \ -accel kvm \ -cpu Broadwell-IBRS,vme=on,ss=on,vmx=on,pdcm=on,f16c=on,rdrand=on,hypervisor=on,arat=on,tsc-adjust=on,umip=on,md-clear=on,stibp=on,arch-capabilities=on,ssbd=on,xsaveopt=on,pdpe1gb=on,abm=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,skip-l1dfl-vmentry=on,p schange-mc-no=on \ -m 8192 \ -overcommit mem-lock=off \ -smp 6,sockets=6,dies=1,cores=1,threads=1 \ -object '{"qom-type":"thread-context","id":"tc-ram-node0","node-affinity":[0]}' \ -object '{"qom-type":"memory-backend-file","id":"ram-node0","mem-path":"/dev/hugepages/libvirt/qemu/26-instance-0000005c","share":true,"prealloc":true,"size":8589934592,"host-nodes":[0],"policy":"bind","prealloc-context":"tc-ram-node0"}' \ -numa node,nodeid=0,cpus=0-5,memdev=ram-node0 \ -uuid 3f425565-a9e6-406d-9e72-290d9d22a35e \ -smbios 'type=1,manufacturer=Red Hat,product=OpenStack Compute,version=23.2.3-1.20230112160825.4f9a306.el9ost,serial=3f425565-a9e6-406d-9e72-290d9d22a35e,uuid=3f425565-a9e6-406d-9e72-290d9d22a35e,family=Virtual Machine' \ -no-user-config \ -nodefaults \ -chardev socket,id=charmonitor,fd=24,server=on,wait=off \ -mon chardev=charmonitor,id=monitor,mode=control \ -rtc base=utc,driftfix=slew \ -global kvm-pit.lost_tick_policy=delay \ -no-hpet \ -no-shutdown \ -boot strict=on \ -device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \ -device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \ -device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \ -device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \ -device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \ -device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \ -device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \ -device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \ -device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \ -device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \ -device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \ -device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \ -device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \ -device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \ -device '{"driver":"pcie-root-port","port":30,"chassis":15,"id":"pci.15","bus":"pcie.0","addr":"0x3.0x6"}' \ -device '{"driver":"pcie-root-port","port":31,"chassis":16,"id":"pci.16","bus":"pcie.0","addr":"0x3.0x7"}' \ -device '{"driver":"pcie-root-port","port":32,"chassis":17,"id":"pci.17","bus":"pcie.0","addr":"0x4"}' \ -device '{"driver":"pcie-pci-bridge","id":"pci.18","bus":"pci.1","addr":"0x0"}' \ -device '{"driver":"piix3-usb-uhci","id":"usb","bus":"pci.18","addr":"0x1"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/df5186b0228cccedd276f35d49ab183a25cf6eb8","node-name":"libvirt-3-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-3-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/3f425565-a9e6-406d-9e72-290d9d22a35e/disk","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \ -device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1,"write-cache":"on"}' \ -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/3f425565-a9e6-406d-9e72-290d9d22a35e/disk.config","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \ -blockdev '{"node-name":"libvirt-1-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \ -device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-1-format","id":"sata0-0-0","write-cache":"on"}' \ -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on \ -netdev '{"type":"vhost-user","chardev":"charnet0","id":"hostnet0"}' \ -device '{"driver":"virtio-net-pci","rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet0","id":"net0","mac":"fa:16:3e:f2:31:9b","bus":"pci.2","addr":"0x0"}' \ -chardev socket,id=charnet1,path=/var/lib/vhost_sockets/vhu40962b01-97,server=on \ -netdev '{"type":"vhost-user","chardev":"charnet1","id":"hostnet1"}' \ -device '{"driver":"virtio-net-pci","rx_queue_size":1024,"tx_queue_size":1024,"netdev":"hostnet1","id":"net1","mac":"fa:16:3e:bb:7f:87","bus":"pci.3","addr":"0x0"}' \ -add-fd set=0,fd=25,opaque=serial0-log \ -chardev pty,id=charserial0,logfile=/dev/fdset/0,logappend=on \ -device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \ -device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -vnc 10.10.120.147:0,audiodev=audio1 \ -device '{"driver":"virtio-vga","id":"video0","max_outputs":1,"bus":"pcie.0","addr":"0x1"}' \ -device '{"driver":"vfio-pci","host":"0000:06:0a.7","id":"hostdev0","bus":"pci.5","addr":"0x0"}' \ -device '{"driver":"vfio-pci","host":"0000:06:00.3","id":"hostdev1","bus":"pci.6","addr":"0x0"}' \ -device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.7","addr":"0x0"}' \ -object '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"}' \ -device '{"driver":"virtio-rng-pci","rng":"objrng0","id":"rng0","bus":"pci.8","addr":"0x0"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on 2023-02-04T02:21:20.797312Z qemu-kvm: -chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu2e5d92b6-e9,server=on 2023-02-04T02:21:21.459234Z qemu-kvm: -chardev socket,id=charnet1,path=/var/lib/vhost_sockets/vhu40962b01-97,server=on: info: QEMU waiting for connection on: disconnected:unix:/var/lib/vhost_sockets/vhu40962b01-97,server=on char device redirected to /dev/pts/1 (label charserial0) 2023-02-04T02:21:22.469403Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/26-instance-0000005c for guest RAM: Permission denied 2023-02-04 02:21:22.472+0000: shutting down, reason=failed ``` Version-Release number of selected component (if applicable): RHOS-17.1-RHEL-9-20230131.n.2 python3-novaclient-17.4.1-1.20220917060716.5ee4427.el9ost.noarch python3-nova-23.2.3-1.20230112160825.4f9a306.el9ost.noarch openstack-nova-common-23.2.3-1.20230112160825.4f9a306.el9ost.noarch qemu-img-7.2.0-5.el9.x86_64 ipxe-roms-qemu-20200823-9.git4bd064de.el9_0.noarch qemu-kvm-tools-7.2.0-5.el9.x86_64 qemu-kvm-common-7.2.0-5.el9.x86_64 qemu-kvm-block-rbd-7.2.0-5.el9.x86_64 qemu-kvm-device-display-virtio-gpu-7.2.0-5.el9.x86_64 qemu-kvm-device-display-virtio-gpu-pci-7.2.0-5.el9.x86_64 qemu-kvm-device-display-virtio-vga-7.2.0-5.el9.x86_64 qemu-kvm-device-usb-host-7.2.0-5.el9.x86_64 qemu-kvm-device-usb-redirect-7.2.0-5.el9.x86_64 qemu-pr-helper-7.2.0-5.el9.x86_64 qemu-kvm-docs-7.2.0-5.el9.x86_64 qemu-kvm-ui-opengl-7.2.0-5.el9.x86_64 qemu-kvm-ui-egl-headless-7.2.0-5.el9.x86_64 qemu-kvm-audio-pa-7.2.0-5.el9.x86_64 qemu-kvm-core-7.2.0-5.el9.x86_64 libvirt-libs-8.10.0-2.el9.x86_64 libvirt-client-8.10.0-2.el9.x86_64 libvirt-daemon-8.10.0-2.el9.x86_64 libvirt-daemon-driver-nwfilter-8.10.0-2.el9.x86_64 libvirt-daemon-driver-qemu-8.10.0-2.el9.x86_64 libvirt-daemon-driver-secret-8.10.0-2.el9.x86_64 libvirt-daemon-driver-storage-core-8.10.0-2.el9.x86_64 libvirt-daemon-driver-nodedev-8.10.0-2.el9.x86_64 python3-libvirt-8.10.0-1.el9.x86_64 openstack-nova-compute-23.2.3-1.20230112160825.4f9a306.el9ost.noarch openstack-nova-migration-23.2.3-1.20230112160825.4f9a306.el9ost.noarch libvirt-daemon-config-nwfilter-8.10.0-2.el9.x86_64 qemu-kvm-7.2.0-5.el9.x86_64 How reproducible: Always Steps to Reproduce: 1. Deploy cloud 2. Attempt to spawn a virtual machine with huge pages Actual results: VM fails to spawn. Expected results: VM spawns successfully. Additional info: I will attach sos logs.
We have also tried with SELinux disabled.
I haven't looked at the logs yet, but I can see this is filed against libvirt-8.10.0. There were some fixes that landed in libvirt-9.0.0 so if you could try to upgrade and see if the bug still persists?
Hi Michal, Sure, I will check this with a newer libvirt and report back.
I have patched the `nova-libvirt` container with the following RPMs, and I am still encountering the same issues: ``` libvirt-client-9.0.0-4.el9.x86_64.rpm libvirt-daemon-9.0.0-4.el9.x86_64.rpm libvirt-daemon-config-nwfilter-9.0.0-4.el9.x86_64.rpm libvirt-daemon-driver-nodedev-9.0.0-4.el9.x86_64.rpm libvirt-daemon-driver-nwfilter-9.0.0-4.el9.x86_64.rpm libvirt-daemon-driver-qemu-9.0.0-4.el9.x86_64.rpm libvirt-daemon-driver-secret-9.0.0-4.el9.x86_64.rpm libvirt-daemon-driver-storage-core-9.0.0-4.el9.x86_64.rpm libvirt-libs-9.0.0-4.el9.x86_64.rpm ``` Am I supposed to include additional RPMs or patch additional nova containers?
(In reply to Vadim Khitrin from comment #7) > I have patched the `nova-libvirt` container with the following RPMs, and I > am still encountering the same issues: > ``` > libvirt-client-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-config-nwfilter-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-driver-nodedev-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-driver-nwfilter-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-driver-qemu-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-driver-secret-9.0.0-4.el9.x86_64.rpm > libvirt-daemon-driver-storage-core-9.0.0-4.el9.x86_64.rpm > libvirt-libs-9.0.0-4.el9.x86_64.rpm > ``` > Am I supposed to include additional RPMs or patch additional nova containers? OSP 17.1 comes with split libvirt daemons. So you _might_ have to patch the `nova_virtqemud` container as well. Can you please also apply the RPMs in that container, restart the daemons, and repeat the test? (I'm not 100% of an expert in the container interactions here; just trying to think out loud.)
Hey Kashyap, Every idea you have is appreciated! I have patched the `nova-libvirt` container, and the `nova_virtqemud` container is using it: ``` [root@compute-0 nova]# podman images | grep nova-libvirt undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt 17.1_20230130.1-hotfix d02dfb2d13c0 22 hours ago 1.77 GB [root@compute-0 nova]# podman ps | grep nova-libvirt 3f8c479aa014 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtlogd_wrapper 4bdeeb2c3276 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix /usr/sbin/virtlog... 17 hours ago Up 17 hours ago nova_virtlogd 5416bef37f91 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtsecretd 500eabf6b837 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtnodedevd 2494a88ece20 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtstoraged 5bdcb68e9820 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtqemud 757992829ba0 undercloud-0.ctlplane.localdomain:8787/rh-osbs/rhosp17-openstack-nova-libvirt:17.1_20230130.1-hotfix kolla_start 17 hours ago Up 17 hours ago nova_virtproxyd ``` RPMs inside the container are updated: ``` [root@compute-0 nova]# podman exec -it nova_virtqemud /usr/bin/rpm -qa | grep 'libvirt' python3-libvirt-8.10.0-1.el9.x86_64 libvirt-libs-9.0.0-4.el9.x86_64 libvirt-client-9.0.0-4.el9.x86_64 libvirt-daemon-9.0.0-4.el9.x86_64 libvirt-daemon-driver-nwfilter-9.0.0-4.el9.x86_64 libvirt-daemon-config-nwfilter-9.0.0-4.el9.x86_64 libvirt-daemon-driver-nodedev-9.0.0-4.el9.x86_64 libvirt-daemon-driver-qemu-9.0.0-4.el9.x86_64 libvirt-daemon-driver-secret-9.0.0-4.el9.x86_64 libvirt-daemon-driver-storage-core-9.0.0-4.el9.x86_64 ``` Looking at the container startup log, we can see that libvirt 9.0.0 is used: ``` [root@compute-0 ~]# podman logs nova_virtqemud + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc/libvirt/passwd.db INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/passwd.db to /etc/libvirt/passwd.db INFO:__main__:Deleting /etc/libvirt/qemu.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/qemu.conf to /etc/libvirt/qemu.conf INFO:__main__:Deleting /etc/libvirt/virtlogd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtlogd.conf to /etc/libvirt/virtlogd.conf INFO:__main__:Deleting /etc/libvirt/virtnodedevd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtnodedevd.conf to /etc/libvirt/virtnodedevd.conf INFO:__main__:Deleting /etc/libvirt/virtproxyd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtproxyd.conf to /etc/libvirt/virtproxyd.conf INFO:__main__:Deleting /etc/libvirt/virtqemud.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtqemud.conf to /etc/libvirt/virtqemud.conf INFO:__main__:Deleting /etc/libvirt/virtsecretd.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtsecretd.conf to /etc/libvirt/virtsecretd.conf INFO:__main__:Deleting /etc/libvirt/virtstoraged.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/libvirt/virtstoraged.conf to /etc/libvirt/virtstoraged.conf INFO:__main__:Deleting /etc/nova/migration/authorized_keys INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/authorized_keys to /etc/nova/migration/authorized_keys INFO:__main__:Deleting /etc/nova/migration/identity INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/migration/identity to /etc/nova/migration/identity INFO:__main__:Creating directory /etc/nova/provider_config INFO:__main__:Deleting /etc/nova/nova.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/nova/nova.conf to /etc/nova/nova.conf INFO:__main__:Deleting /etc/sasl2/libvirt.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/sasl2/libvirt.conf to /etc/sasl2/libvirt.conf INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/ssh/sshd_config to /etc/ssh/sshd_config INFO:__main__:Deleting /etc/login.defs INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/login.defs to /etc/login.defs INFO:__main__:Deleting /var/lib/nova/.ssh/config INFO:__main__:Copying /var/lib/kolla/config_files/src/var/lib/nova/.ssh/config to /var/lib/nova/.ssh/config INFO:__main__:Writing out command to execute ++ cat /run_command + CMD='/usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf' + ARGS= + [[ ! -n '' ]] + . kolla_extend_start + echo 'Running command: '\''/usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf'\''' Running command: '/usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf' + umask 0027 + exec /usr/sbin/virtqemud --config /etc/libvirt/virtqemud.conf 2023-02-15 00:53:00.583+0000: 24056: info : libvirt version: 9.0.0, package: 4.el9 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2023-02-09-05:54:47, ) 2023-02-15 00:53:00.583+0000: 24056: info : hostname: compute-0 2023-02-15 00:53:00.583+0000: 24056: debug : virLogParseOutputs:1641 : outputs=1:file:/var/log/libvirt/virtqemud.log 2023-02-15 00:53:00.583+0000: 24056: debug : virLogParseOutput:1488 : output=1:file:/var/log/libvirt/virtqemud.log ```
Thank you all for all the information. In the end, I believe this is just another instance of bug 2166979. Long story short, virtqemud is started with umask of 0027 which then gets in the way libvirt sets up hugepages. In this particular case, libvirtd/virtqemud creates $hugetlbfs/libvirt/qemu/ (/dev/hugepages/libvirt/qemu/ in this case) which is meant as top level directory, a base where individual, per-guest directories are then created. Now, the top level directory is owned by root:root (so that nobody else can modify its perms) but we also want it to be accessible by other processes (mind you, we don't need to list the directory (o+r), we need to be able to access subdirs (o+x)). Now, while libvirt can certainly work around this problem, I'm not entirely convinced this is a libvirt bug. Setting such restrictive umask for a process that is supposed to prepare environment for other users (QEMU runs under qemu:qemu) is weird decision. At any rate, this is a duplicate of aforementioned bug 2166979. (see bug 2166979#c27 for more detailed analysis) I need somebody from kolla? team to explain the decision, what the aim was in order to decide how to move forward.
*** This bug has been marked as a duplicate of bug 2166979 ***
*** Bug 2166979 has been marked as a duplicate of this bug. ***
FTR https://bugzilla.redhat.com/show_bug.cgi?id=2166979#c30
*** Bug 2166968 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.1 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2023:4577