Bug 2167527
| Summary: | Cannot launch domain with CPU pinning and emulator threads pinning: "qemu-kvm: Setting CPU affinity failed: Invalid argument" | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | James Parker <jparker> |
| Component: | libvirt | Assignee: | Michal Privoznik <mprivozn> |
| libvirt sub component: | General | QA Contact: | Luyao Huang <lhuang> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | high | ||
| Priority: | high | CC: | alifshit, bgibizer, chayang, dasmith, eglynn, jhakimra, jinzhao, juzhang, juzhou, kchamart, lhuang, mprivozn, sbauza, sgordon, virt-maint, vromanso, yafu, zhguo |
| Version: | 9.2 | Keywords: | Triaged |
| Target Milestone: | beta | Flags: | pm-rhel:
mirror+
|
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-02-21 09:52:44 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Comment 6
Michal Privoznik
2023-02-10 15:06:45 UTC
Thanks Michal, I will update the environment and try and get some 9.0.0 results early next week. Michal - so it doesn't get lost/forgotten - I'll assign to you. We can change this if it's determined that it's not a duplicate of bug 2154750. Reproduce this bug on libvirt-8.10.0-2.el9:
0. prepare a host which numa node > 1
# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 31503 MB
node 0 free: 29115 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 32244 MB
node 1 free: 24570 MB
node distances:
node 0 1
0: 10 32
1: 32 10
1. prepare a guest which have emulatorpin + numatune
# virsh dumpxml vm1
<cputune>
<emulatorpin cpuset='16-20'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
2.
# virsh start vm1
error: Failed to start domain 'vm1'
error: internal error: process exited while connecting to monitor: 2023-02-16T04:24:21.584878Z qemu-kvm: Setting CPU affinity failed: Invalid argument
(In reply to John Ferlan from comment #8) > Michal - so it doesn't get lost/forgotten - I'll assign to you. We can > change this if it's determined that it's not a duplicate of bug 2154750. Indeed it is. I'll wait for James to retry with libvirt-9.0.0 and close this afterwards. Cannot reproduce this bug on libvirt-9.0.0-5.el9.x86_64:
0. prepare a host which numa node > 1
# numactl --hard
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 31503 MB
node 0 free: 23993 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 32244 MB
node 1 free: 22261 MB
node distances:
node 0 1
0: 10 32
1: 32 10
1. prepare a guest which have emulatorpin + numatune
# virsh dumpxml vm1
<cputune>
<emulatorpin cpuset='16-20'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
</numatune>
2.
# virsh start vm1
Domain 'vm1' started
I can confirm the issue is no longer reproducible in my environment with libvirt-9.0.0-5.el9. Hi Michal and James,
I cannot reproduce this issue with the latest libvirt-9.0.0-6.el9.x86_64 when I create a new VM with NUMA configuration, but I failed to create a new VM with NUMA+Hugepages configuration, hope the following message can help you.
Package version:[OSP compute node]
libvirt-daemon-9.0.0-6.el9.x86_64
qemu-kvm-7.2.0-9.el9.x86_64
openstack-nova-compute-23.2.3-1.20230208100952.797c5fa.el9ost.noarch
OSP puddle tag:17.1_20230216.1
Test result:
1. Succeed to create a new VM on the OSP server only with NUMA configured.
(overcloud) [stack@dell-per740-66 ~]$ openstack server create --flavor m1.mixed --image r9-q --nic net-id=85bb7932-efb4-417b-beda-7ee4b4eab002 vm_numa
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | vm-numa |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-vjg8aeqe |
| OS-EXT-SRV-ATTR:root_device_name | None |
| OS-EXT-SRV-ATTR:user_data | None |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | BniH42QN2923 |
| config_drive | |
| created | 2023-02-21T01:51:24Z |
| description | None |
| flavor | disk='10', ephemeral='0', extra_specs.hw:cpu_policy='dedicated', extra_specs.hw:cpu_thread_policy='prefer', extra_specs.hw:numa_cpus.0='0', extra_specs.hw:numa_cpus.1='1', extra_specs.hw:numa_mem.0='1024', extra_specs.hw:numa_mem.1='1024', extra_specs.hw:numa_mempolicy='strict', extra_specs.hw:numa_nodes='2', original_name='m1.mixed', ram='2048', swap='0', vcpus='2' |
| hostId | |
| host_status | |
| id | 70ef225c-8360-444c-948f-40ef39ba83dc |
| image | r9-q (e061a2c3-a176-4a3f-a3d0-591a3911c91f) |
| key_name | None |
| locked | False |
| locked_reason | None |
| name | vm_numa |
| progress | 0 |
| project_id | d44934edcd904399b994d2411d99e662 |
| properties | |
| security_groups | name='default' |
| server_groups | [] |
| status | BUILD |
| tags | |
| trusted_image_certificates | None |
| updated | 2023-02-21T01:51:24Z |
| user_id | 06883cd41346460cae0e28b0f5f9455b |
| volumes_attached | |
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@dell-per740-66 ~]$ openstack server list
+--------------------------------------+---------+--------+-----------------------+-------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+---------+--------+-----------------------+-------+----------+
| 70ef225c-8360-444c-948f-40ef39ba83dc | vm_numa | ACTIVE | private=192.168.24.73 | r9-q | m1.mixed |
+--------------------------------------+---------+--------+-----------------------+-------+----------+
[root@compute-0 /]# virsh dumpxml instance-00000005
...
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='28'/>
<vcpupin vcpu='1' cpuset='29'/>
<emulatorpin cpuset='28-29'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
...
2. Failed to create a new VM on the OSP server with NUMA+Hugepages configured.
(overcloud) [stack@dell-per740-66 ~]$ openstack server create --flavor m2.mixed --image r9-q --nic net-id=85bb7932-efb4-417b-beda-7ee4b4eab002 vm_numa_hugepage
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | vm-numa-hugepage |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-6dw85pxh |
| OS-EXT-SRV-ATTR:root_device_name | None |
| OS-EXT-SRV-ATTR:user_data | None |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | SkYawESBG4ow |
| config_drive | |
| created | 2023-02-21T01:52:07Z |
| description | None |
| flavor | disk='10', ephemeral='0', extra_specs.hw:cpu_policy='dedicated', extra_specs.hw:cpu_thread_policy='prefer', extra_specs.hw:mem_page_size='1GB', extra_specs.hw:numa_cpus.0='0', extra_specs.hw:numa_cpus.1='1', extra_specs.hw:numa_mem.0='1024', extra_specs.hw:numa_mem.1='1024', extra_specs.hw:numa_mempolicy='strict', extra_specs.hw:numa_nodes='2', original_name='m2.mixed', ram='2048', swap='0', vcpus='2' |
| hostId | |
| host_status | |
| id | 8a74eaf5-f564-46e5-aec7-e2024ada903f |
| image | r9-q (e061a2c3-a176-4a3f-a3d0-591a3911c91f) |
| key_name | None |
| locked | False |
| locked_reason | None |
| name | vm_numa_hugepage |
| progress | 0 |
| project_id | d44934edcd904399b994d2411d99e662 |
| properties | |
| security_groups | name='default' |
| server_groups | [] |
| status | BUILD |
| tags | |
| trusted_image_certificates | None |
| updated | 2023-02-21T01:52:07Z |
| user_id | 06883cd41346460cae0e28b0f5f9455b |
| volumes_attached | |
+-------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
(overcloud) [stack@dell-per740-66 ~]$ openstack server list
+--------------------------------------+------------------+--------+-----------------------+-------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------+--------+-----------------------+-------+----------+
| 8a74eaf5-f564-46e5-aec7-e2024ada903f | vm_numa_hugepage | ERROR | | r9-q | m2.mixed |
| 70ef225c-8360-444c-948f-40ef39ba83dc | vm_numa | ACTIVE | private=192.168.24.73 | r9-q | m1.mixed |
+--------------------------------------+------------------+--------+-----------------------+-------+----------+
(overcloud) [stack@dell-per740-66 ~]$ openstack server show 8a74eaf5-f564-46e5-aec7-e2024ada903f -c fault -f yaml
fault:
code: 500
created: '2023-02-21T01:52:12Z'
details: "Traceback (most recent call last):\n File \"/usr/lib/python3.9/site-packages/nova/conductor/manager.py\"\
, line 697, in build_instances\n raise exception.MaxRetriesExceeded(reason=msg)\n\
nova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exhausted\
\ all hosts available for retrying build failures for instance 8a74eaf5-f564-46e5-aec7-e2024ada903f.\n"
message: Exceeded maximum number of retries. Exhausted all hosts available for retrying
build failures for instance 8a74eaf5-f564-46e5-aec7-e2024ada903f.
[root@compute-0 qemu]# tail -f -n3 /var/log/containers/libvirt/qemu/instance-00000006.log
char device redirected to /dev/pts/1 (label charserial0)
2023-02-21T01:52:10.508821Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/2-instance-00000006 for guest RAM: Permission denied
2023-02-21 01:52:10.610+0000: shutting down, reason=failed
Log snippet: [root@compute-0 qemu]# vi /var/log/containers/nova/nova-compute.log
...
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
<page size='1048576' unit='KiB' nodeset='1'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='5'/>
<emulatorpin cpuset='4-5'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
...
: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-21T01:52:10.508821Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/2-instance-00000006 for guest RAM: Permission denied
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest Traceback (most recent call last):
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest return self._domain.createWithFlags(flags)
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest rv = execute(f, *args, **kwargs)
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 129, in execute
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest six.reraise(c, e, tb)
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest raise value
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 83, in tworker
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest rv = meth(*args, **kwargs)
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1409, in createWithFlags
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest raise libvirtError('virDomainCreateWithFlags() failed')
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-21T01:52:10.508821Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/2-instance-00000006 for guest RAM: Permission denied
2023-02-21 01:52:10.833 2 ERROR nova.virt.libvirt.guest
2023-02-21 01:52:10.834 2 ERROR nova.virt.libvirt.driver [req-ced894d9-9efc-4741-a97f-c83b4cde19b7 06883cd41346460cae0e28b0f5f9455b d44934edcd904399b994d2411d99e662 - default default] [instance: 8a74eaf5-f564-46e5-aec7-e2024ada903f] Failed to start libvirt guest: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-02-21T01:52:10.508821Z qemu-kvm: can't open backing store /dev/hugepages/libvirt/qemu/2-instance-00000006 for guest RAM: Permission denied
Hi Michal,
And do you think scenario-2 is the same issue with https://bugzilla.redhat.com/show_bug.cgi?id=2166979, please help investigate it, thanks.
(In reply to zhoujunqin from comment #15) > And do you think scenario-2 is the same issue with > https://bugzilla.redhat.com/show_bug.cgi?id=2166979, please help investigate > it, thanks. It's more like: https://bugzilla.redhat.com/show_bug.cgi?id=2167102 but in the end, both issues are the same (OSP starts virtqemud with too restrictive umask). They just differ in the way the restriction demonstrates itself. Closing this per comment 12. *** This bug has been marked as a duplicate of bug 2154750 *** |