Created attachment 1448296 [details] guest xml Description of problem: libvirt unable to hot unplug a vcpu which was hotplugged through qemu monitor command and further vcpu hotplug/unplug actions through libvirt is broken. Version-Release number of selected component (if applicable): Host Kernel: 4.17.0-01306-g716a685fdb89 Guest Kernel: 4.17.0-01306-g716a685fdb89 qemu: v2.12.0-1052-g5d7ad3ce10-dirty libvirt: 4.5.0 (f785aa6c2bd8814ee4282d6ac990b7d7650dfa40) Host lscpu: #lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 80 On-line CPU(s) list: 0,8,16,24,32,40,48,56,64,72 Off-line CPU(s) list: 1-7,9-15,17-23,25-31,33-39,41-47,49-55,57-63,65-71,73-79 Thread(s) per core: 1 Core(s) per socket: 5 Socket(s): 2 NUMA node(s): 2 Model: 2.1 (pvr 004b 0201) Model name: POWER8E (raw), altivec supported CPU max MHz: 3690.0000 CPU min MHz: 2061.0000 L1d cache: 64K L1i cache: 32K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0,8,16,24,32 NUMA node1 CPU(s): 40,48,56,64,72 How reproducible: 100 % Steps to Reproduce: 1. Define a pseries guest vm1 using attached xml and boot with above env guest lscpu: # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: KVM Virtualization type: para L1d cache: 64K L1i cache: 32K NUMA node0 CPU(s): 0 2. Hotplug a vcpu using qemu monitor command #virsh qemu-monitor-command vm1 --hmp --cmd "device_add host-spapr-cpu-core,id=core1,core-id=1" # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 1 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: KVM Virtualization type: para L1d cache: 64K L1i cache: 32K NUMA node0 CPU(s): 0,1 3. Hot unplug using libvirt(virsh) API's # virsh setvcpus vm1 1 --live # echo $? 0 # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 1 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: KVM Virtualization type: para L1d cache: 64K L1i cache: 32K NUMA node0 CPU(s): 0,1 4. Try unplug using a different libvirt API. # virsh setvcpu vm1 1 --disable error: invalid argument: vcpu '1' is already in requested state # echo $? 1 # virsh setvcpus vm1 1 --live # echo $? 0 guest lscpu # lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 2 ---------------NOK(it should have been 1) On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 1 Model: 2.1 (pvr 004b 0201) Model name: POWER8 (architected), altivec supported Hypervisor vendor: (null) Virtualization type: full L1d cache: 64K L1i cache: 32K NUMA node0 CPU(s): 0,1 5. Try hotplug through libvirt, it's broken further # virsh setvcpus vm1 2 --live error: internal error: unable to execute QEMU command 'device_add': core 1 already populated
That is precisely the purpose of qemu-monitor-command - to be able to execute any monitor command, including those that will make the domain unusable by libvirt.
Failed theses testcases: (3/8) type_specific.io-github-autotest-libvirt.libvirt_bench.vcpu_hotplug.cpu_stress.setvcpu_del.monitor_add.online: FAIL (154.79 s) (4/8) type_specific.io-github-autotest-libvirt.libvirt_bench.vcpu_hotplug.cpu_stress.setvcpu_del.monitor_add.offline: FAIL (152.99 s) (5/8) type_specific.io-github-autotest-libvirt.libvirt_bench.vcpu_hotplug.cpu_stress.monitor_del.setvcpu_add.online: FAIL (151.42 s) (6/8) type_specific.io-github-autotest-libvirt.libvirt_bench.vcpu_hotplug.cpu_stress.monitor_del.setvcpu_add.offline: FAIL (150.88 s)
So these testcases has to be removed.
Should we not warn/restrict the user during such usage while guest managed by libvirt?
Atleast the monitor commands which will make the changes to the domain which then can not be updated/managed to/by libvirt, there are many harmless info commands which should not going to be an issue.
'man virsh' says: HYPERVISOR-SPECIFIC COMMANDS NOTE: Use of the following commands is strongly discouraged. They can cause libvirt to become confused and do the wrong thing on subsequent operations. Once you have used these commands, please do not report problems to the libvirt developers; the reports will be ignored. If you find that these commands are the only way to accomplish something, then it is better to request that the feature be added as a first-class citizen in the regular libvirt library. [...] qemu-monitor-command domain { [--hmp] | [--pretty] } command... Send an arbitrary monitor command command to domain domain through the qemu monitor. The results of the command will be printed on stdout. If --hmp is passed, the command is considered to be a human monitor command and libvirt will automatically convert it into QMP if needed. In that case the result will also be converted back from QMP. If --pretty is given, and the monitor uses QMP, then the output will be pretty-printed. If more than one argument is provided for command, they are concatenated with a space in between before passing the single command to the monitor.
Libvirt will not audit which commands are safe. The whole use of the API is unsafe and should be used with caution.
Thanks Peter, Thanks Ján Tomko, It is clear from virsh doc, Makes sense, I will remove those testcases from test suite. Regards, -Satheesh.
Removed respective tests, https://github.com/autotest/tp-libvirt/pull/1645.