Bug 862914 - virDomainSetMemoryFlags() and virDomainSetVcpusFlags() don't work immediately after a reboot.
virDomainSetMemoryFlags() and virDomainSetVcpusFlags() don't work immediately...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.3
x86_64 Linux
unspecified Severity high
: rc
: ---
Assigned To: Martin Kletzander
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-10-03 17:11 EDT by Michael Closson
Modified: 2012-12-19 15:40 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-12-18 21:01:23 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
driver program to reproduce the bug. (1.98 KB, text/plain)
2012-10-03 17:11 EDT, Michael Closson
no flags Details

  None (edit)
Description Michael Closson 2012-10-03 17:11:06 EDT
Created attachment 621161 [details]
driver program to reproduce the bug.

Description of problem:

I have a C program that uses the libvirt API to set memory vcpus and then power on the machine.  The program works during normal operation.  But if I reboot the hypervisor after powering on a VM, then the program executes successfully, but the memory values that I specify don't take affect.


Version-Release number of selected component (if applicable):

[root@hb05b15 ~]# rpm -qa | egrep 'libvirt|kvm'
libvirt-devel-0.9.10-21.el6_3.4.x86_64
qemu-kvm-0.12.1.2-2.295.el6.x86_64
libvirt-client-0.9.10-21.el6_3.4.x86_64
libvirt-0.9.10-21.el6_3.4.x86_64
libvirt-java-devel-0.4.7-1.el6.noarch
libvirt-java-0.4.7-1.el6.noarch
libvirt-python-0.9.10-21.el6_3.4.x86_64



How reproducible:


Always.

Steps to Reproduce:
1. Create a VM.  Mine has a qcow2 disk on an NFS server.
2. Power it on with the attached program.  ./setmemandpoweron VM1 4000000 4
3. Reboot the hypervisor.
4. When the HV comes back up, the VM will be off (of course).
5. Check the VM config

[root@hb05b15 ~]# virsh dumpxml VM1 | head -20
<domain type='kvm'>
  <name>VM1</name>
  <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
  <memory unit='KiB'>4000000</memory>
  <currentMemory unit='KiB'>4000000</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>


6. Start the VM.

[root@hb05b15 src]# ./setmemandpoweron VM1 7000000 7
Config change done.  hit enter to power on.

Before hitting enter to initiate the poweron, check the VM config in a different terminal.

[root@hb05b15 ~]# virsh dumpxml VM1 | head -20
<domain type='kvm'>
  <name>VM1</name>
  <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
  <memory unit='KiB'>7000000</memory>
  <currentMemory unit='KiB'>7000000</currentMemory>
  <vcpu placement='static'>7</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>

After hitting enter (and the VM powers on).  Check the config again.

[root@hb05b15 ~]# virsh dumpxml VM1 | head -20
<domain type='kvm' id='2'>
  <name>VM1</name>
  <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
  <memory unit='KiB'>4000768</memory>
  <currentMemory unit='KiB'>4000768</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>

Also, check the kvm command line.

root      3514     1 35 17:01 ?        00:01:24 /usr/libexec/qemu-kvm -S -M rhel6.3.0 -enable-kvm -m 3907 -smp 4,sockets=4,cores=1,threads=1 -name VM1 -uuid d4385c25-0946-4fb0-bbb8-55092585e7b4 ...






  
Actual results:

The mem/vcpu values set w/ the API are ignored.


Expected results:

The mem/vcpu values set w/ the API should be used.


Additional info:
Comment 2 Eric Blake 2012-10-03 17:54:00 EDT
(In reply to comment #0)
> 
> Steps to Reproduce:
> 1. Create a VM.  Mine has a qcow2 disk on an NFS server.
> 2. Power it on with the attached program.  ./setmemandpoweron VM1 4000000 4
> 3. Reboot the hypervisor.

Are you using the libvirt-guests init script to restore running guests across host reboots?

> 4. When the HV comes back up, the VM will be off (of course).

Apparently not.  At any rate, that shouldn't affect the real analysis.

> 5. Check the VM config
> 
> [root@hb05b15 ~]# virsh dumpxml VM1 | head -20
> <domain type='kvm'>
>   <name>VM1</name>
>   <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
>   <memory unit='KiB'>4000000</memory>
>   <currentMemory unit='KiB'>4000000</currentMemory>
>   <vcpu placement='static'>4</vcpu>

So this is the current offline state of the domain...

> 
> 6. Start the VM.
> 
> [root@hb05b15 src]# ./setmemandpoweron VM1 7000000 7
> Config change done.  hit enter to power on.

and here, you asked to change the offline config...

> 
> Before hitting enter to initiate the poweron, check the VM config in a
> different terminal.
> 
> [root@hb05b15 ~]# virsh dumpxml VM1 | head -20
> <domain type='kvm'>
>   <name>VM1</name>
>   <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
>   <memory unit='KiB'>7000000</memory>
>   <currentMemory unit='KiB'>7000000</currentMemory>
>   <vcpu placement='static'>7</vcpu>

...and those requests appear to take hold...

> After hitting enter (and the VM powers on).  Check the config again.
> 
> [root@hb05b15 ~]# virsh dumpxml VM1 | head -20
> <domain type='kvm' id='2'>
>   <name>VM1</name>
>   <uuid>d4385c25-0946-4fb0-bbb8-55092585e7b4</uuid>
>   <memory unit='KiB'>4000768</memory>
>   <currentMemory unit='KiB'>4000768</currentMemory>
>   <vcpu placement='static'>4</vcpu>

...except that the virDomainCreate() is reverting back to the earlier state.  To me, this implies that the VIR_DOMAIN_AFFECT_CONFIG flag of the API in question is affecting an in-memory copy, and as long as the guest is offline, further dumpXML calls read the in-memory copy.  But it is forgetting to properly flush it back to the persistent state, so that when the domain starts, it reads from the disk instead of in-memory settings, and starts with the wrong values.

Definitely a bug in our VIR_DOMAIN_AFFECT_CONFIG handling.
Comment 3 Michael Closson 2012-10-03 18:42:22 EDT
The guest is not configured to boot when the host boots.



I (unsuccessfully) tried the following to work around the problem. 

To change mem/vcpus:
- Load the XML directly from /etc/libvirt/qemu/VM1.xml
- Change the vcpus node "vcpu".  (and the two mem nodes.)
- Use virDomainDefineXML().

The behaviour is the same.  After the call to virDomainDefineXML(), virsh dumpxml shows the updated values, but they are reverted upon the call to virDomainCreate().

Any suggestions to work around this problem are appreciated.
Comment 4 Eric Blake 2012-10-03 19:00:25 EDT
(In reply to comment #3)
> The guest is not configured to boot when the host boots.
> 
> 
> 
> I (unsuccessfully) tried the following to work around the problem. 
> 
> To change mem/vcpus:
> - Load the XML directly from /etc/libvirt/qemu/VM1.xml

Don't ever directly edit files in /etc/libvirt.  It means you are going behind libvirt's back, and all bets are off.
http://wiki.libvirt.org/page/FAQ#Where_are_VM_config_files_stored.3F_How_do_I_edit_a_VM.27s_XML_config.3F

> Any suggestions to work around this problem are appreciated.

Using virDomainDefineXML() might work as a workaround, if you don't mind hand-editing the XML returned from virDomainGetXMLDesc(), and can't wait for the bug in virDomainSetMemoryFlags() and friends to be patched, although I haven't tried it.
Comment 6 Wayne Sun 2012-11-06 00:09:20 EST
pkgs:
libvirt-0.10.2-6.el6.x86_64
qemu-kvm-0.12.1.2-2.316.el6.x86_64
kernel-2.6.32-330.el6.x86_64

steps:
1. prepare a domain without autostart

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     libvirt_test_api               shut off

# service libvirt-guests stop
# chkconfig --level 345 libvirt-guests off
# virsh autostart libvirt_test_api --disable

2. using python script to start domain:

#!/usr/bin/env python
import sys
import time
import libvirt

guestname = sys.argv[1]
mem = int(sys.argv[2])
vcpu = int(sys.argv[3])

con = libvirt.open(None)
domobj = con.lookupByName(guestname)
domobj.setMemoryFlags(mem, libvirt.VIR_DOMAIN_AFFECT_CONFIG|libvirt.VIR_DOMAIN_MEM_MAXIMUM)
domobj.setMemoryFlags(mem, libvirt.VIR_DOMAIN_AFFECT_CONFIG|libvirt.VIR_DOMAIN_MEM_CURRENT)
domobj.setVcpusFlags(vcpu, libvirt.VIR_DOMAIN_AFFECT_CONFIG|libvirt.VIR_DOMAIN_MEM_MAXIMUM)
domobj.setVcpusFlags(vcpu, libvirt.VIR_DOMAIN_AFFECT_CONFIG|libvirt.VIR_DOMAIN_MEM_CURRENT)
str = domobj.XMLDesc(0)
print str
time.sleep(3)
domobj.create()

# python cpu_mem_set.py libvirt_test_api 4000000 4
...
  <memory unit='KiB'>4000768</memory>
  <currentMemory unit='KiB'>4001792</currentMemory>
  <vcpu placement='static'>4</vcpu>
...

# virsh list
 Id    Name                           State
----------------------------------------------------
 4     libvirt_test_api               running

3. reboot host
# reboot

4. retest with script
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     libvirt_test_api               shut off

# virsh dumpxml libvirt_test_api
...
  <memory unit='KiB'>4000000</memory>
  <currentMemory unit='KiB'>4000000</currentMemory>
  <vcpu placement='static'>4</vcpu>
...

# python cpu_mem_set.py libvirt_test_api 7000000 7
...
  <memory unit='KiB'>7000000</memory>
  <currentMemory unit='KiB'>7000000</currentMemory>
  <vcpu placement='static'>7</vcpu>
...

...
  <memory unit='KiB'>7000064</memory>
  <currentMemory unit='KiB'>7000064</currentMemory>
  <vcpu placement='static'>7</vcpu>
...

Can not reproduce it here.

Also tried with the c program attached on libvirt-0.9.10-21.el6_3.4.x86_64 and with exact steps as described(both with img on nfs and local), still fail to reproduce.
Comment 7 Michael Closson 2012-11-06 18:47:05 EST
Hi Wayne,  I tested this again, with the most recent libvirt code.  Here is my result.

Step 1: Check the versions.

[root@dcdemo04 ~]# rpm -qa | egrep 'libvirt'
libvirt-client-0.9.10-21.el6_3.5.x86_64
libvirt-python-0.9.10-21.el6_3.5.x86_64
libvirt-0.9.10-21.el6_3.5.x86_64


Step 2: Start a VM.

[root@dcdemo04 ~]# setmemandpoweron VM1 2000000 2
Config change done.  hit enter to power on.

[root@dcdemo04 ~]# virsh dumpxml VM1 | egrep 'emory|vcpu'
  <memory unit='KiB'>2000896</memory>
  <currentMemory unit='KiB'>2000000</currentMemory>
  <vcpu placement='static'>2</vcpu>
[root@dcdemo04 ~]# ps -ef | grep kvm
root      2029     2  0 15:26 ?        00:00:00 [kvm-irqfd-clean]
root      5502     1 19 15:34 ?        00:00:08 /usr/libexec/qemu-kvm ... -m 1954 -smp 2,sockets=2,cores=1,threads=1 -name VM1 ...


Step 3: Reboot the hypervisor.


[root@dcdemo04 ~]# reboot

Broadcast message from root@dcdemo04
	(/dev/pts/0) at 15:36 ...

The system is going down for reboot NOW!
[root@dcdemo04 ~]# Connection to dcdemo04 closed by remote host.
Connection to dcdemo04 closed.


Step 4: When the hypervisor comes back up, try to power it on again with a different value for memory and vcpus.  The program will prompt you to check the config and then hit enter.


[root@dcdemo04 ~]# setmemandpoweron VM1 4000000 4
Config change done.  hit enter to power on.

# In another window, check the config.

[root@dcdemo04 ~]# virsh dumpxml VM1 | egrep 'emory|vcpu'
  <memory unit='KiB'>4000000</memory>
  <currentMemory unit='KiB'>4000000</currentMemory>
  <vcpu placement='static'>4</vcpu>

# In the original window, hit enter to actually trigger the api call.  Then check the VM.

[root@dcdemo04 ~]# virsh dumpxml VM1 | egrep 'emory|vcpu'
  <memory unit='KiB'>2000896</memory>
  <currentMemory unit='KiB'>2000000</currentMemory>
  <vcpu placement='static'>2</vcpu>
[root@dcdemo04 ~]# 


root      4454     1  3 15:46 ?        00:00:02 /usr/libexec/qemu-kvm ... -m 1954 -smp 2,sockets=2,cores=1,threads=1 -name VM1 ...



Seems that the bug is still there.  Can you retry and post the same output as mine?
Comment 8 Michael Closson 2012-11-06 19:12:45 EST
I can also reproduce the bug with your python script.

Initial start.

[root@dcdemo04 ~]# ./libvirt_test_api.py VM1 3000000 3
  <memory unit='KiB'>3000000</memory>
  <currentMemory unit='KiB'>3000000</currentMemory>
  <vcpu placement='static'>3</vcpu>

Reboot.

[root@dcdemo04 ~]# ./libvirt_test_api.py VM1 5000000 5
  <memory unit='KiB'>5000000</memory>
  <currentMemory unit='KiB'>5000000</currentMemory>
  <vcpu placement='static'>5</vcpu>


[root@dcdemo04 ~]# virsh dumpxml VM1 | egrep 'emory|vcpu'
  <memory unit='KiB'>3000320</memory>
  <currentMemory unit='KiB'>3000000</currentMemory>
  <vcpu placement='static'>3</vcpu>

root      6663     1 17 16:11 ?        00:00:01 /usr/libexec/qemu-kvm ... -m 2930 -smp 3,sockets=3,cores=1,threads=1 ...


And then if I kill the VM, the mem/vcpu changes back.

[root@dcdemo04 ~]# kill 6663
[root@dcdemo04 ~]# virsh dumpxml VM1 | egrep 'emory|vcpu'
  <memory unit='KiB'>5000000</memory>
  <currentMemory unit='KiB'>5000000</currentMemory>
  <vcpu placement='static'>5</vcpu>
Comment 9 Wayne Sun 2012-11-08 03:15:09 EST
steps 1
# rpm -qa|grep libvirt
libvirt-client-0.9.10-21.el6_3.5.x86_64
libvirt-devel-0.9.10-21.el6_3.5.x86_64
libvirt-python-0.9.10-21.el6_3.5.x86_64
libvirt-debuginfo-0.9.10-21.el6_3.5.x86_64
libvirt-0.9.10-21.el6_3.5.x86_64

Step 2: Start a VM.
# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     libvirt_test_api               shut off


# ./setmemandpoweron libvirt_test_api 2000000 2
Config change done.  hit enter to power on.

# virsh dumpxml libvirt_test_api | egrep 'emory|vcpu'
  <memory unit='KiB'>2000896</memory>
  <currentMemory unit='KiB'>2000896</currentMemory>
  <vcpu placement='static'>2</vcpu>

# ps aux|grep kvm
root      2199  0.0  0.0      0     0 ?        S    15:22   0:00 [kvm-irqfd-clean]
qemu      3914 64.9  0.1 2557568 251432 ?      Sl   15:30   0:07 /usr/libexec/qemu-kvm -S -M rhel6.4.0 -enable-kvm -m 1954 -smp 2,sockets=2,cores=1,threads=1 -name libvirt_test_api ...

Step 3: Reboot the hypervisor.

# reboot

Step 4: 

# ./setmemandpoweron libvirt_test_api 4000000 4
Config change done.  hit enter to power on.

# In another window, check the config.
# virsh dumpxml libvirt_test_api | egrep 'emory|vcpu'
  <memory unit='KiB'>4000000</memory>
  <currentMemory unit='KiB'>4000000</currentMemory>
  <vcpu placement='static'>4</vcpu>

# In the original window, hit enter to actually trigger the api call.  Then check the VM.

# virsh dumpxml libvirt_test_api | egrep 'emory|vcpu'
  <memory unit='KiB'>4000768</memory>
  <currentMemory unit='KiB'>4001792</currentMemory>
  <vcpu placement='static'>4</vcpu>

# ps aux|grep kvm
root      2056  0.0  0.0      0     0 ?        S    15:40   0:00 [kvm-irqfd-clean]
qemu      3645 66.8  0.5 4664060 668300 ?      Sl   15:44   0:20 /usr/libexec/qemu-kvm -S -M rhel6.4.0 -enable-kvm -m 3907 -smp 4,sockets=4,cores=1,threads=1 -name libvirt_test_api ...


still can't reproduce.

Hi Michael,

can you attach your domain xml, check dominfo and libvirt-guests service?

# virsh dumpxml libvirt_test_api > /tmp/domain.xml
then attach it here

# service libvirt-guests status
stopped, with no saved guests

# virsh dominfo libvirt_test_api|grep Autostart
Autostart:      disable
Comment 10 Michael Closson 2012-11-19 13:55:27 EST
The libvirt-guests service was enabled and running on my hypervisor host.  But the domain wasn't set to autostart.

If I disable that service, then I cannot reproduce the problem anymore.

Also, to double check, I enabled the service again, rebooted the HV and was againa able to reproduce the problem.
Comment 11 Michael Closson 2012-12-17 12:41:51 EST
For my requirements this isn't a bug any more. So I will close it.  But the behaviour is not right.

- The libvirt-guests service is enabled.
- A VM is _not_ configured to autostart.
- The HV host gracefully restarts.
- On shutdown, the guest's mem and vcpu config is "remembered".
- When the HV comes backup, the guest isn't automatically restarted.  Which is expected since it wasn't configured to autostart.
- When the guest is started, with a different mem/cpu config, the previous "remembered" config is actually used, not the user supplied one.  IMO, using the user supplied config is the right design.
Comment 12 Dave Allan 2012-12-17 16:15:55 EST
Eric, it looks to me like Michael's program is modifying the persistent config, so that's what's subsequently being used to start the VM so this is behavior by design, is that your reading also?
Comment 13 Eric Blake 2012-12-18 17:01:19 EST
(In reply to comment #10)
> The libvirt-guests service was enabled and running on my hypervisor host. 
> But the domain wasn't set to autostart.

The design of the libvirt-guests service is that it is _supposed_ to re-start any guest that was running when the host shut down, whether or not that guest was marked autostart.  And when it does re-start the guest, it uses the configuration that was in use by the guest at the time the host went down.

> 
> If I disable that service, then I cannot reproduce the problem anymore.

Therefore, what you are seeing is that the guest state saved by libvirt-guests is surviving a host reboot, even if it is different than the state you manually put into the config file.  The fact that the guest is resuming where it left off (including the state it was previously running with, and NOT the state in the persistent config file) is a feature.

> 
> Also, to double check, I enabled the service again, rebooted the HV and was
> againa able to reproduce the problem.

I think what you are seeing is by design, and not a bug, and that you missed the fact that using libvirt-guests will re-start a guest even if it is not marked as autostart.
Comment 14 Michael Closson 2012-12-19 15:40:02 EST
"The design of the libvirt-guests service is that it is _supposed_ to re-start any guest that was running when the host shut down, whether or not that guest was marked autostart."

I see.  The problem that my system had is that the VM disk is on a shared filesystem that isn't available until after libvirt-guests runs.  So resuming the managed save fails.  Not really a problem with libvirt-guests since its probably assumed that the VM disk is available when libvirt-guests starts.  And the negative case handling for the case that the VM disk isn't available probably hasn't been defined yet.

Note You need to log in before you can comment on or make changes to this bug.