Bug 545916

Patches (from stable3) required:

http://git.fedorahosted.org/git/?p=resource-agents.git;a=commit;h=70f5c5dcda594aa29ad07cfe23bd3342e1e7c5c5
http://git.fedorahosted.org/git/?p=resource-agents.git;a=commit;h=1e42add4f2ec63d4c796d544118dfcccbd8a042b
http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=d1b0a2245fab4d93c05a0651fe0f7e714e0bf6b9

These patches allow vm.sh to work in conjunction with libvirt's transient domain functionality to allow storing libvirt domain description XML files in locations other than /etc/libvirt/qemu, for example on a GFS or GFS2 file system.

Existing configurations utilizing the "path" attribute for XM would cause xm to do the searching; the patches move this intelligence into the agent itself when using 'virsh' to manage the virtual machines (which is required for KVM and optional [but strongly encouraged] for Xen).

This preserves upgrade compatibility from a configuration standpoint for users transitioning from Xen to KVM hypervisors without breaking continued Xen support.

http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=19896aac88bd6a0b6ee4b3e47ad4ae7606f9fcad
http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=783372f1feb1bd21c83c633ff5371d47e60f2a36
http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=602b84e1991ece247010b679ef779d09efb0123d

The handling of the path and use_virsh parameters can break previously working xen setups which use a directory other than /etc/xen to hold xen config files.  For example, a system under 5.4 which had xen config files /under /etc/xenconfig previously worked with a line like:

<vm autostart="1" name="virt01" path="/etc/xenconfig" recovery="restart"/>

Without the changes for this BZ, it works and uses xm to start.

clurgmgrd[22293]: <notice> Starting stopped service vm:virt01
clurgmgrd: [22293]: <warning> Cannot use virsh with 'path' attribute set
clurgmgrd: [22293]: <warning> Setting use_virsh to 0.
clurgmgrd: [22293]: <debug> xm create --path="/etc/xenconfig" virt01 on_shutdown="destroy" on_reboot="destroy" on_crash="destroy"
clurgmgrd[22293]: <notice> Service vm:virt01 started

With rgmanager-2.0.52-3 in 5.5-beta, the vm configuration incorrectly tries to use virsh and fails.

clurgmgrd[3975]: <notice> Starting stopped service vm:virt01
clurgmgrd: [3975]: <debug> Using /etc/xenconfig/virt01
clurgmgrd: [3975]: <debug> virsh create /etc/xenconfig/virt01
clurgmgrd[3975]: <notice> start on vm "virt01" returned 1 (generic error)
clurgmgrd[3975]: <warning> #68: Failed to start vm:virt01; return value: 1

The problem appears to be the first if check in choose_management_tool() in /usr/share/cluster/vm.sh .
       #
       # Don't override user value for use_virsh if one is given
       #
       if [ -n "$OCF_RESKEY_use_virsh" ]; then
               return 0
       fi

The xml parameter use_virsh has a default value of 1.  So even though the config file doesn't explicitly set use_virsh, $OCF_RESKEY_use_virsh is still 1 and this check exits out.  This causes vm.sh to try to use virsh.  Prior to the changes for this bugzilla, setting the path parameter would cause OCF_RESKEY_use_virsh to be overridden to 0 by the vm.sh script and xm would be used instead of virsh.

This is peculiar; can you upload your domain description file?

Effectively, we do a number of checks to make this not happen:

 * search the path to see if we can find the config file matching the
   VM name
 * run xmllint on the config file to see if it is XML
   (if it is not XML, assume it is a Xen config file and use xm)

Xen domain configuration files should not pass the xmllint check.

Can you run:

# clusvcadm -d vm:virt01
# OCF_RESKEY_name="virt01" OCF_RESKEY_path="/etc/xenconfig" /usr/share/cluster/vm.sh start

... and paste the output?

On my test:

[root@localhost ~]# cat /sandbox/guests/Default 
name = "Default"
uuid = "6f40e612-e5b5-17ba-c3df-a4091fd62955"
maxmem = 256
memory = 256
vcpus = 1
bootloader = "/usr/bin/pygrub"
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vfb = [  ]
disk = [ "tap:aio:/guests/default.img,xvda,w" ]
vif = [ "mac=00:16:3e:00:cf:fd,bridge=xenbr0" ]


[root@localhost ~]# OCF_RESKEY_path="/sandbox/guests" OCF_RESKEY_name="Default" /usr/share/cluster/vm.sh status
Hypervisor: qemu
<debug>  Using /sandbox/guests/Default
Using /sandbox/guests/Default
<debug>  /sandbox/guests/Default is not XML; using xm
/sandbox/guests/Default is not XML; using xm
Management tool: xm
<err>    Cannot find 'xm'; is it installed?
Cannot find 'xm'; is it installed?

The above correctly failed for me (since I'm not using Xen and xm is not installed).

Oh, I think I understand.

The problem arises because rgmanager sets a default value of OCF_RESKEY_use_virsh.

Changing the default from "1" to "" fixes the problem.

Running through rg_test, with use_virsh default="1":

Checking status of Default...
Hypervisor: qemu
<debug>  Using /sandbox/guests/Default
Using /sandbox/guests/Default
Management tool: virsh
Hypervisor URI: qemu:///system
Migration URI format: qemu+ssh://target_host/system
Virtual machine Default is error: failed to get domain 'Default'
error: Domain not found

Status check of Default failed

Running through rg_test with use_virsh default="" (the _expected_ behavior):

Checking status of Default...
Hypervisor: qemu
<debug>  Using /sandbox/guests/Default
Using /sandbox/guests/Default
<debug>  /sandbox/guests/Default is not XML; using xm
/sandbox/guests/Default is not XML; using xm
Management tool: xm
<err>    Cannot find 'xm'; is it installed?
Cannot find 'xm'; is it installed?
Status check of Default failed

Please verify that changing default="1" -> default="" on ~line 131 in /usr/share/cluster/vm.sh fixes the problem for you.

Created attachment 397187 [details]
Default problem fix

Created attachment 397188 [details]
Patched vm.sh

http://git.fedorahosted.org/git/?p=cluster.git;a=commit;h=d9c1b3a5e50fadac385ee01aa00c57b2f58adbcc

(In reply to comment #13)

Yes, on the local reproducer changing default="1" -> default="" allows the VMs
to start.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2010-0280.html