Bug 923946 - RHEL-6.4: migration is failing: libvirtError: XML error: missing security model when using multiple labels
Summary: RHEL-6.4: migration is failing: libvirtError: XML error: missing security mod...
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.4
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Michal Privoznik
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Keywords: Regression, ZStream
Depends On:
Blocks: 896690 928879
TreeView+ depends on / blocked
 
Reported: 2013-03-20 18:44 UTC by Haim
Modified: 2014-01-13 00:56 UTC (History)
19 users (show)

(edit)
When SELinux was disabled on a host, or the qemu driver was configured not to use it, and the domain XML configuration contained an explicit seclabel option, the code parsed the seclabel option, but ignored it later on when it was generating labels on domain start, and created a new and empty seclabel entry [seclabeltype='none'/]. Consequently, a migration between two hosts running Red Hat Enterprise Linux 6.4 failed with the following error message: libvirtError: XML error: missing security model when using multiple labels With this update, if a seclabel entry already exists, a new one is no longer created, and the migration works as expected in the described scenario
Clone Of:
(edit)
Last Closed: 2013-11-21 08:52:32 UTC


Attachments (Terms of Use)
libvirt and others. (17.82 MB, application/x-tar)
2013-03-20 18:49 UTC, Haim
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2013:1581 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2013-11-21 01:11:35 UTC

Description Haim 2013-03-20 18:44:33 UTC
Description of problem:

migration between host running libvirt libvirt-0.10.2-18 to host running libvirt- 0.10.2-18.el6_4.2 is failing with the following error:

Traceback (most recent call last):
  File "/usr/share/vdsm/vm.py", line 270, in run
    self._startUnderlyingMigration()
  File "/usr/share/vdsm/libvirtvm.py", line 493, in _startUnderlyingMigration
    None, maxBandwidth)
  File "/usr/share/vdsm/libvirtvm.py", line 529, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 104, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: XML error: missing security model when using multiple labels

repro steps:

1) have a VM running on source running latest release of libvirt
2) migrate VM from source to dest running libvirt's RHEL6.4.z version

Comment 1 Haim 2013-03-20 18:49:20 UTC
Created attachment 713416 [details]
libvirt and others.

Comment 3 Jiri Denemark 2013-03-21 10:22:56 UTC
So the root cause of this problem is in libvirt's security driver. In case SELinux is disabled on the host or qemu driver is configured not to use it (security_driver = "none" in /etc/libvirt/qemu.conf) and domain XML contains explicit seclabel (<seclabel type='dynamic' model='selinux' relabel='yes'/>). The code parses the seclabel but ignores it later on when it is generating labels on domain start and create new <seclabel type='none'/>, which is only valid if no other seclabel elements are present. As a result, even restarting libvirtd is enough to observe the bug, since the domain (while still running) will not be listed as running by libvirt.

Luckily, this bug was not introduced in the 6.4.z build. However, it is a regression since 6.3.

Moreover, I think that a domain with explicitly configured seclabel should fail to start if libvirt is not able to provide that seclabel (because the corresponding security driver is disabled). However, this behavior is consistent with 6.3.

Comment 5 Jiri Denemark 2013-03-21 10:35:39 UTC
Package version: libvirt-0.10.2-18

Steps:
1. set security_driver = "none" in /etc/libvirt/qemu.conf (or alternatively disable SELinux on the host)
2. restart libvirtd service (or the host in case the alternative route in step 1 was chosen)
3. start a domain which contains <seclabel type='dynamic' model='selinux' relabel='yes'/> in its XML configuration
4. restart libvirtd service
5. the domain disappears from virsh list and "XML error: security label is missing" error can be found in the logs when libvirtd is trying to parse the domain's XML config from /var/run/libvirt/qemu

Alternatively, instead of the steps 4 and 5 above, one can do:
4. managedsave DOM
5. start DOM

in this case step 5 fails with "XML error: missing security model when using multiple labels".

Comment 7 Michal Privoznik 2013-03-21 10:47:10 UTC
Patch proposed upstream:

https://www.redhat.com/archives/libvir-list/2013-March/msg01217.html

Comment 9 Michal Privoznik 2013-03-28 09:14:28 UTC
Patch is not pushed upstream:

commit e4a28a328188e715de8d587b390cb44e2d6c3fb4
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Mar 21 16:32:07 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Thu Mar 28 10:01:06 2013 +0100

    security: Don't add seclabel of type none if there's already a seclabel
    
    https://bugzilla.redhat.com/show_bug.cgi?id=923946
    
    The <seclabel type='none'/> should be added iff there is no other
    seclabel defined within a domain. This bug can be easily reproduced:
    1) configure selinux seclabel for a domain
    2) disable system's selinux and restart libvirtd
    3) observe <seclabel type='none'/> being appended to a domain on its
       startup

commit 6c4de1161425a610797495549349d194b90fb023
Author:     Michal Privoznik <mprivozn@redhat.com>
AuthorDate: Thu Mar 21 16:12:55 2013 +0100
Commit:     Michal Privoznik <mprivozn@redhat.com>
CommitDate: Thu Mar 28 10:01:06 2013 +0100

    security_manager: Don't manipulate domain XML in virDomainDefGetSecurityLabelDef
    
    The virDomainDefGetSecurityLabelDef was modifying the domain XML.
    It tried to find a seclabel corresponding to given sec driver. If the
    label wasn't found, the function created one which is wrong. In fact
    it's security manager which should modify this part of domain XML.

v1.0.4-rc2-5-ge4a28a3

Comment 11 Eric Blake 2013-04-01 16:04:06 UTC
(In reply to comment #3)
> So the root cause of this problem is in libvirt's security driver. In case
> SELinux is disabled on the host or qemu driver is configured not to use it
> (security_driver = "none" in /etc/libvirt/qemu.conf) and domain XML contains
> explicit seclabel (<seclabel type='dynamic' model='selinux'
> relabel='yes'/>). The code parses the seclabel but ignores it later on when
> it is generating labels on domain start and create new <seclabel
> type='none'/>, which is only valid if no other seclabel elements are
> present. As a result, even restarting libvirtd is enough to observe the bug,
> since the domain (while still running) will not be listed as running by
> libvirt.
> 
> Luckily, this bug was not introduced in the 6.4.z build. However, it is a
> regression since 6.3.

There is an upstream report of someone hitting something that sounds very much like this problem, but while while using libvirt 0.10.2 for migration (which means 6.3 is probably also impacted):
https://www.redhat.com/archives/libvirt-users/2013-April/msg00003.html

Comment 12 Jiri Denemark 2013-04-02 07:06:55 UTC
(In reply to comment #11)
> There is an upstream report of someone hitting something that sounds very
> much like this problem, but while while using libvirt 0.10.2 for migration
> (which means 6.3 is probably also impacted):

0.10.2 is actually 6.4 (6.3 had 0.9.10). We did not have any support for multiple security labels in 6.3.

Comment 13 Michal Privoznik 2013-04-02 10:11:06 UTC
As we've found out in bug 928879 the summary does not correspond to the behaviour described in report. Both are bugs, but a different ones. I am changing the summary back to he original one (slightly modified) and for the other bug I've reported bug 947387.

Comment 15 yanbing du 2013-07-09 11:23:57 UTC
I'm testing this bug, and following steps in comment5, the results are:

1. set security_driver = "none" in /etc/libvirt/qemu.conf (or alternatively disable SELinux on the host)
2. restart libvirtd service (or the host in case the alternative route in step 1 was chosen)
3. start a domain which contains <seclabel type='dynamic' model='selinux' relabel='yes'/> in its XML configuration
4. restart libvirtd service

then the running domian will shutoff, and error in libvirtd.log:
...
013-07-09 11:13:41.161+0000: 23141: info : libvirt version: 0.10.2, package: 19.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2013-07-08-04:51:26, x86-023.build.eng.bos.redhat.com)
2013-07-09 11:13:41.161+0000: 23141: warning : virSecurityManagerNew:148 : Configured security driver "none" disables default policy to create confined guests
 2013-07-09 11:13:41.375+0000: 23141: error : virSecurityLabelDefParseXML:3226 : XML error: security label is missing

OR
4. managedsave DOM
# virsh managedsave tsu-1

Domain tsu-1 state saved by libvirt

5. start DOM
# virsh start tsu-1
Domain tsu-1 started

domain can start, but there are many cgroup error in libvirtd.log:
13-07-09 11:14:39.817+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpu/libvirt/qemu/tsu-1//emulator (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpu/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpuacct/libvirt/qemu/tsu-1//emulator (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpuacct/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpuset/libvirt/qemu/tsu-1//emulator (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/cpuset/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/memory/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/devices/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/freezer/libvirt/qemu/tsu-1/ (16)
2013-07-09 11:14:39.840+0000: 23134: error : virCgroupRemoveRecursively:730 : Unable to remove /cgroup/blkio/libvirt/qemu/tsu-1/ (16)


I'm not sure if the result can verify this bug, so need your confirm, thanks!

Comment 16 Michal Privoznik 2013-07-10 12:28:32 UTC
(In reply to yanbing du from comment #15)
> I'm testing this bug, and following steps in comment5, the results are:
> 
> 1. set security_driver = "none" in /etc/libvirt/qemu.conf (or alternatively
> disable SELinux on the host)
> 2. restart libvirtd service (or the host in case the alternative route in
> step 1 was chosen)
> 3. start a domain which contains <seclabel type='dynamic' model='selinux'
> relabel='yes'/> in its XML configuration

Can you provide virsh dumpxml  after this step?

> 4. restart libvirtd service
> 
> then the running domian will shutoff, and error in libvirtd.log:
> ...
> 013-07-09 11:13:41.161+0000: 23141: info : libvirt version: 0.10.2, package:
> 19.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> 2013-07-08-04:51:26, x86-023.build.eng.bos.redhat.com)
> 2013-07-09 11:13:41.161+0000: 23141: warning : virSecurityManagerNew:148 :
> Configured security driver "none" disables default policy to create confined
> guests
>  2013-07-09 11:13:41.375+0000: 23141: error :
> virSecurityLabelDefParseXML:3226 : XML error: security label is missing
> 
> OR
> 4. managedsave DOM
> # virsh managedsave tsu-1
> 
> Domain tsu-1 state saved by libvirt
> 
> 5. start DOM
> # virsh start tsu-1
> Domain tsu-1 started

Comment 17 yanbing du 2013-07-11 03:02:13 UTC
(In reply to Michal Privoznik from comment #16)
> (In reply to yanbing du from comment #15)
> > I'm testing this bug, and following steps in comment5, the results are:
> > 
> > 1. set security_driver = "none" in /etc/libvirt/qemu.conf (or alternatively
> > disable SELinux on the host)
> > 2. restart libvirtd service (or the host in case the alternative route in
> > step 1 was chosen)
> > 3. start a domain which contains <seclabel type='dynamic' model='selinux'
> > relabel='yes'/> in its XML configuration
> 
> Can you provide virsh dumpxml  after this step?

# virsh start test
Domain test started

# virsh dumpxml test|grep seclabel

  <seclabel type='dynamic' model='selinux' relabel='yes'/>

If delete the <seclabel/> element and start the guest:
# virsh dumpxml test|grep seclabel
  <seclabel type='none'/>

then restart libvirtd, the guest will still running and an warning message in libvirtd.log: 
2013-07-10 18:50:24.670+0000: 48362: warning : virSecurityManagerNew:148 : Configured security driver "none" disables default policy to create confined guests

> 
> > 4. restart libvirtd service
> > 
> > then the running domian will shutoff, and error in libvirtd.log:
> > ...
> > 013-07-09 11:13:41.161+0000: 23141: info : libvirt version: 0.10.2, package:
> > 19.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>,
> > 2013-07-08-04:51:26, x86-023.build.eng.bos.redhat.com)
> > 2013-07-09 11:13:41.161+0000: 23141: warning : virSecurityManagerNew:148 :
> > Configured security driver "none" disables default policy to create confined
> > guests
> >  2013-07-09 11:13:41.375+0000: 23141: error :
> > virSecurityLabelDefParseXML:3226 : XML error: security label is missing
> > 
> > OR
> > 4. managedsave DOM
> > # virsh managedsave tsu-1
> > 
> > Domain tsu-1 state saved by libvirt
> > 
> > 5. start DOM
> > # virsh start tsu-1
> > Domain tsu-1 started

Comment 18 Michal Privoznik 2013-07-12 07:43:53 UTC
(In reply to yanbing du from comment #17)
> (In reply to Michal Privoznik from comment #16)
> > (In reply to yanbing du from comment #15)
> > > I'm testing this bug, and following steps in comment5, the results are:
> > > 
> > > 1. set security_driver = "none" in /etc/libvirt/qemu.conf (or alternatively
> > > disable SELinux on the host)
> > > 2. restart libvirtd service (or the host in case the alternative route in
> > > step 1 was chosen)
> > > 3. start a domain which contains <seclabel type='dynamic' model='selinux'
> > > relabel='yes'/> in its XML configuration
> > 
> > Can you provide virsh dumpxml  after this step?
> 
> # virsh start test
> Domain test started
> 
> # virsh dumpxml test|grep seclabel
> 
>   <seclabel type='dynamic' model='selinux' relabel='yes'/>
> 
> If delete the <seclabel/> element and start the guest:
> # virsh dumpxml test|grep seclabel
>   <seclabel type='none'/>
> 
> then restart libvirtd, the guest will still running and an warning message
> in libvirtd.log: 
> 2013-07-10 18:50:24.670+0000: 48362: warning : virSecurityManagerNew:148 :
> Configured security driver "none" disables default policy to create confined
> guests
> 

Right, thank you for confirmation. I think this is a separate issue - see bug 947387. So I think this bug is fixed.

Comment 19 yanbing du 2013-07-12 13:56:59 UTC
Thanks Michal, then this bug can move to VERIFIED.

Comment 21 errata-xmlrpc 2013-11-21 08:52:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-1581.html


Note You need to log in before you can comment on or make changes to this bug.