RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1138203 - Error occurred when install a rhel guest with disk pool
Summary: Error occurred when install a rhel guest with disk pool
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: virt-manager
Version: 7.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Giuseppe Scrivano
QA Contact: Virtualization Bugs
URL:
Whiteboard:
: 1241037 (view as bug list)
Depends On: 1075143 1173695 1173697 1175795
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-09-04 09:28 UTC by zhoujunqin
Modified: 2015-07-15 13:48 UTC (History)
22 users (show)

Fixed In Version: virt-manager-1.1.0-12.el7
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1173695 1173697 (view as bug list)
Environment:
Last Closed: 2015-03-05 10:07:06 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
screenshot-1 (92.42 KB, application/octet-stream)
2014-09-04 09:29 UTC, zhoujunqin
no flags Details
screenshot-2 (73.30 KB, application/octet-stream)
2014-09-04 09:30 UTC, zhoujunqin
no flags Details
screenshot-3 (71.88 KB, application/octet-stream)
2014-09-04 09:30 UTC, zhoujunqin
no flags Details
log file when installation failed (29.29 KB, application/octet-stream)
2014-09-04 09:31 UTC, zhoujunqin
no flags Details
New log file in /tmp (26.12 KB, application/octet-stream)
2014-09-05 01:48 UTC, zhoujunqin
no flags Details
anaconda-tb (146.80 KB, text/plain)
2014-09-05 19:30 UTC, David Shea
no flags Details
tar file got from guest when installation failed (26.86 KB, text/plain)
2014-09-30 03:47 UTC, zhoujunqin
no flags Details
/var/log/libvirt/qemu/rhel6.5-3.log (3.44 KB, text/plain)
2014-09-30 03:47 UTC, zhoujunqin
no flags Details
the host dmesg message (117.59 KB, text/plain)
2014-09-30 03:48 UTC, zhoujunqin
no flags Details
/var/log/libvirt/qemu/rhel6.5-5.log (3.43 KB, text/plain)
2014-10-08 03:54 UTC, zhoujunqin
no flags Details
totally log in bug_1138203_debug.tar.gz (58.58 KB, application/x-gzip)
2014-11-13 09:48 UTC, zhoujunqin
no flags Details
diskpool-rhel6.6-x64.xml (4.74 KB, text/plain)
2014-12-02 03:06 UTC, zhoujunqin
no flags Details
diskpool-rhel6.6-x64.qemu_log (6.73 KB, text/plain)
2014-12-02 03:08 UTC, zhoujunqin
no flags Details
libvirtd_installation.log (3.25 MB, text/plain)
2014-12-02 03:10 UTC, zhoujunqin
no flags Details
diskpool-rhel6.6-x64.tar.gz (26.82 KB, text/plain)
2014-12-02 03:10 UTC, zhoujunqin
no flags Details
the whole audit.log in my host (3.80 MB, text/plain)
2014-12-02 08:18 UTC, zhoujunqin
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0427 0 normal SHIPPED_LIVE virt-manager bug fix and enhancement update 2015-03-05 14:38:46 UTC

Description zhoujunqin 2014-09-04 09:28:23 UTC
Description of problem:
Error occurred when install a rhel guest with disk pool, using whole disk pool(eg. /dev/sdb) or disk volume(/dev/sdb1).

Version-Release number of selected component (if applicable):
kernel-3.10.0-150.el7.x86_64
virt-manager-0.10.0-20.el7.noarch
libvirt-1.2.8-1.el7.x86_64
qemu-kvm-1.5.3-69.el7.x86_64
anaconda-19.31.87-1.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Launch virt-manager.
2. Try to install a rhel guest(eg:rhel6.5), choose http or iso installation,use tree: 6.5_x8664 released version
3. Use a usb disk as storage disk (as screenshot-1), other step configuration as default, then click "Finish", begin to install guest.
3. Follow installation steps as default, just click "Next" step by step.
4. After click "Write change to disk" (as screenshot-2), error shows:
|Storage Activation Failed|
|An error was encountered while activating your storage configuration|
|Input/output error during write on /dev/vda|
Details see screentshot-3.

5.Part of xml file:
...
   <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
...

Actual results:
As described.

Expected results:
Guest with disk pool can be installed successfully.

Additional info:
File aa.tar.gz is the log file when installation failed, get from the guest.
1. This issue can be reproduced on standard partitions.
2. This issue can be reproduced with a real disk.
3. This issue can be reproduced with ide disk.

Comment 1 zhoujunqin 2014-09-04 09:29:14 UTC
Created attachment 934335 [details]
screenshot-1

Comment 2 zhoujunqin 2014-09-04 09:30:18 UTC
Created attachment 934336 [details]
screenshot-2

Comment 3 zhoujunqin 2014-09-04 09:30:55 UTC
Created attachment 934337 [details]
screenshot-3

Comment 4 zhoujunqin 2014-09-04 09:31:32 UTC
Created attachment 934338 [details]
log file when installation failed

Comment 6 David Shea 2014-09-04 13:40:57 UTC
The logs you have attached do not match the problem you have described. The screenshots indicate a problem writing to a virtio disk, /dev/vda, while the traceback you have attached indicates a problem with /dev/sda and has not mention of /dev/vda. Please attach the log files from /tmp at the time of the problem as individual, text/plain attachments.

Comment 7 zhoujunqin 2014-09-05 01:48:39 UTC
Created attachment 934654 [details]
New log file in /tmp

Sorry for the last attachment, the old one got when i tried choosing ide disk type, please use the new attachment, thanks.

Comment 8 zhoujunqin 2014-09-05 06:52:15 UTC
Hi David Shea,
I found this bug issue can reproduce with any disk type, and pool type, it's a common issue, what i want to say is when the guest installation turn to make partition, it will meet such error as the attachment said.

I cannot reproduce this issue with version:
anaconda-widgets-19.31.79-1.el7.x86_64
anaconda-19.31.79-1.el7.x86_64
and the previous version:
anaconda-widgets-19.31.86-1.el7.x86_64
anaconda-19.31.86-1.el7.x86_64

so i think the key problem is new anaconda package, and duing to i cannot install guest now, also changed the bug Priority to high.

Comment 9 David Shea 2014-09-05 19:30:36 UTC
Created attachment 934900 [details]
anaconda-tb

Comment 10 David Shea 2014-09-05 19:32:35 UTC
01:36:26,651 ERR kernel:Buffer I/O error on device vda, logical block 0
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 1
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 2
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 3
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 4
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 5
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 6
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 7
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda
01:36:26,652 ERR kernel:Buffer I/O error on device vda, logical block 8
01:36:26,652 WARNING kernel:lost page write due to I/O error on vda


This is most likely a kernel bug, or possibly a bug in qemu on the host, or possibly a hardware problem on the host.

Comment 12 Stefan Hajnoczi 2014-09-18 15:47:39 UTC
Do you have access to the host dmesg and libvirt domain logs (/var/lib/libvirt/qemu/<domain>.log)?

Are there any I/O error messages in there?

Comment 13 zhoujunqin 2014-09-30 03:45:57 UTC
(In reply to Stefan Hajnoczi from comment #12)
> Do you have access to the host dmesg and libvirt domain logs
> (/var/lib/libvirt/qemu/<domain>.log)?
> 
> Are there any I/O error messages in there?

Not see flag "needinfo", i just guess you ask me then i tried again.
1.
# rpm -q libvirt anaconda ;uname -r 
libvirt-1.2.8-4.el7.x86_64
anaconda-19.31.94-1.el7.x86_64
3.10.0-171.el7.x86_64
virt-manager-1.1.0-3.el7.noarch

2.Try to install a domain via virt-manager, failed when disk partition.
as you said while installation fail, i check host dmesg

# dmesg |grep error
[20541.057886] traps: virt-manager[17377] trap stack segment ip:7fd82b1a0c1a sp:7fff78198a00 error:0

Result: no I/O error showing.

check the domain.log (/var/log/libvirt/qemu/rhel6.5-3.log)
# grep error rhel6.5-3.log 
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)
block I/O error in device 'drive-virtio-disk0': Permission denied (13)

Result: I/O error showing.

I will attach all the log file later.

Comment 14 zhoujunqin 2014-09-30 03:47:15 UTC
Created attachment 942580 [details]
tar file got from guest when installation failed

Comment 15 zhoujunqin 2014-09-30 03:47:57 UTC
Created attachment 942581 [details]
/var/log/libvirt/qemu/rhel6.5-3.log

Comment 16 zhoujunqin 2014-09-30 03:48:37 UTC
Created attachment 942582 [details]
the host dmesg message

Comment 17 Stefan Hajnoczi 2014-09-30 09:56:53 UTC
(In reply to zhoujunqin from comment #15)
> Created attachment 942581 [details]
> /var/log/libvirt/qemu/rhel6.5-3.log

The QEMU command-line in this log file does not correspond to your description of the issue in Comment 1.

The QEMU command-line:

-drive file=/var/lib/libvirt/images/rhel6.5-3.qcow2,if=none,id=drive-virtio-disk0,format=qcow2

Was not generated from the libvirt domain XML:

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb1'/>

Why are they different?

Based on the EPERM errors in the log, it seems the QEMU process is unable to access /var/lib/libvirt/images/rhel6.5-3.qcow2 and you need to verify the file permissions and SELinux configuration.

Comment 18 zhoujunqin 2014-10-08 03:53:57 UTC
(In reply to Stefan Hajnoczi from comment #17)
> The QEMU command-line in this log file does not correspond to your
> description of the issue in Comment 1.
   <source dev='/dev/sdb1'/>
...
> Why are they different?
> 
Yes, in fact they are different, the reason is that when i send this bug i just focus on installing a guest with disk pool and met the issue as bug described.
Later i met the issue Comment 8, i think they are the same issue, then when i reply your question, i just reproduce it in Comment 13 way.

libvirt domain XML:
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel6.5-3.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

The QEMU command-line:
-drive file=/var/lib/libvirt/images/rhel6.5-2.qcow2,if=none,id=drive-virtio-disk0,format=qcow2

> Based on the EPERM errors in the log, it seems the QEMU process is unable to
> access /var/lib/libvirt/images/rhel6.5-3.qcow2 and you need to verify the
> file permissions and SELinux configuration.

selinux-policy-3.12.1-153.el7.3.noarch
# getenforce 
Enforcing

# ll -Z /var/lib/libvirt/images/
-rw-r--r--. qemu qemu system_u:object_r:virt_content_t:s0 rhel6.5-3.qcow2

Above is about installing a guest with "default" pool storage (Comment 13), and this time i also tried with disk pool type as bug described, installation also failed.
# virsh dumpxml rhel6.5-5 |grep -A9 disk

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdc1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>

# virsh vol-dumpxml --pool sdcpool /dev/sdc1
<volume type='block'>
  <name>sdc1</name>
  <key>/dev/sdc1</key>
  <source>
    <device path='/dev/sdc'>
      <extent start='32256' end='3221257728'/>
    </device>
  </source>
  <capacity unit='bytes'>3221225472</capacity>
  <allocation unit='bytes'>3221225472</allocation>
  <target>
    <path>/dev/sdc1</path>
    <format type='none'/>
    <permissions>
      <mode>0660</mode>
      <owner>0</owner>
      <group>6</group>
      <label>system_u:object_r:fixed_disk_device_t:s0</label>
    </permissions>
    <timestamps>
      <atime>1412738467.714843354</atime>
      <mtime>1412738467.714843354</mtime>
      <ctime>1412738467.714843354</ctime>
    </timestamps>
  </target>
</volume>


The QEMU command-line:
-drive file=/dev/sdc1,if=none,id=drive-virtio
disk0,format=raw,cache=none,aio=native

Check selinux status
selinux-policy-3.13.1-2.el7.noarch
# getenforce 
Enforcing

# dmesg |grep error

I will attach the /var/log/libvirt/qemu/rhel6.5-5.log later.

Comment 19 zhoujunqin 2014-10-08 03:54:56 UTC
Created attachment 944849 [details]
/var/log/libvirt/qemu/rhel6.5-5.log

Comment 22 zhoujunqin 2014-11-13 09:47:05 UTC
(In reply to Stefan Hajnoczi from comment #21)
Hi Stefan Hajnoczi,
I have said i can reproduce this issue not only with disk pool, but also a common pool(eg.default) on my host.

[with default pool]
1.1 
# getenforce
Enforcing

Steps:
Launch virt-manager to install a guest with choosing Network Install(Http used), other configuration just as default, it will report when distribute partitions.

Guest called: rhel6.5

# ll -Z /var/lib/libvirt/images/rhel6.5.qcow2
-rw-------. qemu qemu system_u:object_r:virt_content_t:s0 /var/lib/libvirt/images/rhel6.5.qcow2

1.2
# ps -ef |grep qemu-kvm
qemu     29553     1  2 15:35 ?        00:00:21 /usr/libexec/qemu-kvm -name rhel6.5 -S -machine pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu SandyBridge -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid e9880bf7-f875-4bde-bfdb-f593a3bfe96b -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/rhel6.5.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-reboot -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -kernel /var/lib/libvirt/boot/virtinst-vmlinuz.v8gTT2 -initrd /var/lib/libvirt/boot/virtinst-initrd.img._g5UHS -append method=http://download.englab.nay.redhat.com/pub/rhel/released/RHEL-6/6.5/Server/x86_64/os/ -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/rhel6.5.qcow2,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=26 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:fe:00:b1,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -device usb-tablet,id=input0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -chardev spicevmc,id=charredir2,name=usbredir -device usb-redir,chardev=charredir2,id=redir2 -chardev spicevmc,id=charredir3,name=usbredir -device usb-redir,chardev=charredir3,id=redir3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
root     32298  5289  0 15:49 pts/2    00:00:00 grep --color=auto qemu-kvm

1.3
# dmesg  |grep error
output nothing.

1.4
Attach log named rhel6.5-20141113.tar.gz


2. disabled selinux:
# setenforce 0
# getenforce 
Permissive

2.1 try to install a guest again, guest can be installed successfully.

[with disk pool]
steps
3. disabled selinux:
# setenforce 0
# getenforce 
Permissive

3.1 try to install a guest again, guest can be installed successfully.
...
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdc1'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
...
# virsh vol-dumpxml sdc1 pooldisk 
<volume type='block'>
  <name>sdc1</name>
  <key>/dev/sdc1</key>
  <source>
    <device path='/dev/sdc'>
      <extent start='32256' end='8589966848'/>
    </device>
  </source>
  <capacity unit='bytes'>8589934592</capacity>
  <allocation unit='bytes'>8589934592</allocation>
  <target>
    <path>/dev/sdc1</path>
    <format type='none'/>
    <permissions>
      <mode>0660</mode>
      <owner>0</owner>
      <group>6</group>
      <label>system_u:object_r:fixed_disk_device_t:s0</label>
    </permissions>
    <timestamps>
      <atime>1415869929.541666574</atime>
      <mtime>1415869929.541666574</mtime>
      <ctime>1415869929.541666574</ctime>
    </timestamps>
  </target>
</volume>

4. with enable selinux
# setenforce  1
# getenforce
Enforcing

4.1
Install a guest, named rhel6.5-7, installation failed.

4.2
Will attach debug log: rhel6.5-7-diskpool.tar.gz

As a summary i will put all releated logs to one file, i think it's easy for you to check.

And my package version: selinux-policy-3.13.1-6.el7.noarch i konow it's not the latest, i just want you help me find the key problem for this bug, thanks.

Comment 23 zhoujunqin 2014-11-13 09:48:47 UTC
Created attachment 957060 [details]
totally log in bug_1138203_debug.tar.gz

Comment 24 Stefan Hajnoczi 2014-11-17 17:09:00 UTC
(In reply to zhoujunqin from comment #22)
> 2. disabled selinux:
> # setenforce 0
> # getenforce 
> Permissive
> 
> 2.1 try to install a guest again, guest can be installed successfully.

Thanks for confirming it is an SELinux issue.

I have reassigned the bug to libvirt because I don't know the details of libvirt's SELinux usage.

Comment 25 Martin Kletzander 2014-12-01 15:34:20 UTC
I find the most important log, libvirtd.log missing.  Could you set up debug logs ( http://wiki.libvirt.org/page/DebugLogs ) and reproduce the issue?  I see nothing uncommon in this scenario and it's just weird that libvirt doesn't label those files.

Comment 26 zhoujunqin 2014-12-02 03:03:35 UTC
(In reply to Martin Kletzander from comment #25)
> I find the most important log, libvirtd.log missing.  Could you set up debug
> logs ( http://wiki.libvirt.org/page/DebugLogs ) and reproduce the issue?  I
> see nothing uncommon in this scenario and it's just weird that libvirt
> doesn't label those files.

I reproduce this issue again and will attach logs you need:

libvirt-1.2.8-10.el7.x86_64
kernel-3.10.0-201.el7.x86_64
qemu-kvm-rhev-2.1.2-13.el7.x86_64
selinux-policy-3.13.1-12.el7.noarch

steps:
1. Enable selinux(we test virt-manager under this scenario usually):
# setenforce  1
# getenforce
Enforcing
2. Launch virt-manager, install a new guest named "diskpool-rhel6.6-x64".


Result: Installation failed when make partitions.

3. I will attach logs following:
xml file: diskpool-rhel6.6-x64.xml
qemu.log: diskpool-rhel6.6-x64.qemu_log
libvirtd.log: libvirtd_installation.log
guest log: diskpool-rhel6.6-x64.tar.gz

4. Disable selinux and do following steps again:
# getenforce  
Enforcing
# setenforce  0 
# getenforce  
Permissive

Result: Installation successfully.

Comment 27 zhoujunqin 2014-12-02 03:06:45 UTC
Created attachment 963479 [details]
diskpool-rhel6.6-x64.xml

Comment 28 zhoujunqin 2014-12-02 03:08:11 UTC
Created attachment 963480 [details]
diskpool-rhel6.6-x64.qemu_log

Comment 29 zhoujunqin 2014-12-02 03:10:02 UTC
Created attachment 963481 [details]
libvirtd_installation.log

Comment 30 zhoujunqin 2014-12-02 03:10:47 UTC
Created attachment 963482 [details]
diskpool-rhel6.6-x64.tar.gz

Comment 31 Martin Kletzander 2014-12-02 07:25:56 UTC
So according to the logs the file /dev/sdb1 was properly labelled:

  2014-12-02 02:36:01.837+0000: 7396: info : virSecuritySELinuxSetFileconHelper:884 : Setting SELinux context on '/dev/sdb1' to 'system_u:object_r:svirt_image_t:s0:c87,c717'
  ...
  2014-12-02 02:36:01.837+0000: 7396: info : virSecurityDACSetOwnershipInternal:243 : Setting DAC user and group on '/dev/sdb1' to '107:107'

And according to "sesearch -A -s svirt_t -t svirt_image_t | grep blk_file" it should have access for everything it needs:

  allow virt_domain svirt_image_t : blk_file { ioctl read write getattr lock append open } ;

I wonder what qemu needed to know for it to be disabled.  Would you find the error message from audit.log from the host anywhere, please?  That should be the last thing we need to solve this, unless there's some other hidden surprise anywhere.

Comment 32 zhoujunqin 2014-12-02 08:17:03 UTC
(In reply to Martin Kletzander from comment #31)
I will attach the audit.log later, anything you need please needinfo me, i hope we can fixed it asap, thanks.

Comment 33 zhoujunqin 2014-12-02 08:18:23 UTC
Created attachment 963585 [details]
the whole audit.log in my host

Comment 34 Martin Kletzander 2014-12-04 16:45:21 UTC
Well, so, libvirt changes the context properly to svirt_image_t and it does *not* change it to virt_content_t.  However it has the context virt_content_t when qemu is trying to write to it and that's why it fails.  We need to find out what changes the context when it's not libvirt.

Could you run the following command as root with systemtap and kernel-debuginfo installed (replace ASDF with the filename of the disk, e.g. /var/log/libvirt/images/rhel-something.img):

# stap -ve 'probe syscall.lsetxattr, syscall.setxattr { if (path != "\"ASDF\"") { next }; printf("\nexecname: %s\npath: %s\nname: %s\nvalue:  %s\n", execname(), path, name_str, user_string_quoted(value_uaddr)); }'

You should get output similar to the following one:

Pass 1: parsed user script and 114 library script(s) using 224496virt/41820res/3076shr/39380data kb, in 110usr/10sys/129real ms.
Pass 2: analyzed script: 2 probe(s), 6 function(s), 3 embed(s), 0 global(s) using 262236virt/80628res/4200shr/77120data kb, in 520usr/100sys/621real ms.
Pass 3: translated to C into "/tmp/stap2j6KGo/stap_ab30f868aa6d00f052ed2f24ca879ac0_5499_src.c" using 262236virt/80916res/4488shr/77120data kb, in 10usr/40sys/42real ms.
Pass 4: compiled C into "stap_ab30f868aa6d00f052ed2f24ca879ac0_5499.ko" in 1020usr/260sys/1197real ms.
Pass 5: starting run.

After you see the last line, reproduce the bug.  You should see the ouput continues with lines similar to these:

execname: libvirtd
path:  "/path/to/image"
name:  "security.selinux"
value: "system_u:object_r:svirt_image_t:s0:c87,c717"

execname: something
path:  "/path/to/image"
name:  "security.selinux"
value: "system_u:object_r:virt_content_t:s0:c87,c717"

After the issue comes up again, you can Ctrl+C the script and paste the output here (the lines with execname, path, name and value are enough).

Comment 35 zhoujunqin 2014-12-05 07:27:57 UTC
Reproduce with packages:
virt-manager-1.1.0-9.el7.noarch
kernel-3.10.0-212.el7.x86_64
kernel-debuginfo-3.10.0-212.el7.x86_64
systemd-208-18.el7.x86_64
selinux-policy-3.13.1-12.el7.noarch

Steps:
1. Check selinux status.
# getenforce 
Enforcing------------------->>enabled

2.
# stap -ve 'probe syscall.lsetxattr, syscall.setxattr { if (path != "\"/dev/sdb1\"") { next }; printf("\nexecname: %s\npath: %s\nname: %s\nvalue:  %s\n", execname(), path, name_str, user_string_quoted(value_uaddr)); }'Pass 1: parsed user script and 112 library script(s) using 224328virt/41660res/3076shr/39216data kb, in 140usr/10sys/533real ms.
Pass 2: analyzed script: 2 probe(s), 6 function(s), 3 embed(s), 0 global(s) using 262072virt/80540res/4260shr/76960data kb, in 870usr/400sys/5513real ms.
Pass 3: translated to C into "/tmp/stapOUqpcA/stap_1699a11dad7256baef19b626390c3fc3_5501_src.c" using 262072virt/80824res/4544shr/76960data kb, in 10usr/40sys/44real ms.
Pass 4: compiled C into "stap_1699a11dad7256baef19b626390c3fc3_5501.ko" in 4200usr/730sys/6532real ms.
Pass 5: starting run.

execname: libvirtd
path: "/dev/sdb1"
name: "security.selinux"
value:  "system_u:object_r:svirt_image_t:s0:c447,c762"

execname: libvirtd
path: "/dev/sdb1"
name: "security.selinux"
value:  "system_u:object_r:virt_content_t:s0"--->reproduce issue, than ctrl+c exit
^CWARNING: Number of errors: 0, skipped probes: 7
Pass 5: run completed in 10usr/50sys/244702real ms.

Is it enough, thanks.

Comment 36 Martin Kletzander 2014-12-05 08:06:46 UTC
Thank you very much for going through all the debugging.  So libvirt daemon itself changes the context.  And it looks like it is due to libguestfs.  So if you had virtlockd installed and configured, it would not happen, but default installation does not have it; but that's not the problem.  The problem is that virt-manager should not run libguestfs probes on domains that are running.  Libguestfs starts up the domain with readonly disk and the readonly disk gets virt_content_t context and hence the consequential write from qemu fails.

Moving to virt-manager (even though this could be a libguestfs problem).

Comment 37 Giuseppe Scrivano 2014-12-05 10:07:17 UTC
can you please try to reproduce the issue as:

$ sudo virsh start $YOUR_VM
$ sudo virt-viewer $YOUR_VM

(from the VM try to write to the disk, a "while sleep 1; do echo a > ~/output; done" is fine)

and from the host run this command:

$ sudo guestfish --ro -i /path/to/your/image.qcow2

Where /path/to/your/image.qcow2 is the image disk used by $YOUR_VM.

Is it failing for you?

virt-manager could check if the VM is running before attempt to mount the image through libguestfs, but that is not a correct solution as it will introduce a race condition (i.e. run the VM between the check and the mount).

If you confirm these steps cause the same issue for you, please move to libguestfs.

Comment 38 zhoujunqin 2014-12-08 08:48:11 UTC
(In reply to Giuseppe Scrivano from comment #37)

Hi Giuseppe Scrivano,
i try to reproduce it in your way, please have a look about my steps:

1. Prepare a guest with os already installed on it. 
# virsh start kvm-rhel6.6-i386 
Domain kvm-rhel6.6-i386 started

# virt-viewer  kvm-rhel6.6-i386 

2. Login guest and try to  write to the disk:
# while sleep 1; do echo a > ~/output; done

3. On host run following command:
# guestfish --ro -i /var/lib/libvirt/images/kvm-rhel6.6-i386 

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux Server release 6.6 (Santiago)
/dev/sda1 mounted on /

><fs> 

....

Result: In guest got the following message:

# while sleep 1; do echo a > ~/output; done
Message from syslogd@localhost at Dec  8 03:44:01 ...
 kernel:journal commit I/O error

bash: /root/output: Read-only file system
bash: /root/output: Read-only file system
bash: /root/output: Read-only file system
bash: /root/output: Read-only file system
bash: /root/output: Read-only file system
bash: /root/output: Read-only file system
...

please help me check it, whether it's enough to reproduce my issue, thanks.

Comment 39 Giuseppe Scrivano 2014-12-08 09:58:30 UTC
yes, the steps are enough as you are hitting the write issue in the guest.

Comment 40 Richard W.M. Jones 2014-12-09 13:29:58 UTC
I suspect this is:
https://bugzilla.redhat.com/show_bug.cgi?id=1075143
which in turn depends on a libguestfs RFE:
https://bugzilla.redhat.com/show_bug.cgi?id=1075164

Junqin: Does it still happen if you have SELinux enabled
and you uninstall the libguestfs-python package?

Comment 41 Giuseppe Scrivano 2014-12-09 14:18:22 UTC
you cannot hit this issue, AFAICS, if libguestfs-python is not installed as virt-manager won't attempt any guest inspection.

Comment 42 Giuseppe Scrivano 2014-12-09 19:25:09 UTC
what about using "<disk>....<seclabel relabel='no'/></disk>" when mounting an image through libvirt?

Comment 43 Richard W.M. Jones 2014-12-09 21:04:20 UTC
(In reply to Giuseppe Scrivano from comment #42)
> what about using "<disk>....<seclabel relabel='no'/></disk>" when mounting
> an image through libvirt?

Not sure if that was directed at me, but anyway, because we want
to use sVirt for protection against rogue disk images.  There
would be no point in having libvirt set up all the labelling for
virt-manager, if libguestfs comes along and provides a "backdoor"
for attackers.

There are ways we could pass SELinux labels around and so on, but
by far the cleanest thing is to pass the libvirt virDomainPtr.  However
it's not very easy to implement otherwise we would have done it
already.

Comment 44 Giuseppe Scrivano 2014-12-10 08:31:55 UTC
ok, I see.  As it doesn't seem there is an easy way to fix this issue, should we disable for now guests inspection altogether in virt-manager?

Comment 45 Richard W.M. Jones 2014-12-10 10:56:45 UTC
Disturbingly few people hit this bug.  It may indicate a lot of
people still disable SELinux.

Don't disable it for now.  I'll see how hard it's going to be to
fix this.

Comment 46 Richard W.M. Jones 2014-12-10 18:19:22 UTC
This is not complete, but it's a start:

https://www.redhat.com/archives/libguestfs/2014-December/thread.html#00089

Comment 47 Richard W.M. Jones 2014-12-10 21:17:10 UTC
More complete and tested implementation of the libguestfs
part of this.  Still requires a virt-manager change, but
that's trivial in comparison.

https://www.redhat.com/archives/libguestfs/2014-December/thread.html#00092

Comment 50 tingting zheng 2015-01-13 10:33:56 UTC
I can reproduce this bug with:
libvirt-1.2.8-12.el7.x86_64
virt-manager-1.1.0-11.el7.noarch
python-libguestfs-1.28.1-1.18.el7.x86_64

Tested pass with:
libvirt-1.2.8-12.el7.x86_64
virt-manager-1.1.0-12.el7.noarch
python-libguestfs-1.28.1-1.18.el7.x86_64

Steps:
1. Prepare a guest with os already installed on it. 
# virsh start kvm-rhel6.6-i386 
Domain kvm-rhel6.6-i386 started

# virt-viewer  kvm-rhel6.6-i386 

2. Login guest and try to  write to the disk:
# while sleep 1; do echo a > ~/output; done

3. On host run following command:
# guestfish --ro -i /var/lib/libvirt/images/kvm-rhel6.6-i386 

No "bash: /root/output: Read-only file system" error shows in guest as comment 38.

4.Use virt-manager to install a guest,it can be installed successfully.

Refer to the above comments,move the bug to VERIFIED.

Comment 52 errata-xmlrpc 2015-03-05 10:07:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0427.html

Comment 53 Pavel Hrdina 2015-07-15 13:48:15 UTC
*** Bug 1241037 has been marked as a duplicate of this bug. ***


Note You need to log in before you can comment on or make changes to this bug.