Bug 740478

Summary: [Regression]no output when set character device type to pipe
Product: Red Hat Enterprise Linux 6 Reporter: zhe peng <zpeng>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.2CC: acathrow, berrange, dallan, dyuan, juzhang, mzhan, rwu, vbian, veillard, ydu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.9.4-14.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 11:32:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
guest xml file none

Description zhe peng 2011-09-22 07:48:05 UTC
Created attachment 524341 [details]
guest xml file

Description of problem:
no output in testpipe.out when set character device type to pipe 

Version-Release number of selected component (if applicable):
libvirt-0.9.4-12.el6.x86_64
qemu-kvm-0.12.1.2-2.192.el6.x86_64
virt-manager-0.9.0-6.el6
python-virtinst-0.600.0-3.el6
kernel-2.6.32-197.el6.x86_64


How reproducible:
always

Steps to Reproduce:
1.installed a RHEL guest , and add the following arguments to /boot/grub/grub.conf kernel line

"console=tty0 console=ttyS0,115200"

2. create pipe files with mkfifo OR mknod

$ mkfifo /tmp/testpipe
$ mknod /tmp/testpipe p

$ mkfifo /tmp/testpipe.in
$ mknod /tmp/testpipe.in  p

$ mkfifo /tmp/testpipe.out
$ mknod /tmp/testpipe.out  p

3. Run virt-manager, Open guest virtual machine details, make sure a guest is installed and in shutdown status

4. Click Add Hardware -> Serial -> Device type as "pipe", Set the pipe path to /tmp/testpipe Then finish.

5. Start the guest

6. after a while , cat /tmp/testpipe.out

  
Actual results:
no output 

Expected results:
all of the boot process of guest are record in testpipe.out

Additional info:

libvirt-0.8.7-18.el6 with same guest worked well
so it is a regression bug

Comment 3 Daniel Veillard 2011-09-23 03:31:10 UTC
The test sounds buggy to me !
The steps you give reference only /tmp/testpipe as being given to virt-manager
The XML of the domain only reference /tmp/testpipe:

    <serial type='pipe'>
      <source path='/tmp/testpipe'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pipe'>
      <source path='/tmp/testpipe'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>

How do you expect something to magically appear in /tmp/testpipe.out ???
There is something missing in the report, I don't think that creating
.in and .out fifo named closely to /tmp/testpipe will link them to the
fifo file created for the purpose. I think the guest based on the
information you gave should output the data to /tmp/testpipe not 
/tmp/testpipe.out .

I just ran the experiment and the data actually goes to /tmp/testpipe as
expected, it's also changed ownership to qemu:qemu

Either the test is wrong, or there is something missing in the description
of the bug, please clarify !

Comment 4 zhe peng 2011-09-23 07:17:04 UTC
hi DV:
  This scenario worked well on libvirt-0.8.7-18.el6, cat /tmp/testpipe.out will get all output date,and the guest will started successful.
  I'm not sure about the relationship between testpipe and .in .out file,so i re-test this by your words:
  don't create .in and .out file,only 
 #mkfifo /tmp/demopipe
 set path to '/tmp/demopipe'
 same as you said,the boot process are record in /tmp/demopipe, but after a while the guest will hung during system start,i force off the vm , get error in /var/log/libvirt/libvirt.log:
02:59:11.898: 5027: error : qemuMonitorIORead:487 : Unable to read from monitor: Connection reset by peer
02:59:11.903: 5029: error : virSecurityDACRestoreSecurityFileLabel:143 : cannot resolve symlink /tmp/demopipe.out: No such file or directory
02:59:11.974: 5029: warning : SELinuxRestoreSecurityFileLabel:519 : cannot resolve symlink /tmp/demopipe.out: No such file or directory. 

if i create all three files,the guest can started successful,but no output in demopipe and demopipe.out file
I thought the libvirt check .in .out file automatically, please help confirm this,
thanks in advance.

Comment 5 Daniel Veillard 2011-09-23 08:41:55 UTC
No, libvirt doesn't invent anything about .out or .in files as far as
I can tell. If they are not referenced from the domain XML file they
are not touched. I created the 3 for my test but only  /tmp/testpipe
was used .
So I really don't see how creating the 3 fifo could change behaviour in any way
unless the domain XML you provided does not match the one you use.
Since the XML you provided doesn't list /tmp/demopipe.out anywhere
I can't find a logical explanation on how SELinuxRestoreSecurityFileLabel
might try to touch this file, that makes no sense to me.

Now I found out about the fact that qemu have a problem if one doesn't
read from the pipe. I think it's normal, if qemu writes to the pipe
but no process reads from it qemu will stall, and that's okay. But it
fails to recover when the data is read, which is a completely different bug
from what is reported here, and honnestly I don't think it's a libvirt
bug, maybe qemu.

Daniel

Comment 6 zhe peng 2011-09-23 09:05:40 UTC
i test this with libvirt-0.8.7-18.el6
use the same XML ,only create 1 fifo :/tmp/demopipe2, and set path='/tmp/demopipe2'
<serial type='pipe'>
      <source path='/tmp/demopipe2'/>
      <target port='0'/>
    </serial>
    <console type='pipe'>
      <source path='/tmp/demopipe2'/>
      <target type='serial' port='0'/>
    </console>
then start the guest
#virsh start rhel62_update

error: Failed to start domain rhel62_update
error: unable to set security context 'system_u:object_r:svirt_image_t:s0:c121,c241' on '/tmp/demopipe2.in': No such file or directory.

Comment 7 Daniel Veillard 2011-09-23 09:11:49 UTC
Okay, I was wrong and somehow libvirt touch those file if they exist
and qemu expects them ... but as I was pointed out, if the main file
/tmp/demopipe is present it won't look for /tmp/demopipe.in or
/tmp/demopipe.out and as I pointed out they will still be owned by root:root
and qemu won't be able to open them, which may explain why this breaks

Daniel

Comment 8 Daniel Veillard 2011-09-23 09:13:43 UTC
So what happen if you do not create /tmp/demopipe , but only
create /tmp/demopipe.in and /tmp/demopipe.out


Daniel

Comment 9 Daniel Veillard 2011-09-23 09:32:02 UTC
<armbru> DV, chardev pipe is not supported

Maybe we ought to get that test case out of libvirt regression tests for
RHEL ...

Daniel

Comment 10 zhe peng 2011-09-23 09:37:31 UTC
1: for libvirt-0.8.7-18.el6
not create /tmp/demopipe
only create:
#mkfifo /tmp/demopipe.in
#mkfifo /tmp/demopipe.out
in XML set path='/tmp/demopipe'
can get all boot process from #cat /tmp/demopipe.out

2: for libvirt-0.9.4-12.el6
not create /tmp/demopipe
only create:
#mkfifo /tmp/demopipe.in
#mkfifo /tmp/demopipe.out
in XML set path='/tmp/demopipe'
can get all boot process from #cat /tmp/demopipe.out
actually remove demopipe , all worked well.

Comment 11 Laine Stump 2011-09-23 17:44:58 UTC
The reason that it works in 0.8.7 is that libvirt *only* looks for / chowns+relabels ${name}.in and ${name}.out (completely ignoring the existence of ${name}, while in 0.9.4, it first looks for ${name} - if that exists, it chowns/relabels it and ignores ${name}.(in|out).

(That is the behavior when starting the domain. When the domain is shut down, in both the old and the new libvirt it will chown+relabel *only* ${name}.(in|out).

Although all of this code is an ugly hack to get around the "strange" way that qemu has of passing 2 pipes via a single commandline argument, still it is an incorrect hack. qemu will first attempt to use a separate pipe for each direction via ${name}.(in|out), and only if those do not exist it will attempt to do a single bidirectional fifo. If both are present, it's unfortunately not up to libvirt to decide which of the fifos will be used; that decision is made completely by qemu.

So, we have two possibilities in fixing this:


1) First look for ${name}.(in|out) and if they exist, chown+relabel them and ignore ${name} (this assumes that qemu's algorithm for choosing will not change)

or

2) Always attempt to chown+relabel all three fifos if they exist, and log an error if neither in+out or ${name} exists (or, for example, if the in fifo exists but not out, or vice versa). This could do some extra relabeling work on a file that won't be used, but will ensure that we operate properly no matter what qemu decides to do in the future.

Anyone have a preference? If not, I'll implement (2).

(BTW, I tried using a bidirectional fifo on RHEL6.2 and it worked right up until the kernel loaded the 8250/16550 driver, then locked up the entire guest.)

Comment 12 Daniel Berrangé 2011-09-27 14:30:29 UTC
If both the single pipe and the .in/.out pipes are present, libvirt should choose them in the same order that QEMU chooses them. In other words we need to reverse our current order I believe. There's no need to label all 3 possible pipes, if we follow the correct ordering.

Comment 13 Laine Stump 2011-09-28 06:37:38 UTC
I posted a fix (using option (1) for this problem upstream, awaiting ACK:

https://www.redhat.com/archives/libvir-list/2011-September/msg01094.html

Note that the patch not only reverses the order of checking of the Set method for both the dac and selinux drivers, but also fixes the Restore method to check both (it hadn't ever been updated to look for the bidirectional fifo at all).

Comment 14 Laine Stump 2011-09-28 13:45:30 UTC
The patch in Comment 13 has been committed upstream (with a minor change in logic based on review), and posted to rhvirt-patches for inclusion in the RHEL6 libvirt build:

  http://post-office.corp.redhat.com/archives/rhvirt-patches/2011-September/msg00903.html

commit 46e8dc710a03312e2744fec324937c00effaeec8
Author: Laine Stump <laine>
Date:   Tue Sep 27 14:04:53 2011 -0400

    security: properly chown/label bidirectional and unidirectional fifos

Comment 17 Vivian Bian 2011-09-29 06:46:26 UTC
tested with 
libvirt-0.9.4-14.el6.x86_64
qemu-kvm-0.12.1.2-2.192.el6.x86_64
kernel-2.6.32-197.el6.x86_64

Steps:
1.installed a RHEL guest , and add the following arguments to
/boot/grub/grub.conf kernel line

"console=tty0 console=ttyS0,115200"

2. create pipe files with mkfifo OR mknod

$ mkfifo /tmp/testpipe
$ mknod /tmp/testpipe p

$ mkfifo /tmp/testpipe.in
$ mknod /tmp/testpipe.in  p

$ mkfifo /tmp/testpipe.out
$ mknod /tmp/testpipe.out  p

3. Run virt-manager, Open guest virtual machine details, make sure a guest is
installed and in shutdown status

4. Click Add Hardware -> Serial -> Device type as "pipe", Set the pipe path to
/tmp/testpipe Then finish.

5. Start the guest

6. after a while , cat /tmp/testpipe.out

Actual result:
There is the output in /tmp/testpipe.out . Set bug status to VERIFIED

Comment 18 errata-xmlrpc 2011-12-06 11:32:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html