Bug 439279 - Fully-virtualized guests are able to write to read-only disks
Fully-virtualized guests are able to write to read-only disks
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen (Show other bugs)
5.1
All Linux
medium Severity medium
: rc
: 5.6
Assigned To: Michal Novotny
Virtualization Bugs
:
Depends On:
Blocks: 449772 514500
  Show dependency treegraph
 
Reported: 2008-03-27 17:06 EDT by John Ruemker
Modified: 2014-02-02 17:36 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
For fully-virtualized Xen guests, previous versions of Red Hat Enterprise Linux allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk. Starting with Red Hat Enterprise Linux 5.6, SCSI disks properly report the read-only information, while read-only IDE disks are not allowed (in case of such a configuration, the guest will refuse to start). Migrations and restores of guests started under older versions of Xen have a different behavior to avoid backward incompatibility. SCSI disks will start reporting write errors after successful restore or migration, while for IDE disks the old (incorrect) behaviour will be preserved.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2011-01-13 17:16:03 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Pass readonly flag through to QEMU block backend drivers (13.37 KB, patch)
2008-07-11 08:30 EDT, Daniel Berrange
no flags Details | Diff
Fix to make HVM guests honor the read-only flag (5.48 KB, patch)
2010-06-03 13:09 EDT, Michal Novotny
no flags Details | Diff
Fix to make HVM guests honor the read-only flag v4 (11.99 KB, patch)
2010-06-23 10:42 EDT, Michal Novotny
no flags Details | Diff
Patch v5 (12.26 KB, patch)
2010-07-28 09:40 EDT, Michal Novotny
no flags Details | Diff
scsi disk ro after restored from xen-115 (11.80 KB, text/plain)
2010-09-27 11:16 EDT, Pengzhen Cao
no flags Details
scsi disk ro newly created on xen-116 (60.57 KB, text/plain)
2010-09-27 11:17 EDT, Pengzhen Cao
no flags Details

  None (edit)
Description John Ruemker 2008-03-27 17:06:17 EDT
Description of problem:  Xen does not respect the read-only (r) flag on disks
defined in the guest configuration for fully virtualized domains.  Setting this
on a block-device backed or image backed disk on a paravirt guest works fine and
will prevent the guest from writing to that disk.  Setting it on either type of
disk in a full-virt guest does not block writes from the guest.  

Version-Release number of selected component (if applicable): xen-3.0.3-41.el5
(have also confirmed on 5.2beta version of xen)

How reproducible:  Always

Steps to Reproduce:
1.  Create a full virt and para virt RHEL 5 guest and shutdown after installation
2.  Edit the config file in /etc/xen/ for both domains, adding a new disk.  I
tried both of the following disk lines (on separate occasions) for
paravirtualized guests

   disk = [ "tap:aio:/var/lib/xen/images/rhel5_pv1.img,xvda,w",
"tap:aio:/var/lib/xen/images/rhel5_pv1-disk2.img,xvdb,r" ]

   disk = [ "tap:aio:/var/lib/xen/images/rhel5_pv1.img,xvda,w",
"phy:/dev/vg1/rhel5n1-rodisk,xvdb,r" ]


And tried the following on full-virt:

   disk = [ "file:/var/lib/xen/images/rhel5_n1.img,hda,w",
"file:/var/lib/xen/images/rhel5_n1-disk2.img,sda,r" ]
   disk = [ "file:/var/lib/xen/images/rhel5_n1.img,hda,w",
"phy:/dev/vg1/rhel5n1-rodisk,sda,r" ]



3.  Start up both guests and dd some data to the new disk.   
  
Actual results: On paravirt, the dd fails:

   # dd if=/dev/zero of=/dev/xvdb bs=1 count=10

   dd: writing `/dev/xvdb': Operation not permitted

   1+0 records in

   0+0 records out

   0 bytes (0 B) copied, 0.000874 seconds, 0.0 kB/s


On full-virt, it goes through without error and the data can even be read back
off the device

   # dd if=/etc/samba/smb.conf of=/dev/sda 
   # less -f /dev/sda

Expected results:  The dd should fail on both full-virt and para-virt.  

Additional info:  None
Comment 1 Daniel Berrange 2008-07-09 09:06:32 EDT
While the internal QEMU block drivers have a concept of a 'read only' disk, they
only set the read-only flag if the underlying file/device permissions are
read-only. There's no way to set it based on the Xen configuration. XenStore has
the neccessary info, so its primarily an issue of passing the read-only flag
down into the backend driver.
Comment 2 RHEL Product and Program Management 2008-07-09 09:19:14 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 4 Daniel Berrange 2008-07-11 08:30:53 EDT
Created attachment 311567 [details]
Pass readonly flag through to QEMU block backend drivers

This is a proof of concept patch to make QEMU's block driver honour the
read-only flag on block devices.
Comment 5 Daniel Berrange 2008-09-08 06:14:13 EDT
Upstream rejected this patch and want a completely different approach to dealing with this problem. I very much doubt I'll get a re-written patch that's suitable for upstream, in time for RHEL-5.3, so we'll probably have to re-target this for 5.4
Comment 6 Daniel Berrange 2009-02-11 06:51:53 EST
Still no upstream agreement on this, so putting back to ASSIGNED.
Comment 8 Michal Novotny 2010-04-20 07:41:18 EDT
(In reply to comment #6)
> Still no upstream agreement on this, so putting back to ASSIGNED.    

It appears that upstream doesn't have the fix done this way if it has some read-only file handling applied at all. Unfortunately, I can't still even boot xen-unstable because of some kernel panic. Although if there is no upstream agreement I think we may apply this fix since the code-base is different. Unfortunately, like stated, I can't even boot the upstream xen-unstable to confirm it's still an issue here.

Michal
Comment 10 Michal Novotny 2010-06-03 13:09:01 EDT
Created attachment 419450 [details]
Fix to make HVM guests honor the read-only flag

Hi,
this is the patch to fix the read-only image file handling since the image file was always treated as read-write which means that all the HVM guests were able to write to all the disk images available in domain configuration file no matter what the mode of the image was defined. This patch fixes this functionality to honor the O_RDONLY in the BDRV_O_ACCESS flag in block.c and also fixes the IDE and SCSI interfaces that uses it.

It's been tested on RHEL-5 x86_64 dom0 and with this patch applied I was able to write to a read-write disk but not to read-only disk so it was working fine.

Just a note: when you want to mount an image that is set as read-only in the domain configuration file but you omit to set mode to read-only it results into I/O errors when processing the requests. Remounting as read-only or unmounting and remounting using the `mount /dev/* /path/to/mount -o ro` shall do the mounting the correct way, i.e. with no I/O errors, so make sure you mount those disks as read-only otherwise you can be getting errors like:

end_request: I/O error, dev hdb, sector 52
Buffer I/O error on device hdb1, logical block 1
lost page write due to I/O error on hdb1

and for IDE devices you'll be getting several additional DeviceFault errors since mounting the device read-write (default setting) writes some data onto a disk at the mount-time. 

For SCSI devices the DATA PROTECT request sense has been added
as found at: http://en.wikipedia.org/wiki/SCSI_Request_Sense_Command

The very same patch has been send upstream for review.

Michal
Comment 13 Michal Novotny 2010-06-23 10:42:13 EDT
Created attachment 426287 [details]
Fix to make HVM guests honor the read-only flag v4

This is the new patch to also allow read-only IDE CD-ROM drives since they should be allowed according to the specifications.

Michal
Comment 14 Michal Novotny 2010-06-23 11:08:50 EDT
I also added a technical comments which should be added as release notes since this is the new feature implemented.

Michal
Comment 15 Michal Novotny 2010-06-23 11:08:50 EDT
Technical note added. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

New Contents:
For new guest creation the read-only IDE disks will be prohibited and the guest will refuse to start but for migrations and restores from the older version of Xen the old (incorrect) behaviour will be preserved which means that the guest could be restored or migrated successfully.
Comment 16 Paolo Bonzini 2010-06-23 11:14:00 EDT
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,3 @@
-For new guest creation the read-only IDE disks will be prohibited and the guest will refuse to start but for migrations and restores from the older version of Xen the old (incorrect) behaviour will be preserved which means that the guest could be restored or migrated successfully.+Previous versions of Xen allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk.  Starting with RHEL 5.6, SCSI disks will properly report the read-only bit, while read-only IDE disks are not allowed---in the presence of such a configuration, the guest will refuse to start.
+
+Migrations and restores of guests started under older version of Xen have a different behavior to avoid backward incompatibility.  SCSI disks will start reporting write errors after successful restore or migration, while for IDE disks the old (incorrect) behaviour will be preserved.
Comment 17 Paolo Bonzini 2010-06-23 11:17:33 EDT
Technical note updated. If any revisions are required, please edit the "Technical Notes" field
accordingly. All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,3 +1,3 @@
-Previous versions of Xen allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk.  Starting with RHEL 5.6, SCSI disks will properly report the read-only bit, while read-only IDE disks are not allowed---in the presence of such a configuration, the guest will refuse to start.
+For fully-virtualized Xen guests, Previous versions of RHEL allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk.  Starting with RHEL 5.6, SCSI disks will properly report the read-only bit, while read-only IDE disks are not allowed---in the presence of such a configuration, the guest will refuse to start.
 
 Migrations and restores of guests started under older version of Xen have a different behavior to avoid backward incompatibility.  SCSI disks will start reporting write errors after successful restore or migration, while for IDE disks the old (incorrect) behaviour will be preserved.
Comment 18 Michal Novotny 2010-07-28 09:40:44 EDT
Created attachment 435010 [details]
Patch v5

Well, this is the new version of the patch implementing the silent change of read-only IDE disk flag to read-write flag on restore/migrate when the HVM guest was using the read-only IDE disk. Basically using the read-only IDE disk is basically a misconfiguration since it's not supported by specifications.

Michal
Comment 22 Pengzhen Cao 2010-09-27 11:16:38 EDT
Created attachment 449921 [details]
scsi disk ro after restored from xen-115
Comment 23 Pengzhen Cao 2010-09-27 11:17:27 EDT
Created attachment 449922 [details]
scsi disk ro newly created on xen-116
Comment 24 Pengzhen Cao 2010-09-27 11:18:56 EDT
I tried with rhel6 64bit guest.

1. Create rhel6 HVM guest1  with IDE read-only on xen-115 build without the fix. Read-only limit is not working and expected. Save the guest
2. Create rhel6 HVM guest2  with SCSI read-only on xen-115 build without the fix. Read-only limit is not working and expected. Save the guest

Then, update kernel-xen to latest and xen to 116 build with the fix.

3. Restore guest1 from saved state,  Read-only limit is not working and expected.
Then start the guest again and failed with expected error message saying "IDE read-only not supported"

4. Restore guest2 from saved state,  try to write on the read-only disk but FAILED  and there is some error log in guest's /var/log/messages. 
attachment(attachment 449921 [details])

5. Create guest2 with SCSI read-only again, this time there is no error message in guest and saying the disk is read-only protected.(attachment 449922 [details])

Now, I am wonder if it is OK with the error message in step 4?
Comment 25 Paolo Bonzini 2010-09-28 03:36:19 EDT
In the SCSI case, the journal will look as if there was a power failure after the restore, but otherwise it will work.  So the aborted transactions are okay.

The warning in fs/buffers.c is a separate bug in RHEL6.  I think it's harmless, since whatever happens won't be reflected on the disk (which is read only) but it should be reported.

Thanks!
Comment 26 Pengzhen Cao 2010-09-28 08:52:18 EDT
OK, then I think I can mark this bug verified based on comment 25.
Did same test on xen i386 build with 32bit guest, read only patch is also working fine.

For the error message in attachment 449921 [details] , there is similar bugs reported:
rhbz614206, rhbz591466
Comment 27 Martin Prpic 2010-12-10 10:14:26 EST
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1,3 +1 @@
-For fully-virtualized Xen guests, Previous versions of RHEL allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk.  Starting with RHEL 5.6, SCSI disks will properly report the read-only bit, while read-only IDE disks are not allowed---in the presence of such a configuration, the guest will refuse to start.
+For fully-virtualized Xen guests, previous versions of Red Hat Enterprise Linux allowed the creation of read-only emulated disks (IDE or SCSI), but ignored the read-only attribute and let the guest write to the disk. Starting with Red Hat Enterprise Linux 5.6, SCSI disks properly report the read-only information, while read-only IDE disks are not allowed (in case of such a configuration, the guest will refuse to start). Migrations and restores of guests started under older versions of Xen have a different behavior to avoid backward incompatibility. SCSI disks will start reporting write errors after successful restore or migration, while for IDE disks the old (incorrect) behaviour will be preserved.-
-Migrations and restores of guests started under older version of Xen have a different behavior to avoid backward incompatibility.  SCSI disks will start reporting write errors after successful restore or migration, while for IDE disks the old (incorrect) behaviour will be preserved.
Comment 29 errata-xmlrpc 2011-01-13 17:16:03 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2011-0031.html

Note You need to log in before you can comment on or make changes to this bug.