Bug 729586 - xen: fix drive naming
xen: fix drive naming
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.3
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Laszlo Ersek
Boris Ranto
: EC2, ZStream
: 734794 (view as bug list)
Depends On:
Blocks: 767187 523117 705572 818505
  Show dependency treegraph
 
Reported: 2011-08-10 05:14 EDT by Andrew Jones
Modified: 2014-01-20 05:52 EST (History)
19 users (show)

See Also:
Fixed In Version: kernel-2.6.32-229.el6
Doc Type: Bug Fix
Doc Text:
Red Hat Enterprise Linux 6.1 introduced naming scheme adjustments for emulated SCSI disks used with paravirtual drivers to prevent namespace clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device 'xvd'. Consider the example below: | 6.0 | 6.1 or later emulated IDE | hda -> xvda | unchanged emulated SCSI | sda -> xvda | sda -> xvde, sdb -> xvdf, ... This update introduces a new module parameter, xen_blkfront.sda_is_xvda, that provides a seamless upgrade path from 6.0 to 6.3 kernel release. The default value of xen_blkfront.sda_is_xvda is 0 and it keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to the 6.0-compatible mode. Note that when upgrading from 6.0 to 6.3 release, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers and uses explicit disk names such as xvd[a-d], it is advised to add the xen_blkfront.sda_is_xvda=1 parameter to the kernel command line before performing the upgrade.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-20 03:48:02 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Andrew Jones 2011-08-10 05:14:50 EDT
Bug 729340 comment 2 points out a patch that will restore the drive naming on xen guests to start from 'a', instead of 'e'. We should consider this patch for RHEL6, as long as it doesn't potentially appear to regress current configs.

Bug 729340 says the patch is for 3.1, but I didn't find it in a latest pull of Linus' tree.

commit 196cfe2ae8fcdc03b3c7d627e7dfe8c0ce7229f9
Author: Stefan Bader <stefan.bader@canonical.com>
Date:   Thu Jul 14 15:30:22 2011 +0200

    xen-blkfront: Drop name and minor adjustments for emulated scsi devices

    These were intended to avoid the namespace clash when representing
    emulated IDE and SCSI devices. However that seems to confuse users
    more than expected (a disk defined as sda becomes xvde).
    So for now go back to the scheme which does no adjustments. This
    will break when mixing IDE and SCSI names in the configuration of
    guests but should be by now expected.

    Acked-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
    Signed-off-by: Stefan Bader <stefan.bader@canonical.com>
    Signed-off-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Comment 1 Andrew Jones 2011-09-02 04:42:32 EDT
I don't believe this patch is necessary for RHEL6. While we do have this patch

c80a420995e721099906607b07c09a24543b31d9:
Author: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Date:   Thu Dec 2 17:55:00 2010 +0000

    xen-blkfront: handle Xen major numbers other than XENVBD

at least when running on RHEL5 Xen's qemu we shouldn't need to worry about the naming issue for scsi devices. The patch in question ("Drop name and..."), which is just

diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 0b3e3b3..3c0b00a 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -118,8 +118,8 @@ static DEFINE_SPINLOCK(blkif_io_lock);
 #define BLKIF_MINOR_EXT(dev) ((dev)&(~EXTENDED))
 #define EMULATED_HD_DISK_MINOR_OFFSET (0)
 #define EMULATED_HD_DISK_NAME_OFFSET (EMULATED_HD_DISK_MINOR_OFFSET / 256)
-#define EMULATED_SD_DISK_MINOR_OFFSET (EMULATED_HD_DISK_MINOR_OFFSET + (4 * 16)
-#define EMULATED_SD_DISK_NAME_OFFSET (EMULATED_HD_DISK_NAME_OFFSET + 4)
+#define EMULATED_SD_DISK_MINOR_OFFSET (0)
+#define EMULATED_SD_DISK_NAME_OFFSET (EMULATED_SD_DISK_MINOR_OFFSET / 256)

doesn't hurt anything when running on RHEL5 Xen's qemu either though, so I'm not opposed to it if it fixes a problem when running on upstream Xen. I need confirmation that there is a problem, and then test results showing the patch actually fixes the problem before posting it though.
Comment 2 Jay Greguske 2011-09-02 12:06:25 EDT
I tested a patched kernel in EC2 and this appears to work.

kernel-2.6.32-192.el6729586.i686.rpm

Before:
[root@domU-12-31-39-0B-25-11 ~]# ls /dev/xv*
/dev/xvde1  /dev/xvde3  /dev/xvdj
[root@domU-12-31-39-0B-25-11 ~]# swapon -a
swapon: /dev/xvda3: stat failed: No such file or directory

After:
[root@domU-12-31-39-0B-25-11 ~]# ls /dev/xv*
/dev/xvda1  /dev/xvda3  /dev/xvdf
[root@domU-12-31-39-0B-25-11 ~]# swapon -a
[root@domU-12-31-39-0B-25-11 ~]#

Any guest started in EC2 will be using emulated scsi devices, so I question whether this problem can be ignored until 6.3 in the cloud space.
Comment 3 Andrew Jones 2011-09-05 04:37:10 EDT
Hmm, I don't know what to do with this. I think the real issue is whether or not a guest's /etc/fstab is using disk LABELs. If it isn't then changing the name (whether it from from a to e or e to a) is going to mess up the guest until the fstab is corrected. If the EC2 instances just now being created can be created to use labels instead, then I think 'swapon -a' would work. Not making this change this would also avoid breaking anybody who installed a guest without using labels and got used to the e naming.

Maybe I'm wrong about this being a LABEL issue or something correctable on EC2 instance creation. If I'm wrong, please educate me. If I'm not, then I'm not sure if we want to risk breaking other deployments.
Comment 4 Kevin Stange 2011-09-28 18:49:52 EDT
There was effective breakage between kernel-2.6.32-71.29.1.el6 and kernel-2.6.32-131.el6.  When going from 6.0 to 6.1, the result is that if your Xen domain configuration file specified sda1 as a device name, it was previously renamed to xvda1.  After 2.6.32-131.el6, the device is named xvde1 instead (because the names xvda - xvdd are reserved for hda - hdd device remapping).

In situations where the configuration file explicitly lists "xvda1" or uses "hda1", "xvda1" continues to work.  This therefore breaks a previously working use case and can cause EL 6.0 guests to stop booting when upgraded to EL 6.1.

I would advocate reverting to restore the 6.0 behavior, but this would potentially cause the same issue for anyone moving from 6.1 or later to an updated kernel which restores the old behavior.
Comment 5 Andrew Jones 2011-09-29 03:37:52 EDT
(In reply to comment #4)
> I would advocate reverting to restore the 6.0 behavior, but this would
> potentially cause the same issue for anyone moving from 6.1 or later to an
> updated kernel which restores the old behavior.

Right, that's the crux of this issue. We're now in a bit of a pickle where we've got two minor releases out with two different naming schemes. To complicate matters, the issue wasn't detected until just recently, which was pretty late to be considered for fixing it in the next minor release.

Just to think out loud a bit: I'm guessing that 6.1 is actually more widely deployed than 6.0 (considering general wariness of .0 releases and that pv-on-hvm was off by default for 6.0). Plus, if we stay with the 'e' naming for 6.2 as well, then there will no doubt be even more chance of breakage when switching back to the 'a' naming in 6.3. So, it would appear we shouldn't change it at all. However, upstream (which we followed into the first name change) has now changed it back to the 'a' naming with the patch referenced above. So, if RHEL is to stay consistent with other Linux distributions, then it should change back as well.

I'll do some more head scratching on this issue.
Comment 9 Paolo Bonzini 2011-09-29 05:50:03 EDT
I think this is a false problem.  It seems extremely unlikely that your configuration says sda1 and your fstab says xvda1 (or xvde1).  It is much much more likely that your using labels or uuids, and thus nothing breaks even though your device names change under the hood.  It _is_ possible that other utilities are storing real device names somewhere, of course, but then it is also very unlikely that this is breaking boot.

Supporting sdXX/hdYY in pv-ops (i.e. RHEL6) configurations is strictly a hack.  We probably should have never supported anything but xvdXX in RHEL6; we should have disabled sdXX/hdYY in RHEL6.0.

I'd be wary of changing this stuff.  If anything, we could add a loud syslog warning if anybody uses sdXX/hdYY device names in a PV configuration.
Comment 10 Kevin Stange 2011-09-29 12:39:44 EDT
Paolo,

I've been using OnApp where this is exactly how things are currently set up.  They are going to change the software to avoid this problem in the future, but it does in fact break boot at this time.  From my reading this problem also affected some fairly common SolusVM setups[1] and a number of Fedora VMs in Amazon EC2 until Bug 729340 was fixed.  Obviously labels are a good way to go, but it's not a false problem and does affect real-world implementations.

[1] http://stacklet.com/story/centos/6.0/solusvm-notes-xvda
Comment 11 Paolo Bonzini 2011-09-29 16:44:23 EDT
Still, changing things twice (which also breaks things that were fixed by the original change) is probably the only thing worse than changing them once.

We should create *now* a RHEL7 bug about prohibiting sdXX/hdYY, or perhaps even disallow it upstream.
Comment 12 Jay Greguske 2011-10-05 10:28:51 EDT
This bug is the cause of 705572 and 734794 because other parts of the system configuration is expecting xvda. I would really like to see this fixed.
Comment 13 Laszlo Ersek 2011-10-06 04:28:31 EDT
What about turing EMULATED_SD_DISK_MINOR_OFFSET and EMULATED_SD_DISK_NAME_OFFSET into module-private variables, initialized at module load time by a boolean flag:

modparam: adjust_emulated_scsi

If true:

emulated_sd_disk_minor_offset = EMULATED_HD_DISK_MINOR_OFFSET + (4 * 16);
emulated_sd_disk_name_offset = EMULATED_HD_DISK_NAME_OFFSET + 4;

if false:

emulated_sd_disk_minor_offset = 0;
emulated_sd_disk_name_offset = emulated_sd_disk_minor_offset / 256;

I'd default "adjust_emulated_scsi" to "false". That's the broken behavior ("This will break when mixing IDE and SCSI names in the configuration of guests"), but people seem to depend on it in hard-to-change places.

Thoughts?
Comment 14 Laszlo Ersek 2011-10-06 04:28:58 EDT
(Everything can be solved by another layer of indirection.)
Comment 15 Andrew Jones 2011-10-06 05:00:29 EDT
(In reply to comment #13)
> modparam: adjust_emulated_scsi

Not a bad idea, however it would still require installers to change something, the modprobe.d conf file. So maybe they should just change to using labels instead. That said, if it's too difficult or risky to try and find all the hardcoded xvdas in the installers, then maybe giving the option to just make one change in a modprobe.d conf file would be a nice "solution".
Comment 27 Mike McGrath 2011-10-18 17:21:47 EDT
Just adding a use case:

We're automatically building machines in ec2 and attaching block devices.  Ours scripts no longer work as the device letters on the guest don't match what ec2 is using so we aren't getting to the stage where we can even label a device for reference.
Comment 28 RHEL Product and Program Management 2011-12-12 23:38:10 EST
This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.
Comment 29 Jay Greguske 2011-12-13 10:36:50 EST
*** Bug 734794 has been marked as a duplicate of this bug. ***
Comment 30 Aristeu Rozanski 2012-02-10 14:29:22 EST
Patch(es) available on kernel-2.6.32-229.el6
Comment 36 Chris Fordham 2012-06-11 22:13:49 EDT
@Aristeu Rozanski where is this kernel available publicly, I can't find it?
Comment 37 Igor Mammedov 2012-06-13 11:55:03 EDT
    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Release 6.1 introduced naming scheme adjustments for SCSI disks when they used with paravirtual drivers to prevent name-space clashes between IDE and  SCSI disks. i.e. 6.0 naming scheme  hda -> xvda, sda -> xvda and since 6.1 SCSI disks are mapped starting from sda -> xvde, IDE mapping stays unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda to provide a seamless upgrade path from 6.0 to 6.3 kernel release. Default value of xen_blkfront.sda_is_xvda is 0, it keeps naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, naming scheme reverts to 6.0 compatible mode. In the case of upgrade from 6.0 to 6.3 release, if customer specifies disks in VM configuration as SCSI and guest utilizes paravirtual drivers and uses explicit disk names (i.e. xvd[a-d]) it is advised to add parameter xen_blkfront.sda_is_xvda=1 to current kernel command line before performing upgrade.
Comment 38 Andrew Jones 2012-06-14 03:10:48 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Release 6.1 introduced naming scheme adjustments for SCSI disks when they used with paravirtual drivers to prevent name-space clashes between IDE and  SCSI disks. i.e. 6.0 naming scheme  hda -> xvda, sda -> xvda and since 6.1 SCSI disks are mapped starting from sda -> xvde, IDE mapping stays unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda to provide a seamless upgrade path from 6.0 to 6.3 kernel release. Default value of xen_blkfront.sda_is_xvda is 0, it keeps naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, naming scheme reverts to 6.0 compatible mode. In the case of upgrade from 6.0 to 6.3 release, if customer specifies disks in VM configuration as SCSI and guest utilizes paravirtual drivers and uses explicit disk names (i.e. xvd[a-d]) it is advised to add parameter xen_blkfront.sda_is_xvda=1 to current kernel command line before performing upgrade.+Release 6.1 introduced naming scheme adjustments for emulated SCSI disks when they are used with paravirtual drivers to prevent name-space clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device xvd. For 6.0, if a config had both an hda and an sda, then they were both mapped to xvda. Since 6.1 emulated SCSI disks are mapped starting from xvde, i.e. sda -> xvde, sdb -> xvdf, and on. The IDE mapping remains unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda that provides a seamless upgrade path from 6.0 to 6.3. The default value of xen_blkfront.sda_is_xvda is 0, which keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to 6.0 compatible mode. In the case of an upgrade from 6.0 to 6.3, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers with explicit disk names (i.e. xvd[a-d]), then it is advised to add the parameter xen_blkfront.sda_is_xvda=1 to kernel command line before performing the upgrade.
Comment 39 Tomas Capek 2012-06-14 06:49:34 EDT
    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1,10 @@
-Release 6.1 introduced naming scheme adjustments for emulated SCSI disks when they are used with paravirtual drivers to prevent name-space clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device xvd. For 6.0, if a config had both an hda and an sda, then they were both mapped to xvda. Since 6.1 emulated SCSI disks are mapped starting from xvde, i.e. sda -> xvde, sdb -> xvdf, and on. The IDE mapping remains unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda that provides a seamless upgrade path from 6.0 to 6.3. The default value of xen_blkfront.sda_is_xvda is 0, which keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to 6.0 compatible mode. In the case of an upgrade from 6.0 to 6.3, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers with explicit disk names (i.e. xvd[a-d]), then it is advised to add the parameter xen_blkfront.sda_is_xvda=1 to kernel command line before performing the upgrade.+Red Hat Enterprise Linux 6.1 introduced naming scheme adjustments for emulated SCSI disks used with paravirtual drivers to prevent namespace clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device 'xvd'. Consider the example below:
+
+
+    	                | 6.0 	        | 6.1 or later
+    emulated IDE 	| hda -> xvda 	| unchanged
+    emulated SCSI 	| sda -> xvda 	| sda -> xvde, sdb -> xvdf, ...
+
+    This update introduces a new module parameter, xen_blkfront.sda_is_xvda, that provides a seamless upgrade path from 6.0 to 6.3 kernel release. The default value of xen_blkfront.sda_is_xvda is 0 and it keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to the 6.0-compatible mode.
+
+    Note that when upgrading from 6.0 to 6.3 release, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers and uses explicit disk names such as xvd[a-d], it is advised to add the xen_blkfront.sda_is_xvda=1 parameter to the kernel command line before performing the upgrade.
Comment 41 errata-xmlrpc 2012-06-20 03:48:02 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html

Note You need to log in before you can comment on or make changes to this bug.