Bug 729586

I don't believe this patch is necessary for RHEL6. While we do have this patch

c80a420995e721099906607b07c09a24543b31d9:
Author: Stefano Stabellini <stefano.stabellini.com>
Date:   Thu Dec 2 17:55:00 2010 +0000

    xen-blkfront: handle Xen major numbers other than XENVBD

at least when running on RHEL5 Xen's qemu we shouldn't need to worry about the naming issue for scsi devices. The patch in question ("Drop name and..."), which is just

diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index 0b3e3b3..3c0b00a 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -118,8 +118,8 @@ static DEFINE_SPINLOCK(blkif_io_lock);
 #define BLKIF_MINOR_EXT(dev) ((dev)&(~EXTENDED))
 #define EMULATED_HD_DISK_MINOR_OFFSET (0)
 #define EMULATED_HD_DISK_NAME_OFFSET (EMULATED_HD_DISK_MINOR_OFFSET / 256)
-#define EMULATED_SD_DISK_MINOR_OFFSET (EMULATED_HD_DISK_MINOR_OFFSET + (4 * 16)
-#define EMULATED_SD_DISK_NAME_OFFSET (EMULATED_HD_DISK_NAME_OFFSET + 4)
+#define EMULATED_SD_DISK_MINOR_OFFSET (0)
+#define EMULATED_SD_DISK_NAME_OFFSET (EMULATED_SD_DISK_MINOR_OFFSET / 256)

doesn't hurt anything when running on RHEL5 Xen's qemu either though, so I'm not opposed to it if it fixes a problem when running on upstream Xen. I need confirmation that there is a problem, and then test results showing the patch actually fixes the problem before posting it though.

I tested a patched kernel in EC2 and this appears to work.

kernel-2.6.32-192.el6729586.i686.rpm

Before:
[root@domU-12-31-39-0B-25-11 ~]# ls /dev/xv*
/dev/xvde1  /dev/xvde3  /dev/xvdj
[root@domU-12-31-39-0B-25-11 ~]# swapon -a
swapon: /dev/xvda3: stat failed: No such file or directory

After:
[root@domU-12-31-39-0B-25-11 ~]# ls /dev/xv*
/dev/xvda1  /dev/xvda3  /dev/xvdf
[root@domU-12-31-39-0B-25-11 ~]# swapon -a
[root@domU-12-31-39-0B-25-11 ~]#

Any guest started in EC2 will be using emulated scsi devices, so I question whether this problem can be ignored until 6.3 in the cloud space.

Hmm, I don't know what to do with this. I think the real issue is whether or not a guest's /etc/fstab is using disk LABELs. If it isn't then changing the name (whether it from from a to e or e to a) is going to mess up the guest until the fstab is corrected. If the EC2 instances just now being created can be created to use labels instead, then I think 'swapon -a' would work. Not making this change this would also avoid breaking anybody who installed a guest without using labels and got used to the e naming.

Maybe I'm wrong about this being a LABEL issue or something correctable on EC2 instance creation. If I'm wrong, please educate me. If I'm not, then I'm not sure if we want to risk breaking other deployments.

There was effective breakage between kernel-2.6.32-71.29.1.el6 and kernel-2.6.32-131.el6.  When going from 6.0 to 6.1, the result is that if your Xen domain configuration file specified sda1 as a device name, it was previously renamed to xvda1.  After 2.6.32-131.el6, the device is named xvde1 instead (because the names xvda - xvdd are reserved for hda - hdd device remapping).

In situations where the configuration file explicitly lists "xvda1" or uses "hda1", "xvda1" continues to work.  This therefore breaks a previously working use case and can cause EL 6.0 guests to stop booting when upgraded to EL 6.1.

I would advocate reverting to restore the 6.0 behavior, but this would potentially cause the same issue for anyone moving from 6.1 or later to an updated kernel which restores the old behavior.

(In reply to comment #4)
> I would advocate reverting to restore the 6.0 behavior, but this would
> potentially cause the same issue for anyone moving from 6.1 or later to an
> updated kernel which restores the old behavior.

Right, that's the crux of this issue. We're now in a bit of a pickle where we've got two minor releases out with two different naming schemes. To complicate matters, the issue wasn't detected until just recently, which was pretty late to be considered for fixing it in the next minor release.

Just to think out loud a bit: I'm guessing that 6.1 is actually more widely deployed than 6.0 (considering general wariness of .0 releases and that pv-on-hvm was off by default for 6.0). Plus, if we stay with the 'e' naming for 6.2 as well, then there will no doubt be even more chance of breakage when switching back to the 'a' naming in 6.3. So, it would appear we shouldn't change it at all. However, upstream (which we followed into the first name change) has now changed it back to the 'a' naming with the patch referenced above. So, if RHEL is to stay consistent with other Linux distributions, then it should change back as well.

I'll do some more head scratching on this issue.

I think this is a false problem.  It seems extremely unlikely that your configuration says sda1 and your fstab says xvda1 (or xvde1).  It is much much more likely that your using labels or uuids, and thus nothing breaks even though your device names change under the hood.  It _is_ possible that other utilities are storing real device names somewhere, of course, but then it is also very unlikely that this is breaking boot.

Supporting sdXX/hdYY in pv-ops (i.e. RHEL6) configurations is strictly a hack.  We probably should have never supported anything but xvdXX in RHEL6; we should have disabled sdXX/hdYY in RHEL6.0.

I'd be wary of changing this stuff.  If anything, we could add a loud syslog warning if anybody uses sdXX/hdYY device names in a PV configuration.

Paolo,

I've been using OnApp where this is exactly how things are currently set up.  They are going to change the software to avoid this problem in the future, but it does in fact break boot at this time.  From my reading this problem also affected some fairly common SolusVM setups[1] and a number of Fedora VMs in Amazon EC2 until Bug 729340 was fixed.  Obviously labels are a good way to go, but it's not a false problem and does affect real-world implementations.

[1] http://stacklet.com/story/centos/6.0/solusvm-notes-xvda

Still, changing things twice (which also breaks things that were fixed by the original change) is probably the only thing worse than changing them once.

We should create *now* a RHEL7 bug about prohibiting sdXX/hdYY, or perhaps even disallow it upstream.

This bug is the cause of 705572 and 734794 because other parts of the system configuration is expecting xvda. I would really like to see this fixed.

What about turing EMULATED_SD_DISK_MINOR_OFFSET and EMULATED_SD_DISK_NAME_OFFSET into module-private variables, initialized at module load time by a boolean flag:

modparam: adjust_emulated_scsi

If true:

emulated_sd_disk_minor_offset = EMULATED_HD_DISK_MINOR_OFFSET + (4 * 16);
emulated_sd_disk_name_offset = EMULATED_HD_DISK_NAME_OFFSET + 4;

if false:

emulated_sd_disk_minor_offset = 0;
emulated_sd_disk_name_offset = emulated_sd_disk_minor_offset / 256;

I'd default "adjust_emulated_scsi" to "false". That's the broken behavior ("This will break when mixing IDE and SCSI names in the configuration of guests"), but people seem to depend on it in hard-to-change places.

Thoughts?

(Everything can be solved by another layer of indirection.)

(In reply to comment #13)
> modparam: adjust_emulated_scsi

Not a bad idea, however it would still require installers to change something, the modprobe.d conf file. So maybe they should just change to using labels instead. That said, if it's too difficult or risky to try and find all the hardcoded xvdas in the installers, then maybe giving the option to just make one change in a modprobe.d conf file would be a nice "solution".

Just adding a use case:

We're automatically building machines in ec2 and attaching block devices.  Ours scripts no longer work as the device letters on the guest don't match what ec2 is using so we aren't getting to the stage where we can even label a device for reference.

This request was evaluated by Red Hat Product Management for inclusion
in a Red Hat Enterprise Linux maintenance release. Product Management has 
requested further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed 
products. This request is not yet committed for inclusion in an Update release.

*** Bug 734794 has been marked as a duplicate of this bug. ***

Patch(es) available on kernel-2.6.32-229.el6

@Aristeu Rozanski where is this kernel available publicly, I can't find it?

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
Release 6.1 introduced naming scheme adjustments for SCSI disks when they used with paravirtual drivers to prevent name-space clashes between IDE and  SCSI disks. i.e. 6.0 naming scheme  hda -> xvda, sda -> xvda and since 6.1 SCSI disks are mapped starting from sda -> xvde, IDE mapping stays unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda to provide a seamless upgrade path from 6.0 to 6.3 kernel release. Default value of xen_blkfront.sda_is_xvda is 0, it keeps naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, naming scheme reverts to 6.0 compatible mode. In the case of upgrade from 6.0 to 6.3 release, if customer specifies disks in VM configuration as SCSI and guest utilizes paravirtual drivers and uses explicit disk names (i.e. xvd[a-d]) it is advised to add parameter xen_blkfront.sda_is_xvda=1 to current kernel command line before performing upgrade.

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1 @@
-Release 6.1 introduced naming scheme adjustments for SCSI disks when they used with paravirtual drivers to prevent name-space clashes between IDE and  SCSI disks. i.e. 6.0 naming scheme  hda -> xvda, sda -> xvda and since 6.1 SCSI disks are mapped starting from sda -> xvde, IDE mapping stays unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda to provide a seamless upgrade path from 6.0 to 6.3 kernel release. Default value of xen_blkfront.sda_is_xvda is 0, it keeps naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, naming scheme reverts to 6.0 compatible mode. In the case of upgrade from 6.0 to 6.3 release, if customer specifies disks in VM configuration as SCSI and guest utilizes paravirtual drivers and uses explicit disk names (i.e. xvd[a-d]) it is advised to add parameter xen_blkfront.sda_is_xvda=1 to current kernel command line before performing upgrade.+Release 6.1 introduced naming scheme adjustments for emulated SCSI disks when they are used with paravirtual drivers to prevent name-space clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device xvd. For 6.0, if a config had both an hda and an sda, then they were both mapped to xvda. Since 6.1 emulated SCSI disks are mapped starting from xvde, i.e. sda -> xvde, sdb -> xvdf, and on. The IDE mapping remains unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda that provides a seamless upgrade path from 6.0 to 6.3. The default value of xen_blkfront.sda_is_xvda is 0, which keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to 6.0 compatible mode. In the case of an upgrade from 6.0 to 6.3, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers with explicit disk names (i.e. xvd[a-d]), then it is advised to add the parameter xen_blkfront.sda_is_xvda=1 to kernel command line before performing the upgrade.

    Technical note updated. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    Diffed Contents:
@@ -1 +1,10 @@
-Release 6.1 introduced naming scheme adjustments for emulated SCSI disks when they are used with paravirtual drivers to prevent name-space clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device xvd. For 6.0, if a config had both an hda and an sda, then they were both mapped to xvda. Since 6.1 emulated SCSI disks are mapped starting from xvde, i.e. sda -> xvde, sdb -> xvdf, and on. The IDE mapping remains unchanged. This update introduces a new module parameter xen_blkfront.sda_is_xvda that provides a seamless upgrade path from 6.0 to 6.3. The default value of xen_blkfront.sda_is_xvda is 0, which keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to 6.0 compatible mode. In the case of an upgrade from 6.0 to 6.3, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers with explicit disk names (i.e. xvd[a-d]), then it is advised to add the parameter xen_blkfront.sda_is_xvda=1 to kernel command line before performing the upgrade.+Red Hat Enterprise Linux 6.1 introduced naming scheme adjustments for emulated SCSI disks used with paravirtual drivers to prevent namespace clashes between emulated IDE and emulated SCSI disks. Both emulated disk types use the paravirt block device 'xvd'. Consider the example below:
+
+
+    	                | 6.0 	        | 6.1 or later
+    emulated IDE 	| hda -> xvda 	| unchanged
+    emulated SCSI 	| sda -> xvda 	| sda -> xvde, sdb -> xvdf, ...
+
+    This update introduces a new module parameter, xen_blkfront.sda_is_xvda, that provides a seamless upgrade path from 6.0 to 6.3 kernel release. The default value of xen_blkfront.sda_is_xvda is 0 and it keeps the naming scheme consistent with 6.1 and later releases. When xen_blkfront.sda_is_xvda is set to 1, the naming scheme reverts to the 6.0-compatible mode.
+
+    Note that when upgrading from 6.0 to 6.3 release, if a virtual machine specifies emulated SCSI devices and utilizes paravirtual drivers and uses explicit disk names such as xvd[a-d], it is advised to add the xen_blkfront.sda_is_xvda=1 parameter to the kernel command line before performing the upgrade.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2012-0862.html