Bug 442723 - Xen Support more than 16 disk devices (kernel)
Xen Support more than 16 disk devices (kernel)
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: kernel-xen (Show other bugs)
5.0
All Linux
medium Severity medium
: rc
: ---
Assigned To: Chris Lalancette
Martin Jenner
:
: 437349 (view as bug list)
Depends On:
Blocks: RHEL5u2_relnotes 448753 RHEL5u3_relnotes 456392
  Show dependency treegraph
 
Reported: 2008-04-16 09:26 EDT by Bill Burns
Modified: 2009-02-23 09:31 EST (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, paravirtualized guests could only have a maximum of 16 disk devices. In this update, this limit has been increased to a maximum of 256 disk devices.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2009-01-20 15:09:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Expand SCSI devices to support sdaa and up (2.35 KB, patch)
2008-07-23 06:06 EDT, Chris Lalancette
no flags Details | Diff
Expand blkfront to recognize the new expanded xvd devices (6.05 KB, patch)
2008-07-23 06:07 EDT, Chris Lalancette
no flags Details | Diff
Expand blktap to understand the new IOCTL and handle new and old userland (2.14 KB, patch)
2008-07-23 06:07 EDT, Chris Lalancette
no flags Details | Diff

  None (edit)
Description Bill Burns 2008-04-16 09:26:13 EDT
Description of problem:
Guests are limited to 16 disk devices.

Version-Release number of selected component (if applicable):
RHEL 5 up through RHEL 5.2

This is derived from bug 219298, which asked for > 16 Scsi
devices. That issue has been closed but the issue of a 16
disk device limit needs to be looked at.
Comment 3 Chris Lalancette 2008-04-17 18:15:41 EDT
Unfortunately, this one might be tough.  Looking at
http://www.lanana.org/docs/device-list/devices-2.6+.txt, I can see that block
major 202 has been assigned to Xen virtual block devices.  However, the minor
numbering goes something like:

/dev/xvda = 0
/dev/xvda1 = 1
...
/dev/xvda15 = 15
/dev/xvdb = 16
/dev/xvdb1 = 17
...
/dev/xvdp = 240
...
/dev/xvdp17 = 255

As you can see, this is 16 whole disks, plus up to 15 partitions per disk. 
Adding additional block devices is going to require expanding into another major
number, which might be possible, but has to be done carefully.

Chris Lalancette
Comment 5 Chris Lalancette 2008-04-17 19:31:08 EDT
OK, just out of curiousity, I saw what I could do to increase the block devices.
 The method I used is completely inappropriate in a stable product, but I just
wanted to see what would happen, and what exactly I needed to change.  Basically
I cut the number of minor numbers available to each /dev/xvd? device down to 8,
which gave me a maximum of 32 device numbers.  Just for posterity, though, there
are actually 2 parts that need to change:

1) in the guest kernel, you need to change drivers/xen/blkfront/vbd.c to either
increase the number of majors or decrease the number of minors

2)  in the xen tools, you need to change python/xen/util/blkif.py to edit the
blkdev_name_to_number method to fix up the re.match() for the /dev/xvd devices.

Again, this was just proof of concept, but what I proved is that any solution
will require at least the above 2 changes.

Chris Lalancette
Comment 6 Daniel Berrange 2008-04-17 19:42:09 EDT
It would also require changes to libvirt to make the blockstats method caculate
the correct device IDs for extracting stats from xenstore/sysfs.
Comment 9 Don Domingo 2008-04-21 19:40:39 EDT
[RE: release notes] Calvin, is this limitation true for guests that are
paravirtualized, fully-virtualized, or both?
Comment 10 Calvin Smith 2008-04-22 10:13:38 EDT
This has been tested on a paravirtualized system. I suspect that this would not
be an issue on a fully virtualized system since it would have access to the full
sd* block device numbers (instead of doing the weird things to the sd names that
the paravirtualized system is doing after 26 characters), but I'm not entirely sure.
Comment 11 Don Domingo 2008-04-22 19:59:16 EDT
understood. for now, added to "Feature Updates => Virtualization => Known Issues":

<quote>
Paravirtualized guests can only have a maximum of 16 disk devices.
</quote>

please advise if any further revisions are required. note that any further
revisions will go into the release notes updates, not the distro. thanks!
Comment 23 Jason Willeford 2008-06-05 11:40:31 EDT
Has collaboration with upstream produced any approach to the fix for this yet? 
I don't want this to miss scheduling for 5.3.
Comment 25 RHEL Product and Program Management 2008-06-19 09:50:56 EDT
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.
Comment 27 Chris Lalancette 2008-06-23 15:22:03 EDT
I just posted a patch series upstream:

http://lists.xensource.com/archives/html/xen-devel/2008-06/msg00709.html

Assuming upstream takes this, we can then backport it into RHEL-5.  I'll keep
you updated on the status.

Chris Lalancette
Comment 28 Bill Burns 2008-07-03 08:26:39 EDT
*** Bug 437349 has been marked as a duplicate of this bug. ***
Comment 31 Chris Lalancette 2008-07-23 06:06:47 EDT
Created attachment 312451 [details]
Expand SCSI devices to support sdaa and up
Comment 32 Chris Lalancette 2008-07-23 06:07:24 EDT
Created attachment 312452 [details]
Expand blkfront to recognize the new expanded xvd devices
Comment 33 Chris Lalancette 2008-07-23 06:07:54 EDT
Created attachment 312453 [details]
Expand blktap to understand the new IOCTL and handle new and old userland
Comment 34 Don Zickus 2008-08-13 12:07:11 EDT
in kernel-2.6.18-104.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 35 Don Zickus 2008-08-13 13:25:39 EDT
in kernel-2.6.18-104.el5
You can download this test kernel from http://people.redhat.com/dzickus/el5
Comment 39 Ryan Lerch 2008-11-06 17:46:45 EST
Release note added. If any revisions are required, please set the 
"requires_release_notes" flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

New Contents:
Paravirtualized guests can only have a maximum of 16 disk devices.
Comment 41 Chris Lalancette 2008-11-07 03:22:36 EST
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,3 @@
-Paravirtualized guests can only have a maximum of 16 disk devices.+In previous versions of RedHat virtualization, paravirtualized guests could have a maximum of 16 disks attached.
+
+This set of patches increases the number of disks a paravirtualized guest can have to a maximum of 256 disks.
Comment 43 Ryan Lerch 2008-11-11 22:03:00 EST
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,3 +1 @@
-In previous versions of RedHat virtualization, paravirtualized guests could have a maximum of 16 disks attached.
+Previously, paravirtualized guests could only have a maximum of 16 disk devices. In this update, this limit has been increased to a maximum of 256 disk devices.-
-This set of patches increases the number of disks a paravirtualized guest can have to a maximum of 256 disks.
Comment 46 errata-xmlrpc 2009-01-20 15:09:26 EST
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2009-0225.html

Note You need to log in before you can comment on or make changes to this bug.