Bug 659628 - [Hitachi 6.6 FEAT] Persistent Device Naming [NEEDINFO]
[Hitachi 6.6 FEAT] Persistent Device Naming
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: kernel (Show other bugs)
6.2
All Linux
medium Severity medium
: beta
: 6.6
Assigned To: Red Hat Kernel Manager
Red Hat Kernel QE team
: FutureFeature, OtherQA
Depends On:
Blocks: 767187 960739 846704 958503
  Show dependency treegraph
 
Reported: 2010-12-03 04:12 EST by Nao Nishijima
Modified: 2014-01-20 18:11 EST (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-20 18:11:54 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
ltroan: needinfo? (kernel-mgr)


Attachments (Terms of Use)

  None (edit)
Description Nao Nishijima 2010-12-03 04:12:49 EST
This feature requires persistent and consistent device naming in every layer of linux system.

This feature has two purposes
- A persistent device name always points to a same device (e.g. storage)
- Keep consistency of device names in system(kernel) logs and the interfaces
  accessed by user. (e.g. dmesg always shows the device name which has been
  used by mount command)

Device names(e.g. sda, sdb, ...) are assigned in the order of driver loading, and device recognizing(usually from small bus number).

This method caused device name mismatch when device configuration changing which changes bus number (Hot-plug, Device Breakdown), or system configuration changing (Driver loading order, changing modprobe.conf).

Typically, to avoid this problem we use persistent device names provided by udev. But dmesg, procfs information and some commands show device names instead of persistent device names.

For example, udev provides by-uuid "6a18e581-..." for sda (at that boot time), user will use "6a18e581-..." as a persistent device name, however, dmesg says it is "sda";

$ls -l /dev/disk/by-uuid/ | grep sda1
lrwxrwxrwx 1 root root 10 2010-12-01 13:47 6a18e581-badb-40c1-9b53-22fc239d4cad -> ../../sda1
   
$dmesg
...
[    1.892502]  sda:
[    1.962113]  sda1 sda2 sda3 sda4 sda5 sda6
[    1.962534] sd 0:0:0:0: [sda] Attached SCSI disk
...

Thus, since the device name in dmesg can be changed to "sdb" or something else, users have to check the real name every boot time. It's not the real persistent device name in the system.

This mismatch makes it difficult to solve problems related to the storages. It could take a long time for problem solving.

We want to solve this mismatch.

We have an idea to solve this issue. To make a new device status, we call it intermediate device. All devices go into intermediate device while kernel booting up. Intermediate device is unavailable devices, but kernel can access to their information. udev assigns an intermediate device a new name and go into available device.

Why is this feature needed?
This feature is needed to introduce Linux to the mission critical market in Japan. The mission critical customer systems have many I/O devices and hot-plug capability.

The mission critical customers expect device names always point to same disks and the name accessed the device is displayed in kernel log.
Comment 2 Nao Nishijima 2011-01-06 19:58:45 EST
The device name is changed at each boot up in the server with
several kinds of HBA. we want to use always the same device name.

We've found similar issues have been reported to Bugzilla.
We believe this issue is not only our problem but also annoying
other RHEL users and administrators.
- https://bugzilla.redhat.com/show_bug.cgi?id=566555
- https://bugzilla.redhat.com/show_bug.cgi?id=453948
- https://bugzilla.redhat.com/show_bug.cgi?id=654063

I've already started a discussion about device naming at LKML.
http://kerneltrap.org/mailarchive/linux-kernel/2010/10/8/4629496/thread

Does anyone in Red Hat have any plan to solve this issue,
and could you work on this issue with us?
Comment 3 RHEL Product and Program Management 2011-01-06 23:21:22 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 4 Suzanne Yeghiayan 2011-01-07 11:19:45 EST
This request was erroneously denied for the current release of Red Hat
Enterprise Linux.  The error has been fixed and this request has been
re-proposed for the current release.
Comment 5 Nao Nishijima 2011-01-11 08:10:41 EST
(In reply to comment #4)
> This request was erroneously denied for the current release of Red Hat
> Enterprise Linux.  The error has been fixed and this request has been
> re-proposed for the current release.

Could you tell me how did you fix this issue?
I'd like to test the solution, whether it can solve our issue.
Comment 7 RHEL Product and Program Management 2011-02-01 01:00:02 EST
This request was evaluated by Red Hat Product Management for
inclusion in the current release of Red Hat Enterprise Linux.
Because the affected component is not scheduled to be updated
in the current release, Red Hat is unfortunately unable to
address this request at this time. Red Hat invites you to
ask your support representative to propose this request, if
appropriate and relevant, in the next release of Red Hat
Enterprise Linux. If you would like it considered as an
exception in the current release, please ask your support
representative.
Comment 8 RHEL Product and Program Management 2011-02-01 13:56:26 EST
This request was erroneously denied for the current release of
Red Hat Enterprise Linux.  The error has been fixed and this
request has been re-proposed for the current release.
Comment 9 Nao Nishijima 2011-03-02 09:36:44 EST
I have an idea for solving this issue by using udev.

With this idea, user can always assign a same device name (e.g. sda)
to a specific device according to udev rules and device id.

Here, I show how my idea changes the device naming process.

(Current)
Device driver finds a Logical Unit(LU)
 -> The LU is registered as a block device and assigned a new device name.

(New)
Device driver finds a LU
 -> The LU is registered as a intermediate device and inform to udev
 -> udev finds the intermediate device
 -> udev assigns a persistent device name and registers as a block device

Intermediate Device:
User can not read/write this device, but it can advertise device
information in sysfs.

This requires just a small amount of changes in scsi subsystem.

Here, I released a concept code.
http://sourceforge.net/mailarchive/forum.php?thread_name=20110225150403.3878.64576.stgit%40localhost6.localdomain6&forum_name=dle-develop

Currently, this code just supports scsi, but I think it shows the
feasibility of this idea.

What would you think of this idea?
Comment 11 Jon Masters 2011-03-15 03:21:42 EDT
Dear Nishijima-san,

I would like to first say that I hope you are well and that you and your family are not affected too much by the terrible events in Japan this week. Everyone in Japan is in our hearts at the moment and we wish you all the best.

I reviewed your proposal in comment #9. It is an interesting proposal, and it is one that we had considered previously. However, we have a few concerns about the proposal, and its ability to be implemented in an upstream solution. These concerns are the following:

*). Upstream are unlikely to accept a fundamental change to block device naming (this almost certain in my mind, and several others share a similar view).

*). Such an interim naming implementation introduces the potential for unexpected problems causing ill-defined behavior. It creates a complex dependency between the kernel and userspace that did not exist previously.

*). It will be hard to justify this renaming approach since many people already use the symlinks provided by udev. The problem you want to solve is very real indeed, but many users are not affected by it, and therefore it will be hard to explain why such a renaming is required.

Additionally, we are interested in finding a solution that can be more easily incorporated into RHEL6. For these reasons, I would like to make a counter-proposal that I have discussed with Red Hat engineers (and I am told had been discussed by other upstream engineers previously - Harald Hoyer tells me that a similar proposal was previously endorsed by Kay Sievers in another situation):

*). Block devices will gain a new NULL-terminated string "label" attribute.

*). When a new block device is registered, an additional entry is created in /sys/class/block/<device name>/label. Through this interface, a label can be assigned to the device. That label should be unique in some fashion.

*). When the uevent is emitted on the device probe/registration, and udev runs, it will execute a rule that will look for a "label" entry in sysfs. If it finds one, it will apply a label according to the output of the "blkid" command (similar to the existing by-id symlink in sysfs) by writing this into sysfs.

*). When this entry is written, the kernel may optionally output a message indicating that a new label has been assigned, for the benefit of log files.

*). The dev_printk function will be modified to print the label assigned to block devices (in the case of a block device). Some other printk calls may need to be updated in certain kernel drivers to use the standard dev_printk or similar functions, with the correct use of labels. The kernel log will in any case contain the name of the device and label when they are assigned.

The above implements the same functionality, but does not require that devices be renamed. It does mean that there is still some potential to get non-useful log messages if direct calls to printk() are made in some code paths, but that should be fixable. It is also fairly trivial to implement, perhaps in less code than your example, can be relatively easily included into RHEL6 without risk of breaking behavior, and can apply to block devices in general (not just SCSI).

This should serve as a medium-term solution to the problem. I am examining the shorter-term workarounds, and the longer-term options, and will update this BZ shortly.

Please let me know what you think.

Thank you kindly,

Jon Masters
Comment 12 Nao Nishijima 2011-03-25 09:09:05 EDT
Dear Jon-san,

First of all, I'd like to explain our request for solution of this issue.
We want to solve the issues on external storages, because an enterprise system
needs multiple external storages in order for treat large-scale data.
And we also concern about some commands which output device names,
as well as kernel messages. Those outputs do not always point a same device.

To solve those issues, I create unnamed device (not a block device, but an intermediate device). When Device driver finds a LU, the LU is registered as an unnamed device. After udev finds the unnamed device, udev assigns a persistent device name and registers it as a block device. Hence, this is just a naming, not a renaming. 

Some users are satisfied with current udev solution.
Therefore our proposal is implemented as an option.


Operational changes in our proposal are described, as follows.

Installation:
Currently, users need to write udev rules after installation of OS.
1) Add the kernel boot parameter persistent_name=1 to the boot loader configuration.
2) Write udev rules as follows. 

SUBSYSTEM=="scsi_unnamed", ATTR{byid}=="xxx", PROGRAM="/lib/udev/write_persistent_name sda %p"
SUBSYSTEM=="scsi_unnamed", ATTR{byid}=="yyy", PROGRAM="/lib/udev/write_persistent_name sdb %p"

In the future, Installer generates udev rules automatically
1) Display and Select the persistent device name option during installation process.
2) Installer displays a list of devices.
3) Users assign a name to each device.
4) Installer generate udev rules automatically according to 3)


Change of a device name:
Users need to change the udev rules already created.

(Example) sda -> sdroot
(before)
SUBSYSTEM=="scsi_unnamed", ATTR{byid}=="xxx", PROGRAM="/lib/udev/write_persistent_name sda %p"
(after)
SUBSYSTEM=="scsi_unnamed", ATTR{byid}=="xxx", PROGRAM="/lib/udev/write_persistent_name sdroot %p"

In the future, we want to make a new udev key (KDEVNAME) in the udev rule, as follows.
SUBSYSTEM=="scsi_unnamed", ATTR{byid}=="xxx", KDEVNAME="sdroot"


Adding a new LU:
The udev appends a new rule manually when a new LU is added.
In the future, it is appended automatically, as like NIC.


Removing a LU:
Do nothing. 
The udev rule for the LU removed is not deleted. Therefore, the same name is assigned for the LU whenever it is added again. This behavior is also similar to the NIC.

Please tell me if you have any questions.

Thank you kindly,

Nao Nishijima
Comment 13 Jon Masters 2011-03-28 21:17:50 EDT
Nao-san,

We will discuss this on Wednesday, according to the call tonight. Thanks very much.

Jon.
Comment 14 RHEL Product and Program Management 2011-04-03 22:19:08 EDT
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.
Comment 15 Larry Troan 2011-05-05 10:02:40 EDT
Comment #5:
> Could you tell me how did you fix this issue?
> I'd like to test the solution, whether it can solve our issue.

We have an automated bugbot that ran incorrectly on 2011-01-07 and erronously NACK'd this request for 6.1 at that time. As I explained to Aguchi-san, there is no chance to get a fix in 6.1 are we are nearing the end of the beta cycle.

There is no solution yet and this is now a PUBLIC feature request for 6.2. 
Resetting NEDINFO.
Comment 17 Larry Troan 2011-06-22 13:58:00 EDT
We are working on near and strategic solutions here. The stategic solution won't make 6.2 as it may require an ACPI spec change per Jon Masters. 

Should we push this to 6.3 to track the strategic solution?
Comment 18 Nao Nishijima 2011-06-23 01:39:03 EDT
Our solution for 6.2 requires not an ACPI spec change but to add udev's rules.

We intend to provide the following solution for 6.2
1. After installed, we add udev's rules as pairs of a disk's id and an alias name.
2. In the next booting, udev assigns an alias name to a disk.
3. Then kernel print alias name in kernel messages.

Kernel maintainers agreed this idea. I make a patch and as soon as
possible send the patch to upstream (this week or early next week).

If upstream accepted the patch, I can submit the patch to RedHat anytime.

NOTICE:
Our purpose is to use alias name anywhere. But some commands does't use
alias name. We need to modify commands which use device name. Now, I am
modifying those commands. I will send those command patches In July.

Thanks,
Comment 19 Larry Troan 2011-11-10 10:10:39 EST
Missed RHEL6.2. Last entry was 2011-06-23. I assume the patch mentioned in comment #18 was not accepted upstream. Pushing to RHEL6.3 for consideration.

I'm resetting the NEEDINFO as I can not find any open questions for me.
Comment 20 Larry Troan 2012-01-14 17:31:24 EST
No activity and no ACKs on this but on kernel priority list for RHEL6.3.

Does the kernel team plan to proceed in RHEL6.3?
Comment 24 Suzanne Yeghiayan 2012-05-18 16:46:21 EDT
This request was not resolved in time for the current release.
Red Hat invites you to ask your support representative to
propose this request, if still desired, for consideration in
the next release of Red Hat Enterprise Linux.
Comment 25 Larry Troan 2013-12-16 15:54:15 EST
Para-phrasing comment #20
> Does the kernel team plan to proceed with this feature in RHEL6.6?
Comment 27 Tom Coughlan 2014-01-20 18:11:54 EST
The last update to this bug: 

(In reply to Nao Nishijima from comment #18)

...

> We intend to provide the following solution for 6.2
> 1. After installed, we add udev's rules as pairs of a disk's id and an alias
> name.
> 2. In the next booting, udev assigns an alias name to a disk.
> 3. Then kernel print alias name in kernel messages.

That was not accepted upstream. 

...

> NOTICE:
> Our purpose is to use alias name anywhere. But some commands does't use
> alias name. We need to modify commands which use device name. Now, I am
> modifying those commands. I will send those command patches In July.

This work was done in RHEL 6, see:

https://bugzilla.redhat.com/show_bug.cgi?id=727046#c7

and RHEL 7, for example:

Bug 828482 - [Hitachi 7.0 FEAT] iostat supports the persistent device names
Bug 828483 - [Hitachi 7.0 FEAT] sar supports the persistent device names

There remains additional work to be done in 7.* to take advantage of journald's capability to search the log using persistent device names: 

Bug 828479 - [Hitachi 7.0 FEAT] Persistent Device Naming

I do not expect this to be ported to RHEL 6. 

I propose to close this ageing BZ. I suggest that if there are additional enhancements that are desired and appear feasible for RHEL 6 that you open new, specific, BZs for those features. 

Tom

Note You need to log in before you can comment on or make changes to this bug.