Bug 456283

Summary: encrypted software RAID devices broken
Product: Red Hat Enterprise Linux 5 Reporter: David Lehman <dlehman>
Component: anacondaAssignee: David Lehman <dlehman>
Status: CLOSED ERRATA QA Contact:
Severity: medium Docs Contact:
Priority: medium    
Version: 5.2CC: atodorov, ddumas, jlaska, mganisin, rlerch
Target Milestone: rc   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Note Category: Known_Issues>Installation Applies to Architecture(s): All <para> Creating or using encrypted software RAID member disks (i.e. software RAID partitions) is not supported. However, creating encrypted software RAID arrays (e.g. /dev/md0) is supported. </para>
Story Points: ---
Clone Of: Environment:
Last Closed: 2009-01-20 21:36:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 454962    
Attachments:
Description Flags
Installation configuration
none
/etc/fstab
none
/etc/crypttab
none
/proc/mdstat
none
blkid output none

Description David Lehman 2008-07-22 16:33:39 UTC
Description of problem:
Because of several factors, there is no way to differentiate between an
encrypted software RAID partition and an unencrypted software RAID partition
that is part of an encrypted RAID1 array. Since the use of encrypted software
RAID partitions is not advisable in the vast majority of cases, we propose to
simply disable the ability to access or create encrypted software RAID partitions.

Note the difference between "software RAID partition", which is a potential
member of a RAID array, and the actual RAID array.

Version-Release number of selected component (if applicable):
anaconda-11.1.2.115

How reproducible:
Always

Steps to Reproduce:
1. Create an encrypted RAID1 array containing on encrypted member and one
non-encrypted
2. wait until anaconda has assembled and formatted the array
3. inspect the members' underlying block devices using 'cryptsetup isLuks
<device>' and isys.raidsb(<device>)
  
Actual results:
Both members appear to be encrypted, apparently due to the location of the RAID
metadata at the end of the device.

Expected results:
The encrypted member should appear to be encrypted, the non-encrypted should not.

Additional info:
This is not an issue in F9, so will not be for RHEL6 either.

Comment 1 RHEL Program Management 2008-07-25 17:01:25 UTC
This request was evaluated by Red Hat Product Management for
inclusion, but this component is not scheduled to be updated in
the current Red Hat Enterprise Linux release. If you would like
this request to be reviewed for the next minor release, ask your
support representative to set the next rhel-x.y flag to "?".

Comment 2 Ludek Smid 2008-07-25 22:03:56 UTC
Unfortunately the previous automated notification about the
non-inclusion of this request in Red Hat Enterprise Linux 5.3 used
the wrong text template. It should have read: this request has been
reviewed by Product Management and is not planned for inclusion
in the current minor release of Red Hat Enterprise Linux.

If you would like this request to be reviewed for the next minor
release, ask your support representative to set the next rhel-x.y
flag to "?" or raise an exception.

Comment 4 David Lehman 2008-08-04 17:06:25 UTC
Added New Release Notes Contents.

Comment 5 David Lehman 2008-08-05 20:32:46 UTC
FWIW I have a patch waiting that disables the encryption checkbox for partitions of type "software RAID" in the GUI and also disables discovery of preexisting encrypted swraid partitions.

Comment 6 Alexander Todorov 2008-08-07 07:05:33 UTC
(In reply to comment #0)
> Additional info:
> This is not an issue in F9, so will not be for RHEL6 either.

Dave,
can you shortly explain the difference between RHEL 5 and F9 and how this is not an issue with Fedora. I'm just curious and it's a good information to have in the records.

Comment 7 Alexander Todorov 2008-08-07 07:06:17 UTC
(In reply to comment #5)
> FWIW I have a patch waiting that disables the encryption checkbox for
> partitions of type "software RAID" in the GUI and also disables discovery of
> preexisting encrypted swraid partitions.

Does this patch also cover kickstart?

Comment 8 David Lehman 2008-08-07 15:13:31 UTC
(In reply to comment #7)
> (In reply to comment #5)
> > FWIW I have a patch waiting that disables the encryption checkbox for
> > partitions of type "software RAID" in the GUI and also disables discovery of
> > preexisting encrypted swraid partitions.
> 
> Does this patch also cover kickstart?

Yes.

Comment 9 David Lehman 2008-08-07 15:16:15 UTC
(In reply to comment #6)
> (In reply to comment #0)
> > Additional info:
> > This is not an issue in F9, so will not be for RHEL6 either.
> 
> Dave,
> can you shortly explain the difference between RHEL 5 and F9 and how this is
> not an issue with Fedora. I'm just curious and it's a good information to have
> in the records.

The main difference I am aware of is the fact that the raid metadata has moved from the end of the device to the beginning of the device. One other possiblely important difference is that we use libblkid for filesystem probing in F9 and our own ad-hoc probing in RHEL5.

Comment 10 Ryan Lerch 2008-08-12 03:29:08 UTC
Tracking this bug for the Red Hat Enterprise Linux 5.3 Release Notes.

Comment 11 Ryan Lerch 2008-08-12 05:24:29 UTC
added contents of the release_note field to 5.3 Release Notes source.

This note is located in the "Installation-Related Notes" section.

Comment 12 Ryan Lerch 2008-08-12 05:24:29 UTC
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1 @@
-Creation or use of encrypted software RAID member disks (partitions of type "software RAID") is not supported. Encrypted software RAID arrays (eg: /dev/md0), however, are supported.+Creating or using encrypted software RAID member disks (i.e. software RAID partitions) is not supported. However, creating encrypted software RAID arrays (e.g. /dev/md0) is supported.

Comment 13 RHEL Program Management 2008-08-14 19:25:13 UTC
This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release.  Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products.  This request is not yet committed for inclusion in an Update
release.

Comment 14 David Lehman 2008-09-18 00:31:28 UTC
Fixed in anaconda-11.1.2.126-1.

Comment 16 Denise Dumas 2008-10-13 17:33:56 UTC
Fix was incomplete, see 466761 - reported situation should not have been possible.

Comment 17 David Lehman 2008-10-14 15:48:44 UTC
Fix in anaconda-11.1.2.141-1.

Comment 19 Marian Ganisin 2008-10-15 11:39:21 UTC
Created attachment 320416 [details]
Installation configuration

Comment 20 Marian Ganisin 2008-10-15 11:40:11 UTC
Created attachment 320417 [details]
/etc/fstab

Comment 21 Marian Ganisin 2008-10-15 11:41:10 UTC
Created attachment 320418 [details]
/etc/crypttab

Comment 22 Marian Ganisin 2008-10-15 11:41:42 UTC
Created attachment 320420 [details]
/proc/mdstat

Comment 23 Marian Ganisin 2008-10-15 11:42:22 UTC
Created attachment 320422 [details]
blkid output

Comment 24 Marian Ganisin 2008-10-15 11:42:51 UTC
I propose to review support of sw RAID+LUKS in RHEL 5.3
I performed some tests, even if it wasn't with latest fix (I used RHEL5.3-Server-20081006.0-i386), my finding is interesting. I installed system with encrypted mdX devices, this system isn't able to start properly. Issue belongs to initscripts, however I want to explain it here first.

As it is stated here, sw RAID metada is stored at the end of disk. This causes incorrect detection during boot. There's for example encrypted /dev/md2 device (RAID1) with /dev/hda5 and /dev/hda10 members. However partition /dev/hda5 is detected as a crypt_LUKS device, user is asked for passphrase and obviously initialization fails.
The same behavior occurs also with RAID0 configuration.
All important logs are attached.

I assume, this could require significant changes in init scripts, which are far away of Fedora mainstream and from my point of view it's more reasonable to disable sw RAID+LUKS combination.
What's the decision?

Comment 25 Alexander Todorov 2008-10-15 14:44:33 UTC
Hmm, interesting. I was able to perform several manual installs on ia64 where I had / on md0, encrypted and/or / and /home on md0/md1 encrypted/plain and I didn't see a problem. System was bootable.

Marian,
are you using the provided kisktart configuration only?  Have you tried to do a manual installation?

My (wild) guess is that:
raid /eraid0 --fstype ext3 --encrypted --passphrase="longpassword" --level=RAID0 --device=md1 raid.7 raid.5
raid /eraid1 --fstype ext3 --encrypted --passphrase="longpassword" --level=RAID1 --device=md2 raid.17 raid.4
 

is actually trying to encrypt the underlying RAID members and not the mdX device itself. This is true for example when you manually select "Remove all partitions and create default layout", where a volume group is created containing / and swap as logical volumes but the underlying physical volumes are the ones that get encrypted.

Comment 26 David Lehman 2008-10-15 16:03:31 UTC
This may have just popped up when we began using UUIDs for device identification. It seems as though blkid sees both of the raid member disks as having the raid array's UUID.

We can't really hack blkid to assume if it's swraid it cannot be luks, so it seems like we have two options: 1) go back to using devices in crypttab, but only for encrypted raid arrays. 2) completely disallow encrypted raid.

Comment 27 David Lehman 2008-10-15 16:08:36 UTC
(In reply to comment #25)
> Hmm, interesting. I was able to perform several manual installs on ia64 where I
> had / on md0, encrypted and/or / and /home on md0/md1 encrypted/plain and I
> didn't see a problem. System was bootable.

My guess is that was when we were using, eg: /dev/sd3 instead of a UUID. There was no opportunity for blkid to get confused by the member disks containing the exact same data as the array device. Or perhaps you used a higher RAID level.

> is actually trying to encrypt the underlying RAID members and not the mdX
> device itself. This is true for example when you manually select "Remove all
> partitions and create default layout", where a volume group is created
> containing / and swap as logical volumes but the underlying physical volumes
> are the ones that get encrypted.

This is not happening. We do encrypt the PVs instead of the LVs for automatic partitioning, but that is at least partly to minimize the number of times you have to enter your passphrase during bootup. We do not alter the partitioning information from kickstart. Just like if you specify in kickstart that your LVs be encrypted, anaconda will do what you have asked it to.

Comment 28 Alexander Todorov 2008-10-16 09:48:48 UTC
(In reply to comment #27)
> (In reply to comment #25)
> > Hmm, interesting. I was able to perform several manual installs on ia64 where I
> > had / on md0, encrypted and/or / and /home on md0/md1 encrypted/plain and I
> > didn't see a problem. System was bootable.
> 
> My guess is that was when we were using, eg: /dev/sd3 instead of a UUID. There
> was no opportunity for blkid to get confused by the member disks containing the
> exact same data as the array device. Or perhaps you used a higher RAID level.
> 

I'm using the same build as in comment #24. So far tried with / on RAID 0,1 and 5 and system boots. 

Here's the relevant data from my system:

1) message from init:
Loading /lib/kbd/keymaps/i386/qwerty/us.map
raid5: raid level 5 set md0 active with 3 out of 4 devices, algorithm 2
Enter LUKS passphrase: 
key slot 0 unlocked.
Command successful.


[root@roentgen ~]# cat /etc/fstab 
/dev/mapper/luks-7aa8c29d-c2fc-447d-a7d3-d937037f1795 /                       ext3    defaults        1 1
/dev/mapper/luks-e9e0b642-a14c-4c16-9e24-801170499b41 /home                   ext3    defaults        1 2
/dev/mapper/luks-592f0948-3794-4f2f-ba8a-e52634fd0355 /data                   vfat    defaults        0 0
/dev/sda1               /boot/efi               vfat    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0


[root@roentgen ~]# cat /etc/crypttab 
luks-7aa8c29d-c2fc-447d-a7d3-d937037f1795 UUID=7aa8c29d-c2fc-447d-a7d3-d937037f1795 none
luks-e9e0b642-a14c-4c16-9e24-801170499b41 UUID=e9e0b642-a14c-4c16-9e24-801170499b41 none
luks-592f0948-3794-4f2f-ba8a-e52634fd0355 UUID=592f0948-3794-4f2f-ba8a-e52634fd0355 none


[root@roentgen ~]# cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdb2[3] sdb1[2] sda3[1] sda2[0]
      31456512 blocks level 5, 256k chunk, algorithm 2 [4/4] [UUUU]
      
unused devices: <none>


[root@roentgen ~]# blkid 
/dev/mapper/luks-592f0948-3794-4f2f-ba8a-e52634fd0355: UUID="48F6-E97D" TYPE="vfat" 
/dev/mapper/luks-e9e0b642-a14c-4c16-9e24-801170499b41: LABEL="/home" UUID="e0207af4-7e89-4682-b4ab-f9df1dd5f40d" TYPE="ext3" 
/dev/mapper/luks-7aa8c29d-c2fc-447d-a7d3-d937037f1795: UUID="0ab6c8f9-95b1-490a-9fb8-96ce74f1bd9e" TYPE="ext3" 
/dev/sda1: SEC_TYPE="msdos" LABEL="/boot/efi" UUID="48F6-E97F" TYPE="vfat" 
/dev/sda2: UUID="7aa8c29d-c2fc-447d-a7d3-d937037f1795" TYPE="crypt_LUKS" 
/dev/sdb1: UUID="5e0a52ff-ff8a-4f5c-b425-3bc3e48c4f40" TYPE="crypt_LUKS" 
/dev/md0: UUID="7aa8c29d-c2fc-447d-a7d3-d937037f1795" TYPE="crypt_LUKS" 
/dev/sdb3: UUID="592f0948-3794-4f2f-ba8a-e52634fd0355" TYPE="crypt_LUKS" 
/dev/sda4: UUID="e9e0b642-a14c-4c16-9e24-801170499b41" TYPE="crypt_LUKS" 


From the above blkid output only sda2 and sdb1 are members of md0, others are stand alone partitions. Only sda2 has the same UUID as md0.

After talking with Marian here's some more info:

- I'm using a physical machine with 2 disks (sdX)
- He was using virtual machine with 1 disk (hdX)
- Marian supposed that disk names order could matter

Comment 29 Alexander Todorov 2008-10-16 10:32:15 UTC
btw when I try to rescue this system and enter the pass phrase for /dev/md0 anaconda spits a message:

      +----------------------+ Duplicate Labels +-----------------------+       
      |                                                                 |       
      | Multiple devices on your system are labelled Logical sector     |       
      | size is zero..  Labels across devices must be unique for your   |       
      | system to function properly.                                    |       
      |                                                                 |       
      | Please fix this problem and restart the installation process.   |       
      |                                                                 |       
      |                           +--------+                            |       
      |                           | Reboot |                            |       
      |                           +--------+                            |       
      |                                                                 |       
      |                                                                 |       
      +-----------------------------------------------------------------+       
                                                                                

I bet this has to do with the duplicate UUIDs although the message is not very informative.

Comment 30 Alexander Todorov 2008-10-16 11:47:06 UTC
From some additional tests I did:

bare metal, 1 disk (sda), / on raid5, encrypted - works
bare metal, 2 disks (sda, sdb), / on raid5, encrypted - works
xen hvm, 1 disk (hda), / on raid5, encrypted - works
xen hvm, 2 disks (hda, hdb), / on raid 5, encrypted - works

where all raid arrays are md0 and contain 4 partitions with almost the same size. Bare metal machines were ia64 and Xen VMs were i386. All used the same build (1006.0) as the above comments.

Comment 31 David Lehman 2008-10-16 17:46:51 UTC
(In reply to comment #29)
> btw when I try to rescue this system and enter the pass phrase for /dev/md0
> anaconda spits a message:
> 
<error dialog about duplicate labels removed for readability>
> 
> I bet this has to do with the duplicate UUIDs although the message is not very
> informative.

This is actually a separate bug. I will open a new report for it. IIUC this error will appear on any ia64 box with more than one pre-existing PV or more than one pre-existing VG.

Bug 467289 filed to track this separate problem.

Comment 32 David Lehman 2008-11-10 22:35:31 UTC
(In response to comment #24)

I just did an install of RHEL5.3-Client-20081029.0 (i386) with root on RAID1 (hda2, hda3). After install it rebooted and correctly prompted me for the passphrase for md0. After I entered the passphrase the system continued to boot all the way into runlevel 3.

Comment 33 Marian Ganisin 2008-11-11 08:54:36 UTC
Similar Bug 470027 reported (RAID0 configured), there's patch for init script. It could solve issues with encrypted md devices. Patch is ready for snapshot 3, testing will be performed.

Comment 34 Alexander Todorov 2008-11-14 12:08:03 UTC
Marian, Dave,
can you clarify what's the status of this bug? Marian, were you experiencing bug 470027 which is fixed in snap #3 or a separate issue. All tests I've done with encrypted RAID devices (mdX) PASS.

Comment 35 Marian Ganisin 2008-11-18 12:38:18 UTC
Two different issues described, both verified.
Verified, checkbox for encryption is not available when using 'software RAID' partition type in Anaconda.
Verified, same UUIDs of MD device and its members don't confuse init anymore. Tested with RAID0 and RAID1.

Comment 38 errata-xmlrpc 2009-01-20 21:36:25 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0164.html

Comment 39 Ryan Lerch 2009-04-22 04:11:45 UTC
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1 +1,3 @@
-Creating or using encrypted software RAID member disks (i.e. software RAID partitions) is not supported. However, creating encrypted software RAID arrays (e.g. /dev/md0) is supported.+<para>
+Creating or using encrypted software RAID member disks (i.e. software RAID partitions) is not supported. However, creating encrypted software RAID arrays (e.g. /dev/md0) is supported.
+</para>

Comment 40 Ryan Lerch 2009-04-22 04:56:27 UTC
Release note updated. If any revisions are required, please set the 
"requires_release_notes"  flag to "?" and edit the "Release Notes" field accordingly.
All revisions will be proofread by the Engineering Content Services team.

Diffed Contents:
@@ -1,3 +1,5 @@
+Note Category: Known_Issues>Installation
+Applies to Architecture(s): All
 <para>
 Creating or using encrypted software RAID member disks (i.e. software RAID partitions) is not supported. However, creating encrypted software RAID arrays (e.g. /dev/md0) is supported.
 </para>