Bug 1420696

The sosreport logs show that the VG which was not activated is vg_test and its associated LV lv_test.

Based on logs from pvs, the device that holds PV which belongs to VG vg_test is /dev/sdc (sos_commands/lvm2/pvs_-a_-v_-o_pv_mda_free_pv_mda_size_pv_mda_count_pv_mda_used_count_pe_start_--config_global_locking_type_0)

 /dev/sdc                 vg_test   lvm2 a--   508.00m    0   512.00m P9p01V-jm8a-4uBf-qAs7-IXAJ-dccG-pScuic 


At the same time the /dev/sdc device is identified as containing "dos" signature instead of PV signature by blkid - the log from blkid (sos_commands/block/blkid_-c_.dev.null):

/dev/sdc: PTTYPE="dos"


This is also what udev sees based on blkid call there (/sos_commands/devices/udevadm_info_--export-db)

P: /devices/pci0000:00/0000:00:08.0/virtio2/host2/target2:0:0/2:0:0:0/block/sdc
N: sdc
...
E: ID_PART_TABLE_TYPE=dos


And because /dev/sdc is identified by blkid as containing dos partition table signature instead of PV signature (ID_FS_TYPE=LVM2_member), the lvm2-pvscan@8:32.service is not instantiated for this device. This service would execute "pvscan --cache --activate ay 8:32" which would in activate the vg_test with lv_test in turn.


Looking further, I don't see any partitions for /dev/sdc (any of sdc<number>). That means, this is just a partition table signature that is written at the beginning of the disk, but there are no partitions defined further.

And this is where LVM2 and blkid differs in how it processes such situation:

  - LVM2 ignores partition table signatures without any partitions defined and looks scans further for PVs

  - blkid, if it sees partition table signature, no matter if it's with or without actual partitions defined further, it identifies such device as having partition table and it does not scan further (...for PVs)

I think we already hit this problem with util-linux/libblkid once, but I need to look this up. I don't remember right now what was the actual conclusion and fix here (...also, it's possible the version of util-linux in RHEL7 is a bit older that this is not fixed yet, but I need to make sure).

(In reply to Peter Rajnoha from comment #4)
> I think we already hit this problem with util-linux/libblkid once, but I
> need to look this up. I don't remember right now what was the actual
> conclusion and fix here (...also, it's possible the version of util-linux in
> RHEL7 is a bit older that this is not fixed yet, but I need to make sure).

I think this is the right bugzilla that was tracking this problem: bug #1303157. The conclusion there was that fdisk would check for any existing signatures and offer a "wipe" option (appearead first in util-linux 2.29):

https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/commit/?id=ba465623d84b9e330f248a477d078b5f280b7943

Unfortunately, based on the bug #1303157 comment#12, this patch did not get backported into RHEL7 util-linux due to the invasive nature of the change as it was considered by util-linux maintainer.

===

LVM2 already has this feature though - it scans for existing signatures and offers wiping before it creates the actual PV, for example:

# pvcreate /dev/sda
WARNING: dos signature detected on /dev/sda at offset 510. Wipe it? [y/n]: y
  Wiping dos signature on /dev/sda.
  Physical volume "/dev/sda" successfully created.

# pvs /dev/sda
  Failed to find physical volume "/dev/sda".

However, if you answer "n" for the wiping question above in pvcreate, the PV is not created:

# pvcreate /dev/sda
WARNING: dos signature detected on /dev/sda at offset 510. Wipe it? [y/n]: n
  Aborted wiping of dos.
  1 existing signature left on the device.

# pvs /dev/sda
  Failed to find physical volume "/dev/sda".

So LVM2 already protects from this to happen.

===

Since the patch for util-linux is not in RHEL7 yet, you can fix the issue by using wipefs directly, for example, let's consider the problematic device is /dev/sda (I took the dd copy of the start of the disk which was attached here in this report):

# wipefs /dev/sda
offset               type
----------------------------------------------------------------
0x1fe                PMBR   [partition table]

0x218                LVM2_member   [raid]
                     UUID:  P9p01V-jm8a-4uBf-qAs7-IXAJ-dccG-pScuic


You can see that there are two signatures detected - "PMBR" and "LVM2_member" - we need to get rid of the "PMBR" one so that only LVM2_member stays - use the offset as reported for the "PMBR" signature type:

# wipefs --offset 0x1fe /dev/sda
/dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
/dev/sda: calling ioctl to re-read partition table: Success


And check the only one that stays is the LVM2_member signature type:

# wipefs /dev/sda
offset               type
----------------------------------------------------------------
0x218                LVM2_member   [raid]
                     UUID:  P9p01V-jm8a-4uBf-qAs7-IXAJ-dccG-pScuic

# blkid /dev/sda
/dev/sda: UUID="P9p01V-jm8a-4uBf-qAs7-IXAJ-dccG-pScuic" TYPE="LVM2_member"

So blkid now properly sees the LVM2_member (the PV) signature.

We should document these steps (knowledge base article) for manual repair because even if the patch for fdisk appears in RHEL7, people can still have existing disks edited with older tools where this mixture of signatures can still be made.

(In reply to Peter Rajnoha from comment #5)
> LVM2 already has this feature though - it scans for existing signatures and
> offers wiping before it creates the actual PV, for example:
> 
> # pvcreate /dev/sda
> WARNING: dos signature detected on /dev/sda at offset 510. Wipe it? [y/n]: y
>   Wiping dos signature on /dev/sda.
>   Physical volume "/dev/sda" successfully created.
> 
> # pvs /dev/sda
>   Failed to find physical volume "/dev/sda".
> 

This should have been:

# pvs /dev/sda
  PV         Fmt  Attr PSize   PFree  
  /dev/sda   lvm2 ---  128.00m 128.00m

...copy-paste error, sorry

Would the multipath layer potentially hide this issue? If not, what am I doing wrong here to reproduce this issue?


# Rhel6.9                                                                                                                                                                               
[root@mckinley-02 ~]# uname -ar                                                                                                                                                      
Linux mckinley-02.lab.msp.redhat.com 2.6.32-696.el6.x86_64 #1 SMP Tue Feb 21 00:53:17 EST 2017 x86_64 x86_64 x86_64 GNU/Linux                                                        
                                                                                                                                                                                     
[root@mckinley-02 ~]# fdisk /dev/mapper/mpatha                                                                                                                                
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to                                                                                                      
         switch off the mode (command 'c') and change display units to                                                                                                        
         sectors (command 'u').                                                                                                                                               
                                                                                                                                                                              
Command (m for help): o                                                                                                                                                     
Building a new DOS disklabel with disk identifier 0xc8abefa6.                                                                                                               
Changes will remain in memory only, until you decide to write them.                                                                                                 
After that, of course, the previous content won't be recoverable.                                                                                                   
                                                                                                                                                                    
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)                                                                                      
                                                                                                                                                                    
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to                                                                                            
         switch off the mode (command 'c') and change display units to                                                                                           
         sectors (command 'u').                                                                                                                                  
                                                                                                                                                                 
Command (m for help): w                                                                                                                                          
The partition table has been altered!                                                                                                                            
                                                                                                                                                                
Calling ioctl() to re-read partition table.                                                                                                                       
                                                                                                                                                                   
WARNING: Re-reading the partition table failed with error 22: Invalid argument.                                                                                        
The kernel still uses the old table. The new table will be used at                                                                                                               
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

[root@mckinley-02 ~]# partprobe /dev/mapper/mpatha
[root@mckinley-02 ~]# blkid /dev/mapper/mpatha
[root@mckinley-02 ~]# pvcreate /dev/mapper/mpatha
  Physical volume "/dev/mapper/mpatha" successfully created
[root@mckinley-02 ~]# blkid /dev/mapper/mpatha
/dev/mapper/mpatha: UUID="DboPt2-eAkB-x99q-1NUa-h8nn-BK6z-kkeulV" TYPE="LVM2_member" 
[root@mckinley-02 ~]# vgcreate VG /dev/mapper/mpatha
  Volume group "VG" successfully created
[root@mckinley-02 ~]# lvcreate -n LV -L 100M VG
  Logical volume "LV" created.
[root@mckinley-02 ~]# lvs -a -o +devices
  LV  VG  Attr       LSize    Devices              
  LV  VG  -wi-a----- 100.00m  /dev/mapper/mpatha(0)
[root@mckinley-02 ~]# vgchange -an VG
  0 logical volume(s) in volume group "VG" now active


# Rhel7.4 machine connected to the same storage
[root@mckinley-03 ~]# uname -ar
Linux mckinley-03.lab.msp.redhat.com 3.10.0-693.el7.x86_64 #1 SMP Thu Jul 6 19:56:57 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[root@mckinley-03 ~]# reboot -fin
Rebooting.

# After boot, lv was activated fine
[root@mckinley-03 ~]# lvs -a -o +devices
  LV  VG  Attr       LSize    Devices              
  LV  VG  -wi-a----- 100.00m  /dev/mapper/mpathb(0)

[root@mckinley-03 ~]# blkid /dev/mapper/mpathb
/dev/mapper/mpathb: UUID="DboPt2-eAkB-x99q-1NUa-h8nn-BK6z-kkeulV" TYPE="LVM2_member"

Even with multipath out of the picture and creating w/ rhel6.6 like mentioned in comment #0, I'm still not seeing this.

# rhel6.6
[root@host-035 ~]# uname -ar
Linux host-035.virt.lab.msp.redhat.com 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@host-035 ~]# wipefs /dev/sdb
wipefs: WARNING: /dev/sdb: appears to contain 'dos' partition table
[root@host-035 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created
[root@host-035 ~]# blkid /dev/sdb
/dev/sdb: UUID="nKBAIT-BoId-kmNL-lKOb-99oD-FCKn-evdlp2" TYPE="LVM2_member" 
[root@host-035 ~]# vgcreate vg /dev/sdb
  Volume group "vg" successfully created
[root@host-035 ~]# lvcreate -L 100M vg
  Logical volume "lvol0" created


# rhel7.1 same storage
[root@host-034 ~]# uname -ar
Linux host-034.virt.lab.msp.redhat.com 3.10.0-229.el7.x86_64 #1 SMP Thu Jan 29 18:37:38 EST 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@host-034 ~]# reboot -fin
# Activates fine
[root@host-034 ~]# lvs -a -o +devices
  LV    VG  Attr       LSize   Devices       
  lvol0 vg  -wi-a----- 100.00m /dev/sdh(0)

(In reply to Corey Marthaler from comment #9)
> Even with multipath out of the picture and creating w/ rhel6.6 like
> mentioned in comment #0, I'm still not seeing this.
> 

You can use the dd dump from the attachment in this bug report (the rhel6_dd.out) and copy that onto a disk... 

So, to close this bug report once for all, does anybody know who to contact about writing the kbase article? Or can I write one directly?

Content for KCS might be something like this, feel free to edit if you feel we need to write that in more user-approachable way:


Possible ambiguity when combining partition table signature and PV signature
============================================================================

When a device already contains a PV signature, it is still possible to create a partition table on such a device - partitioning tools (like fdisk) do not check if there are any existing signatures when creating the partition table signature. The partition table signature can coexist with the PV signature because each one has different offset within the device and the signatures do not interleave.

If the partition table is empty and hence there are *no partitions defined yet*, there is an ambiguity in how such device content is identified by various system components:

  - blkid, which is the core component used for scanning device's content and identification of device subsystem based on found signatures, gives preference to partition table signature (reported as 'PTTYPE="dos"') over PV signature (reported as 'TYPE="LVM2_member"'). The blkid's result is also used in udev rules to decide and trigger appropriate hooks based on incoming udev events for the device, including LVM's event-based activation.

  - On the contrary, LVM tools give preference to its own PV signature if it sees both partition table signature and PV signature.

The existence of this ambiguity can be checked by running pvscan and blkid - the tools will report different identification results for a device.

This has a consequence on automatic event-based LVM activation - with this ambiguity, any VGs/LVs defined on top of such PVs are not automatically activated based on incoming device events.

Only newer versions of partitioning tools always check for any existing signatures before creating its own. If there are any signatures detected on the device during partitioning, the partitioning tools warn about this condition and user is informed about erasure of any existing signatures before writing partition table signatures to avoid the ambiguity as described above. This functionality is implemented in fdisk version 2.28 and higher only.

When the actions are done in opposite way, that is, if we create the partition table first on the device and then we create a PV out of the device, LVM tools already check for any existing signatures before creating a PV, hence avoiding the signature ambiguity.

If there is already a device where this ambiguity is observed and if we need to give preference to PV signatures, first we need to find the offset of the partition table which is causing the ambiguity and then we need to erase this signature manually.

For example:

  [0] rhel7-a/~ # pvscan
    PV /dev/sda    VG testvg          lvm2 [100.00 MiB / 96.00 MiB free]
    PV /dev/vda2   VG rhel            lvm2 [<9.51 GiB / 0    free]
    Total: 2 [<9.61 GiB] / in use: 2 [<9.61 GiB] / in no VG: 0 [0   ]

  [0] rhel7-a/~ # blkid
  /dev/vda2: UUID="2cBtQX-IKw0-vsJi-ZLB3-pUfS-w48b-2kA4H9" TYPE="LVM2_member" 
  /dev/sda: PTTYPE="dos" 

Here, we can see that vda2 is correctly identified as PV by both pvscan and blkid. However, there's ambiguity with sda. We will check for the "dos" signature offset with wipefs tool:

  [0] rhel7-a/~ # wipefs /dev/sda
  offset               type
  ----------------------------------------------------------------
  0x1fe                dos   [partition table]
  
  0x218                LVM2_member   [raid]
                       UUID:  hvD7wH-9dPi-q4tp-6PJd-v9ir-fQ2o-WLueaS

The wipefs correctly reports two signatures: "dos" and "LVM2_member". We need to erase the "dos" to resolve the ambiguity:

  [0] rhel7-a/~ # wipefs -o 0x1fe /dev/sda
  /dev/sda: 2 bytes were erased at offset 0x000001fe (dos): 55 aa
  /dev/sda: calling ioclt to re-read partition table: Success

After this step, only PV signature should be detected by all tools on the "sda" device:

  [0] rhel7-a/~ # wipefs /dev/sda
  offset               type
  ----------------------------------------------------------------
  0x218                LVM2_member   [raid]
                       UUID:  hvD7wH-9dPi-q4tp-6PJd-v9ir-fQ2o-WLueaS

  [0] rhel7-a/~ # pvscan
    PV /dev/sda    VG testvg          lvm2 [100.00 MiB / 96.00 MiB free]
    PV /dev/vda2   VG rhel            lvm2 [<9.51 GiB / 0    free]
    Total: 2 [<9.61 GiB] / in use: 2 [<9.61 GiB] / in no VG: 0 [0   ]

  [0] rhel7-a/~ # blkid
  /dev/vda2: UUID="2cBtQX-IKw0-vsJi-ZLB3-pUfS-w48b-2kA4H9" TYPE="LVM2_member" 
  /dev/sda: UUID="hvD7wH-9dPi-q4tp-6PJd-v9ir-fQ2o-WLueaS" TYPE="LVM2_member"

KCSs created and attached

OK, thanks! So I'm closing this bug then as it just required this kind of documentation...

Closing as CLOSED/NEXTRELEASE because the automatic fix requires patching partitioning tools (fdisk) to check existing signatures before adding its own. This is already included in RHEL8, but it won't be backported to RHEL7.