Bug 365741 - pvdisplay when using >2TB gets PV wrong after vgcreate
pvdisplay when using >2TB gets PV wrong after vgcreate
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
7
x86_64 Linux
low Severity medium
: ---
: ---
Assigned To: Alasdair Kergon
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-11-04 06:37 EST by Bryce
Modified: 2007-11-30 17:12 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-04 20:54:03 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Metadata (192.00 KB, application/octet-stream)
2007-11-04 11:00 EST, Bryce
no flags Details
VolGroup00 archive files (4.90 KB, application/octet-stream)
2007-11-04 11:35 EST, Bryce
no flags Details

  None (edit)
Description Bryce 2007-11-04 06:37:24 EST
Description of problem:

I'm trying to create a large FS over 5x 1TB drives
When I create a physical volume it correctly assigns a size of 3.64 TB

When I then progress to make a volume group pvdisplay then shows this has
changed to 8192.00 EB (ExaBytes of storage) errrrrr....


Version-Release number of selected component (if applicable):
lvm2-2.02.24-1.fc7
kernel-2.6.23.1-10.fc7

How reproducible:
(Heres a great chance to get your hands on 5 TB drives for testing...)

grab 5 1TB drives
create a raid5 set 
(your milage may vary but something like this... )
mdadm -C -l5 -n5 -c256  /dev/md1  /dev/sdb1 /dev/sdc1 /dev/sdk1 /dev/sdl1 /dev/sdt1

create the physical volume
pvcreate /dev/md1

display the physical volume
pvdisplay ->
  --- NEW Physical volume ---
  PV Name               /dev/md1
  VG Name               
  PV Size               3.64 TB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN

Ok,. looks correct, now create a volume group
vgcreate VolGroup00 /dev/md1
  Volume group "VolGroup00" successfully created
vgdisplay ->
  --- Volume group ---
  VG Name               VolGroup00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.64 TB
  PE Size               4.00 MB
  Total PE              953866
  Alloc PE / Size       0 / 0   
  Free  PE / Size       953866 / 3.64 TB
  VG UUID               7uQVFV-QbEp-jpJg-jXix-G0op-igNS-Cb6Xac

Ok, it still looks reasonable,.. the VG size is the same as what we started out
with BUT now type pvdisplay...

pvdisplay ->
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               VolGroup00
  PV Size               8192.00 EB / not usable 8192.00 EB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              953866
  Free PE               953866
  Allocated PE          0
  PV UUID               PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN

Note that now the PV size has changed to 8192.00 EB ?!?
err ok... what does pvscan say?
pvscan ->
  PV /dev/md1    VG VolGroup00   lvm2 [3.64 TB / 3.64 TB free]
hurm,...

did vgcreate do something wierd? lets delete it
vgremove VolGroup00
pvdisplay ->
  --- NEW Physical volume ---
  PV Name               /dev/md1
  VG Name               
  PV Size               3.64 TB
  Allocatable           NO
  PE Size (KByte)       0
  Total PE              0
  Free PE               0
  Allocated PE          0
  PV UUID               PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN

Ideas?
I /think/ it's actually ok but the PV size change does concern me esp when I
start to fill up this FS to any serious extent

Phil
=--=
Comment 1 Bryce 2007-11-04 07:08:58 EST
On an additional confused note,..
this is lvm2 (well the utilities claim to be from the lvm2 package...

I ask because of the -v output below

[root@emerald ~]# pvscan -v
File descriptor 5 left open
File descriptor 7 left open
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
    Walking through all physical volumes
    Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2
    Fixing up missing format1 size (2.27 TB) for PV /dev/md0
  PV /dev/sdd2   VG VolGrp00     lvm2 [465.66 GB / 32.00 MB free]
  PV /dev/md1    VG VolGroup00   lvm2 [3.64 TB / 3.64 TB free]
  PV /dev/md0    VG VolGroup01   lvm2 [2.27 TB / 0    free]
  Total: 3 [2.37 TB] / in use: 3 [2.37 TB] / in no VG: 0 [0   ]
[root@emerald ~]# pvdisplay  -v
File descriptor 5 left open
File descriptor 7 left open
    Scanning for physical volume names
    Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2
    Fixing up missing format1 size (2.27 TB) for PV /dev/md0
    Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2
  --- Physical volume ---
  PV Name               /dev/sdd2
  VG Name               VolGrp00
  PV Size               465.66 GB / not usable 3.56 MB
  Allocatable           yes 
  PE Size (KByte)       32768
  Total PE              14901
  Free PE               1
  Allocated PE          14900
  PV UUID               MLKmma-3vQi-tR5w-Us5E-TAjS-4pIt-Mm9kNJ
   
  --- Physical volume ---
  PV Name               /dev/md1
  VG Name               VolGroup00
  PV Size               8192.00 EB / not usable 8192.00 EB
  Allocatable           yes 
  PE Size (KByte)       4096
  Total PE              953866
  Free PE               953866
  Allocated PE          0
  PV UUID               KHMh9T-igJh-SGkg-6JGK-jPqc-6EGf-3NvKUE
   
    Fixing up missing format1 size (2.27 TB) for PV /dev/md0
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               VolGroup01
  PV Size               2.27 TB / not usable 2.88 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              596171
  Free PE               0
  Allocated PE          596171
  PV UUID               KWGw1x-LmYe-PqvN-V5uF-RlgV-rbhe-WP74Yb


Am I to infer from the above that the pv and vg are being created in lvm1
format?! (since the tools don't complain about missing lvm1 formatting)
Maybe I should just pull down the src and lok myself.. something is just not
quite right

Phil
=--=
Comment 2 Bryce 2007-11-04 07:32:46 EST
Never mind I'm going nuts figuring this out

pvdisplay -> 
  PV Size               8192.00 EB / not usable 8192.00 EB

[root@emerald ~]# pvs
File descriptor 5 left open
File descriptor 7 left open
  PV         VG         Fmt  Attr PSize   PFree 
  /dev/md0   VolGroup01 lvm2 a-     2.27T     0 
  /dev/md1   VolGroup00 lvm2 a-     3.64T  3.64T
  /dev/sdd2  VolGrp00   lvm2 a-   465.66G 32.00M

[root@emerald ~]# vgs
File descriptor 5 left open
File descriptor 7 left open
  VG         #PV #LV #SN Attr   VSize   VFree 
  VolGroup00   1   0   0 wz--n-   3.64T  3.64T
  VolGroup01   1   1   0 wz--n-   2.27T     0 
  VolGrp00     1   2   0 wz--n- 465.66G 32.00M

Phil
=--=

Comment 3 Alasdair Kergon 2007-11-04 10:38:57 EST
Could try the F8 packages to see if we already fixed this (update device-mapper
+ lvm2 packages together).

The 'format1' message looks like a mistake in the message text.  I'll fix it.
Comment 4 Bryce 2007-11-04 10:59:10 EST
vgcfgbackup ->
# Generated by LVM2: Sun Nov  4 15:50:54 2007

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing 'vgcfgbackup'"

creation_host = "emerald"       # Linux emerald 2.6.23.1-10.local.fc7 #1 SMP Fri
 Nov 2 19:37:26 GMT 2007 x86_64
creation_time = 1194191454      # Sun Nov  4 15:50:54 2007

VolGroup01 {
        id = "Qr20q0-lkea-Iesc-aIsW-4LA4-0L2I-f427J4"
        seqno = 4
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0

        physical_volumes {

                pv0 {
                        id = "KWGw1x-LmYe-PqvN-V5uF-RlgV-rbhe-WP74Yb"
                        device = "/dev/md0"     # Hint only

                        status = ["ALLOCATABLE"]
                        dev_size = 4883838720   # 2.27421 Terabytes
                        pe_start = 384
                        pe_count = 596171       # 2.27421 Terabytes
                }
        }

        logical_volumes {

                LogVol00 {
                        id = "fmlNTo-FIaf-n6VM-41bh-xvw5-wvpA-21Ze0P"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 596171   # 2.27421 Terabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
        }
}


attaching dd if=/dev/md1 bs=512 count=384 of=/metadata
Comment 5 Bryce 2007-11-04 11:00:16 EST
Created attachment 247601 [details]
Metadata

Metadata fields off /dev/md1
Comment 6 Bryce 2007-11-04 11:02:20 EST
Helps if I add the RIGHT vg config file... <grumble> not my day,...
sorry about that

# Generated by LVM2: Sun Nov  4 15:50:54 2007

contents = "Text Format Volume Group"
version = 1

description = "Created *after* executing 'vgcfgbackup'"

creation_host = "emerald"       # Linux emerald 2.6.23.1-10.local.fc7 #1 SMP Fri
 Nov 2 19:37:26 GMT 2007 x86_64
creation_time = 1194191454      # Sun Nov  4 15:50:54 2007

VolGroup00 {
        id = "gVFkvt-cN7R-VEg0-Uk3Q-FW0d-zOeV-anNquR"
        seqno = 2
        status = ["RESIZEABLE", "READ", "WRITE"]
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0

        physical_volumes {

                pv0 {
                        id = "ztSu45-Wx9U-Maw0-1smV-zKDC-8604-9xJdRZ"
                        device = "/dev/md1"     # Hint only

                        status = ["ALLOCATABLE"]
                        dev_size = 18446744073709551615 # 8 (null)
                        pe_start = 384
                        pe_count = 953866       # 3.63871 Terabytes
                }
        }

        logical_volumes {

                LogVol00 {
                        id = "rNbnws-9es8-wNOl-RZqU-p2YI-lunc-ttfHhn"
                        status = ["READ", "WRITE", "VISIBLE"]
                        segment_count = 1

                        segment1 {
                                start_extent = 0
                                extent_count = 953866   # 3.63871 Terabytes

                                type = "striped"
                                stripe_count = 1        # linear

                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }
        }
}
Comment 7 Bryce 2007-11-04 11:35:46 EST
Created attachment 247611 [details]
VolGroup00 archive files

Tar gzip of the VolGroup00 archives from /etc/lvm/archive
Comment 8 Alasdair Kergon 2007-11-04 19:42:05 EST
Still not sure how this came about, but I've noticed similar before.  I've added
the LVM2 version number to the metadata from now on to help with future diagnosis.

As I explained on IRC, this field is only used for display and after the PV is
removed from the VG with vgreduce or vgremove, so it's not particularly important.

We'll change the code to detect 'rogue' values like this and ignore them.
Comment 9 Alasdair Kergon 2007-11-04 19:43:59 EST
Again, as on IRC, the workaround is to use vgcfgbackup and vgcfgrestore, editing
the field back to its correct value.
Comment 10 Alasdair Kergon 2007-11-04 19:57:08 EST
Interestingly pvdisplay and pvs use different methods to get the PV size to display.
Comment 11 Alasdair Kergon 2007-11-04 20:54:03 EST
Updated upstream CVS for 2.02.29.

- Ignore and fix up large device sizes found in metadata.
- Adjusted the numbers in that pvdisplay line: 'not usable' is either any space
not used by the actual physical extents or, if the disk is too small for the PV,
it's the missing space.

Note You need to log in before you can comment on or make changes to this bug.