Description of problem: I'm trying to create a large FS over 5x 1TB drives When I create a physical volume it correctly assigns a size of 3.64 TB When I then progress to make a volume group pvdisplay then shows this has changed to 8192.00 EB (ExaBytes of storage) errrrrr.... Version-Release number of selected component (if applicable): lvm2-2.02.24-1.fc7 kernel-2.6.23.1-10.fc7 How reproducible: (Heres a great chance to get your hands on 5 TB drives for testing...) grab 5 1TB drives create a raid5 set (your milage may vary but something like this... ) mdadm -C -l5 -n5 -c256 /dev/md1 /dev/sdb1 /dev/sdc1 /dev/sdk1 /dev/sdl1 /dev/sdt1 create the physical volume pvcreate /dev/md1 display the physical volume pvdisplay -> --- NEW Physical volume --- PV Name /dev/md1 VG Name PV Size 3.64 TB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN Ok,. looks correct, now create a volume group vgcreate VolGroup00 /dev/md1 Volume group "VolGroup00" successfully created vgdisplay -> --- Volume group --- VG Name VolGroup00 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 3.64 TB PE Size 4.00 MB Total PE 953866 Alloc PE / Size 0 / 0 Free PE / Size 953866 / 3.64 TB VG UUID 7uQVFV-QbEp-jpJg-jXix-G0op-igNS-Cb6Xac Ok, it still looks reasonable,.. the VG size is the same as what we started out with BUT now type pvdisplay... pvdisplay -> --- Physical volume --- PV Name /dev/md1 VG Name VolGroup00 PV Size 8192.00 EB / not usable 8192.00 EB Allocatable yes PE Size (KByte) 4096 Total PE 953866 Free PE 953866 Allocated PE 0 PV UUID PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN Note that now the PV size has changed to 8192.00 EB ?!? err ok... what does pvscan say? pvscan -> PV /dev/md1 VG VolGroup00 lvm2 [3.64 TB / 3.64 TB free] hurm,... did vgcreate do something wierd? lets delete it vgremove VolGroup00 pvdisplay -> --- NEW Physical volume --- PV Name /dev/md1 VG Name PV Size 3.64 TB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID PVSW1A-j502-ekKB-zIFQ-N8BA-dujK-mx0JcN Ideas? I /think/ it's actually ok but the PV size change does concern me esp when I start to fill up this FS to any serious extent Phil =--=
On an additional confused note,.. this is lvm2 (well the utilities claim to be from the lvm2 package... I ask because of the -v output below [root@emerald ~]# pvscan -v File descriptor 5 left open File descriptor 7 left open Wiping cache of LVM-capable devices Wiping internal VG cache Walking through all physical volumes Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2 Fixing up missing format1 size (2.27 TB) for PV /dev/md0 PV /dev/sdd2 VG VolGrp00 lvm2 [465.66 GB / 32.00 MB free] PV /dev/md1 VG VolGroup00 lvm2 [3.64 TB / 3.64 TB free] PV /dev/md0 VG VolGroup01 lvm2 [2.27 TB / 0 free] Total: 3 [2.37 TB] / in use: 3 [2.37 TB] / in no VG: 0 [0 ] [root@emerald ~]# pvdisplay -v File descriptor 5 left open File descriptor 7 left open Scanning for physical volume names Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2 Fixing up missing format1 size (2.27 TB) for PV /dev/md0 Fixing up missing format1 size (465.66 GB) for PV /dev/sdd2 --- Physical volume --- PV Name /dev/sdd2 VG Name VolGrp00 PV Size 465.66 GB / not usable 3.56 MB Allocatable yes PE Size (KByte) 32768 Total PE 14901 Free PE 1 Allocated PE 14900 PV UUID MLKmma-3vQi-tR5w-Us5E-TAjS-4pIt-Mm9kNJ --- Physical volume --- PV Name /dev/md1 VG Name VolGroup00 PV Size 8192.00 EB / not usable 8192.00 EB Allocatable yes PE Size (KByte) 4096 Total PE 953866 Free PE 953866 Allocated PE 0 PV UUID KHMh9T-igJh-SGkg-6JGK-jPqc-6EGf-3NvKUE Fixing up missing format1 size (2.27 TB) for PV /dev/md0 --- Physical volume --- PV Name /dev/md0 VG Name VolGroup01 PV Size 2.27 TB / not usable 2.88 MB Allocatable yes (but full) PE Size (KByte) 4096 Total PE 596171 Free PE 0 Allocated PE 596171 PV UUID KWGw1x-LmYe-PqvN-V5uF-RlgV-rbhe-WP74Yb Am I to infer from the above that the pv and vg are being created in lvm1 format?! (since the tools don't complain about missing lvm1 formatting) Maybe I should just pull down the src and lok myself.. something is just not quite right Phil =--=
Never mind I'm going nuts figuring this out pvdisplay -> PV Size 8192.00 EB / not usable 8192.00 EB [root@emerald ~]# pvs File descriptor 5 left open File descriptor 7 left open PV VG Fmt Attr PSize PFree /dev/md0 VolGroup01 lvm2 a- 2.27T 0 /dev/md1 VolGroup00 lvm2 a- 3.64T 3.64T /dev/sdd2 VolGrp00 lvm2 a- 465.66G 32.00M [root@emerald ~]# vgs File descriptor 5 left open File descriptor 7 left open VG #PV #LV #SN Attr VSize VFree VolGroup00 1 0 0 wz--n- 3.64T 3.64T VolGroup01 1 1 0 wz--n- 2.27T 0 VolGrp00 1 2 0 wz--n- 465.66G 32.00M Phil =--=
Could try the F8 packages to see if we already fixed this (update device-mapper + lvm2 packages together). The 'format1' message looks like a mistake in the message text. I'll fix it.
vgcfgbackup -> # Generated by LVM2: Sun Nov 4 15:50:54 2007 contents = "Text Format Volume Group" version = 1 description = "Created *after* executing 'vgcfgbackup'" creation_host = "emerald" # Linux emerald 2.6.23.1-10.local.fc7 #1 SMP Fri Nov 2 19:37:26 GMT 2007 x86_64 creation_time = 1194191454 # Sun Nov 4 15:50:54 2007 VolGroup01 { id = "Qr20q0-lkea-Iesc-aIsW-4LA4-0L2I-f427J4" seqno = 4 status = ["RESIZEABLE", "READ", "WRITE"] extent_size = 8192 # 4 Megabytes max_lv = 0 max_pv = 0 physical_volumes { pv0 { id = "KWGw1x-LmYe-PqvN-V5uF-RlgV-rbhe-WP74Yb" device = "/dev/md0" # Hint only status = ["ALLOCATABLE"] dev_size = 4883838720 # 2.27421 Terabytes pe_start = 384 pe_count = 596171 # 2.27421 Terabytes } } logical_volumes { LogVol00 { id = "fmlNTo-FIaf-n6VM-41bh-xvw5-wvpA-21Ze0P" status = ["READ", "WRITE", "VISIBLE"] segment_count = 1 segment1 { start_extent = 0 extent_count = 596171 # 2.27421 Terabytes type = "striped" stripe_count = 1 # linear stripes = [ "pv0", 0 ] } } } } attaching dd if=/dev/md1 bs=512 count=384 of=/metadata
Created attachment 247601 [details] Metadata Metadata fields off /dev/md1
Helps if I add the RIGHT vg config file... <grumble> not my day,... sorry about that # Generated by LVM2: Sun Nov 4 15:50:54 2007 contents = "Text Format Volume Group" version = 1 description = "Created *after* executing 'vgcfgbackup'" creation_host = "emerald" # Linux emerald 2.6.23.1-10.local.fc7 #1 SMP Fri Nov 2 19:37:26 GMT 2007 x86_64 creation_time = 1194191454 # Sun Nov 4 15:50:54 2007 VolGroup00 { id = "gVFkvt-cN7R-VEg0-Uk3Q-FW0d-zOeV-anNquR" seqno = 2 status = ["RESIZEABLE", "READ", "WRITE"] extent_size = 8192 # 4 Megabytes max_lv = 0 max_pv = 0 physical_volumes { pv0 { id = "ztSu45-Wx9U-Maw0-1smV-zKDC-8604-9xJdRZ" device = "/dev/md1" # Hint only status = ["ALLOCATABLE"] dev_size = 18446744073709551615 # 8 (null) pe_start = 384 pe_count = 953866 # 3.63871 Terabytes } } logical_volumes { LogVol00 { id = "rNbnws-9es8-wNOl-RZqU-p2YI-lunc-ttfHhn" status = ["READ", "WRITE", "VISIBLE"] segment_count = 1 segment1 { start_extent = 0 extent_count = 953866 # 3.63871 Terabytes type = "striped" stripe_count = 1 # linear stripes = [ "pv0", 0 ] } } } }
Created attachment 247611 [details] VolGroup00 archive files Tar gzip of the VolGroup00 archives from /etc/lvm/archive
Still not sure how this came about, but I've noticed similar before. I've added the LVM2 version number to the metadata from now on to help with future diagnosis. As I explained on IRC, this field is only used for display and after the PV is removed from the VG with vgreduce or vgremove, so it's not particularly important. We'll change the code to detect 'rogue' values like this and ignore them.
Again, as on IRC, the workaround is to use vgcfgbackup and vgcfgrestore, editing the field back to its correct value.
Interestingly pvdisplay and pvs use different methods to get the PV size to display.
Updated upstream CVS for 2.02.29. - Ignore and fix up large device sizes found in metadata. - Adjusted the numbers in that pvdisplay line: 'not usable' is either any space not used by the actual physical extents or, if the disk is too small for the PV, it's the missing space.