Created attachment 1432811 [details] rdsosreport.txt Description of problem: Got a pair of storage in Linux mdraid software raid 1. /boot is on raid 1. (md127) / is on LVM, the LVM is on another raid 1. (md126) Originally It is working fine in FC27. After upgrade to FC28. OS is not bootable with the new fc28 kernel. I can boot into the old fc27 kernel. But if I run dracut to replace the initramfs again, the OS is not bootable. Version-Release number of selected component (if applicable): kernel-4.16.6-202.fc27.x86_64 dracut-047-8.git20180305.fc28.x86_64 How reproducible: Always Steps to Reproduce: 1. FC27 upgrade to FC28 2. Boot to FC28 new kernel 3. cannot boot Actual results: Cannot boot into OS Expected results: Boot into OS Additional info: I can drop into rd.shell. The most important thing is that I can see my raid in /proc/mdstat. But there is problem for those LVM on the mdraid: WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk on /dev/sdb2 was already found on /dev/sda2. WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk on /dev/md126 was already found on /dev/sda2. WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk prefers device /dev/sda2 because device was seen first. WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk prefers device /dev/md126 because device size is correct. I am unable to activate the LV in rd.shell Attached is the rdsosreport.
Created attachment 1432974 [details] /etc/fstab
I have tried to downgrade dracut to: dracut-046-8.git20180105.fc27.x86_64.rpm dracut-config-rescue-046-8.git20180105.fc27.x86_64.rpm dracut-network-046-8.git20180105.fc27.x86_64.rpm And rebuild initramfs, the problem still persist.
(In reply to Patrick Dung from comment #0) > Additional info: > I can drop into rd.shell. > The most important thing is that I can see my raid in /proc/mdstat. > But there is problem for those LVM on the mdraid: > WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk on /dev/sdb2 was > already found on /dev/sda2. > WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk on /dev/md126 was > already found on /dev/sda2. > WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk prefers device > /dev/sda2 because device was seen first. > WARNING: PV mphetV-LMWe-se6x-86Bv-osHk-rVk2-K01UNk prefers device > /dev/md126 because device size is correct. It looks like the MD component filtering doesn't work. Can you get output from these, please? rpm -q lvm2 lvmconfig --type diff pvs -vvvv Thanks.
Hello Peter, Please find the info in below: # rpm -q lvm2 lvm2-2.02.177-4.fc28.x86_64 # lvmconfig --type diff devices { issue_discards=1 }
Created attachment 1433616 [details] pvs -vvvv on a live machine
> comment # is on a bootable / normal machine > If you need the output of pvs -vvvv when it hanged. I need to arrange time to reboot and get the result. Please refer to rdsosreport for info first.
Created attachment 1433786 [details] output of pvs -vvvv when having problem. in dracut shell output of pvs -vvvv when having problem. in dracut shell
Created attachment 1433787 [details] output of vgs -vvvv when having problem. in dracut shell
Created attachment 1433788 [details] output of lvs -vvvv when having problem. in dracut shell
Hello Peter, Please note the requested command are run on the problem machine. Please kindly check the above attachment. Thanks.
Workaround. I have downgraded lvm2 packages to FC27 and rebuild initramfs. Then I can use the new FC28 kernel and the machine boot normally. Downloading Packages: (1/6): device-mapper-1.02.144-1.fc27.x86_64.rpm 634 kB/s | 357 kB 00:00 (2/6): device-mapper-libs-1.02.144-1.fc27.x86_64.rpm 563 kB/s | 390 kB 00:00 (3/6): device-mapper-event-1.02.144-1.fc27.x86_64.rpm 1.2 MB/s | 254 kB 00:00 (4/6): device-mapper-event-libs-1.02.144-1.fc27.x86_64.rpm 1.2 MB/s | 254 kB 00:00 (5/6): lvm2-2.02.175-1.fc27.x86_64.rpm 1.1 MB/s | 1.4 MB 00:01 (6/6): lvm2-libs-2.02.175-1.fc27.x86_64.rpm 1.6 MB/s | 1.1 MB 00:00 ----------------------------------------------------------------------------------------------------------------------------------------- Total 1.4 MB/s | 3.7 MB 00:02 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: device-mapper-libs-1.02.144-1.fc27.x86_64 1/1 Downgrading : device-mapper-libs-1.02.144-1.fc27.x86_64 1/12 Running scriptlet: device-mapper-libs-1.02.144-1.fc27.x86_64 1/12 Downgrading : device-mapper-1.02.144-1.fc27.x86_64 2/12 Downgrading : device-mapper-event-libs-1.02.144-1.fc27.x86_64 3/12 Running scriptlet: device-mapper-event-libs-1.02.144-1.fc27.x86_64 3/12 Downgrading : device-mapper-event-1.02.144-1.fc27.x86_64 4/12 Running scriptlet: device-mapper-event-1.02.144-1.fc27.x86_64 4/12 Downgrading : lvm2-libs-2.02.175-1.fc27.x86_64 5/12 Running scriptlet: lvm2-libs-2.02.175-1.fc27.x86_64 5/12 Downgrading : lvm2-2.02.175-1.fc27.x86_64 6/12 Running scriptlet: lvm2-2.02.175-1.fc27.x86_64 6/12 Running scriptlet: lvm2-2.02.177-4.fc28.x86_64 7/12 Erasing : lvm2-2.02.177-4.fc28.x86_64 7/12 Running scriptlet: lvm2-2.02.177-4.fc28.x86_64 7/12 Erasing : lvm2-libs-2.02.177-4.fc28.x86_64 8/12 Running scriptlet: lvm2-libs-2.02.177-4.fc28.x86_64 8/12 Running scriptlet: device-mapper-event-1.02.146-4.fc28.x86_64 9/12 Erasing : device-mapper-event-1.02.146-4.fc28.x86_64 9/12 Erasing : device-mapper-1.02.146-4.fc28.x86_64 10/12 Erasing : device-mapper-event-libs-1.02.146-4.fc28.x86_64 11/12 Running scriptlet: device-mapper-event-libs-1.02.146-4.fc28.x86_64 11/12 Erasing : device-mapper-libs-1.02.146-4.fc28.x86_64 12/12 Running scriptlet: device-mapper-libs-1.02.146-4.fc28.x86_64 12/12 Verifying : lvm2-2.02.175-1.fc27.x86_64 1/12 Verifying : device-mapper-1.02.144-1.fc27.x86_64 2/12 Verifying : device-mapper-libs-1.02.144-1.fc27.x86_64 3/12 Verifying : device-mapper-event-1.02.144-1.fc27.x86_64 4/12 Verifying : lvm2-libs-2.02.175-1.fc27.x86_64 5/12 Verifying : device-mapper-event-libs-1.02.144-1.fc27.x86_64 6/12 Verifying : device-mapper-1.02.146-4.fc28.x86_64 7/12 Verifying : device-mapper-event-1.02.146-4.fc28.x86_64 8/12 Verifying : device-mapper-event-libs-1.02.146-4.fc28.x86_64 9/12 Verifying : device-mapper-libs-1.02.146-4.fc28.x86_64 10/12 Verifying : lvm2-2.02.177-4.fc28.x86_64 11/12 Verifying : lvm2-libs-2.02.177-4.fc28.x86_64 12/12 Downgraded: device-mapper.x86_64 1.02.144-1.fc27 device-mapper-event.x86_64 1.02.144-1.fc27 device-mapper-event-libs.x86_64 1.02.144-1.fc27 device-mapper-libs.x86_64 1.02.144-1.fc27 lvm2.x86_64 2.02.175-1.fc27 lvm2-libs.x86_64 2.02.175-1.fc27 Complete!
OK, I've managed to reproduce, the reproducer is simple: - using older MD metadata with signature at the end of the device, so MD metadata <= 1.0 (this has the consequence that if we place a PV header on top of the MD device, the PV header is then also visible on MD mirrors directly because there's no offset caused by MD metadata that would make PV header invisible) - without lvmetad (this has the consequence that the persistent filter is not used in this case which is the problematic part as explained below) [0] fed/~ # lsblk -s /dev/md0 NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT md0 9:0 0 128M 0 raid1 ├─sda 8:0 0 128M 0 disk └─sdb 8:16 0 128M 0 disk [0] fed/~ # lvmconfig --type diff global { use_lvmetad=0 } Original version from F28 repository compiled with -O2 ====================================================== [0] fed/~ # pvs WARNING: PV 5kKBVR-OuNU-hBvC-Ifr4-c2aH-fVOR-yQlxit on /dev/md0 was already found on /dev/sda. WARNING: PV 5kKBVR-OuNU-hBvC-Ifr4-c2aH-fVOR-yQlxit on /dev/sdb was already found on /dev/sda. WARNING: PV 5kKBVR-OuNU-hBvC-Ifr4-c2aH-fVOR-yQlxit prefers device /dev/md0 because device size is correct. WARNING: PV 5kKBVR-OuNU-hBvC-Ifr4-c2aH-fVOR-yQlxit prefers device /dev/md0 because device size is correct. PV VG Fmt Attr PSize PFree /dev/md0 lvm2 --- <127.94m <127.94m /dev/vda2 fedora lvm2 a-- <19.50g 0 (from the pvs -vvvv log) #filters/filter-md.c:35 /dev/sda: Skipping md component device #device/dev-cache.c:1536 /dev/sda: Using device (8:0) #label/label.c:286 Reading label from device /dev/sda #device/dev-io.c:599 Opened /dev/sda RO O_DIRECT #device/dev-io.c:168 /dev/sda: Block size is 4096 bytes #device/dev-io.c:179 /dev/sda: Physical block size is 512 bytes #device/dev-io.c:255 Widening request for 2048 bytes at 0 to 4096 bytes at 0 on /dev/sda (for PV labels) #device/dev-io.c:96 Read /dev/sda: 4096 bytes (sync) at 0 (for PV labels) #label/label.c:167 /dev/sda: lvm2 label detected at sector 1 !!!THIS PART IS WRONG!!! We detected that the "sda" is an md component device and we incorectly continue to scan for PV and read the PV headers. Version compiled with -O0 ========================= [0] fed/~ # pvs PV VG Fmt Attr PSize PFree /dev/md0 lvm2 --- <127.94m <127.94m /dev/vda2 fedora lvm2 a-- <19.50g 0 (from the pvs -vvvv log) #filters/filter-md.c:35 /dev/sda: Skipping md component device !!! HERE, IT'S CORRECT!!! We don't continue to PV scan if we detect that the device is an MD component, we stop processing the device further. ======================= The problematic piece of the code is this one (in lib/filters/filter-persistent.c): static int _lookup_p(struct dev_filter *f, struct device *dev) { struct pfilter *pf = (struct pfilter *) f->private; void *l = dm_hash_lookup(pf->devices, dev_name(dev)); struct dm_str_list *sl; ...some code here... /* Uncached */ if (!l) { l = pf->real->passes_filter(pf->real, dev) ? PF_GOOD_DEVICE : PF_BAD_DEVICE; dm_list_iterate_items(sl, &dev->aliases) if (!dm_hash_insert(pf->devices, sl->str, l)) { log_error("Failed to hash alias to filter."); return 0; } } return (l == PF_BAD_DEVICE) ? 0 : 1; } The "l = pf->real->passes_filter(pf->real, dev) ? PF_GOOD_DEVICE : PF_BAD_DEVICE;" correctly returns "PF_BAD_DEVICE" for a device that is an MD component in both optimized and non-optimized versions. The difference is in the way the dm_list_iterate_items is processed: - in optimized version, it returns with "1" within dm_list_iterate_items without evaluating the "return" part as I see that in gdb. It's optimized, that's why I don't see it evaluating the last line with "return (l == PF_BAD_DEVICE) ? 0 : 1;". - in non-optimized version, it evaluates the last line as expected "return (l == PF_BAD_DEVICE) ? 0 : 1;" and hence it returns with "0" correctly to skip the device as it does not pass filters So we probably need to get the assembly code to compare that part in optimized and non-optimized version and possibly report this against GCC if we find the optimization is not correct and causes wrong evaluation here.
(Will probably try compiling with older gcc version...)
Created attachment 1437927 [details] objdump -S -d on lvm compiled with gcc 7.3 and 8.0 (In reply to Peter Rajnoha from comment #13) > (Will probably try compiling with older gcc version...) If compiled with older gcc version (gcc-7.3.1-1.fc28.x86_64), it works as expected so it seems this is a bug in gcc-8 optimization (I used gcc-8.1.1-1.fc28.x86_64). I'm attaching objdumps with disassembly of lvm binaries compiled under those different versions.
Jakub, what do you think?
Hello, I have the same problem. The only solution is downgrade the lvm2 package? What command I have to execute? Thanks.
I just upgrade to kernel-4.16.13-300.fc28.x86_64 and the problem is the same. Are there not any solution to this problems? Thanks.
See also bug Bug 1589444. A workaround that worked for me is to disable lvm's protection against changes with duplicates (since these are not actual duplicates) by setting: allow_changes_with_duplicate_pvs=1
Yes, I changed the same and now I can boot in fc28 kernels.
Hello Peter, I think the problem can be narrowed down to using the following code pattern: --- t.c: #define ONE ((void *)1) #define TWO ((void *)2) int f(void *p, int x) { if (p == ONE) return 0; if (!p) { p = x ? TWO : ONE; } return (p == ONE) ? 0 : 1; } u.c: #include <stdio.h> int f(void *p, int x); int main(int argc, char *argv[]) { printf("%d %d\n", f(NULL, 0), f(NULL, 1)); } --- % gcc t.c u.c && ./a.out 0 1 % gcc -O2 t.c u.c && ./a.out 1 1 --- A compiler bug? Don't know, but it wouldn't surprise me if the gcc developers will argue that ONE and TWO are not valid pointer values and the result of the entire program therefore is undefined.
Thanks Olle! Using a valid pointer to a static variable should fix the issue. Let's try that.
(In reply to Olle Liljenzin from comment #20) I'll have a look tomorrow. It started with http://gcc.gnu.org/r251264, but looks to me like a VRP issue to me. Meeting [1B, 2B] and ~[0B, 1B] EQUIVALENCES: { p_4(D) p_8 } (2 elements) to ~[0B, -1B] Found new range for p_2: ~[0B, -1B] The range [1, 2] for the x ? TWO : ONE result is correct, ~[0, 1] is also correct for the other cases where p falls through (if it is 1, then it returns early, if it is 0, then it is the x ? TWO : ONE), but they should merge into VARYING, not ~[0, -1].
lvm2-2.02.177-5.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-4fe9bf6535
Patrick, could you try testing with the above RPMs? You will need to rebuild initramfs: - after installing the package run `dracut -f` as root. Now rereading the comments, I am not sure it helps. Maybe I have misunderstood it, and it is still broken.
(In reply to Marian Csontos from comment #24) > Patrick, could you try testing with the above RPMs? > > You will need to rebuild initramfs: - after installing the package run > `dracut -f` as root. > > Now rereading the comments, I am not sure it helps. Maybe I have > misunderstood it, and it is still broken. I had installed the rpm from https://koji.fedoraproject.org/koji/buildinfo?buildID=1096618 And initramfs is rebuilt and then reboot. I can boot into OS and it seems fine. The duplicated PV does not appear now. # rpm -qf /usr/sbin/lvm lvm2-2.02.177-5.fc28.x86_64 # pvs PV VG Fmt Attr PSize PFree /dev/md127 vg00 lvm2 a-- <26.56g 0 # vgs VG #PV #LV #SN Attr VSize VFree vg00 1 2 0 wz--n- <26.56g 0 # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root vg00 -wi-ao---- <25.56g swap01 vg00 -wi-a----- 1.00g
lvm2-2.02.177-5.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-4fe9bf6535
I confirm that this update fixes the problem for me - thanks!
lvm2-2.02.177-5.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
I can confirm that this fixed the issue for bug 1593836. I can create vgs and lvs and everything looks good.
FYI: Just tested this on my server that was failing due to this issue. Both the root and home directories are are Raid5 and LVM2. Applied the new LVM2 and devicemapper updates in the repo. Device fails to boot with exactly the same symptoms. Appears that LVM is seeing PV on each of the 4 disk partitions in the MD array. This confuses LVM. Note that after installing the LVM and device-mapper files, I did dracut to rebuild the initramfs. This failed. I then forced a reinstallation of the latest Fedora 28 kernel to get it to do it's own dracut. This failed as well. I've reverted back to the old LVM2 from Fedora 27 and things started working again.
(In reply to Alex Lang from comment #30) > Note that after installing the LVM and device-mapper files, I did dracut to > rebuild the initramfs. This failed. I then forced a reinstallation of the > latest Fedora 28 kernel to get it to do it's own dracut. This failed as well. If dracut fails, your problem is likely elsewhere. Full /boot? Check `df /boot` output. Check logs. > > I've reverted back to the old LVM2 from Fedora 27 and things started working > again.
With this fix now in the stable repo - are there any plans to provide new, updated install images? otherwise users installing from ISO images would still run into the same situation...
(In reply to Alex Lang from comment #30) > FYI: Just tested this on my server that was failing due to this issue. Both > the root and home directories are are Raid5 and LVM2. Applied the new LVM2 > and devicemapper updates in the repo. Device fails to boot with exactly the > same symptoms. Appears that LVM is seeing PV on each of the 4 disk > partitions in the MD array. This confuses LVM. Alex, given individual disks in RAID5 contain different data, LVM cannot see them all as PVs, as only one disk would contain PV label - so it could see at most 2 disks as PVs - one of the RAID component and MD device - which would still be incorrect, but is likely addressed by the updated version. But to verify that you need to rebuild initramfs. So first let's get solved the problem with dracut unable to create initramfs. After that is solved, and the problem with boot persists, open a separate BZ please. Attach outputs of these commands: pvs vgs lvs -avo+devices lsblk blkid cat /proc/mdstat Check failing services during boot. Any errors in journal?
Jakub, we have a workaround published for LVM. Clone please if you need a Fedora tracker. Wolfgang, open a fedora rel-eng ticket - that's where creating ISO images should be discussed - https://pagure.io/releng/new_issue. Is the bug present in Live and installation images? Looks like it contains the affected version.
Done: https://pagure.io/releng/issue/7589 Yes, both live and install images have this problem. [That was my initial attempt to recover from the non-bootinng system.]
After re-reading my comment above, I realize that folks could have read this the wrong way. Dracut ran successfully (in both the manual case as well as when re-installing the latest kernel from DNF). So, in both cases I had a new initramfs. However, the boot with the new initramfs failed in both cases with LVM complaining that it couldn't scan the devices because of duplicates.
*** Bug 1592783 has been marked as a duplicate of this bug. ***
Just tried again with the existing LVM and device-mapper file that were updated a couple of weeks ago along with the newly update dracut and kernel 4.17.5. All this now seems to work on my machine now. Raid is correctly detected and managed at boot time now.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days