Created attachment 1021010 [details] Log from failed test run. Description of problem: When virtual snapshot is created like this: "lvcreate --type snapshot -s -l 100%FREE -n $lv $vg --virtualsize 1024T" There appears to be tiny racy moment, when the command in cluster has already returned to a user, but udev links are still not properly pointing to a right device. Attached is full test log. Version-Release number of selected component (if applicable): 2.02.118 How reproducible: Occasionally with lvm2 test: lvcreate-large.sh in cluster. Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
As prajnoha notified me: Following comments applies for attachment: Top-level device @PREFIX@vg-LV has udev link in /dev: LV -> ../dm-4 However udevdb already thinks differently: P: /devices/virtual/block/dm-7 S: @PREFIX@vg/LV S: mapper/@PREFIX@vg-LV P: /devices/virtual/block/dm-4 S: mapper/@PREFIX@vg-LV_vorigin-real so it's clear udev fault - it knows how the links should look like, but it fails to managed those links accordingly.
Created attachment 1043209 [details] Trace from failing vgremove This trace show our problem with snapshot and volume removal. In a single udev transaction we run 'resume' (ADD) 'remove' (DEL) event for COW volume - and while lvm2 code collapses this into a single 'DEL' it happens with varios version of udev the -cow is left in /dev file system. It's also noticeable that even usage of 'verify_udev_operations = 1' will not help - since in the moment of check link is removed, but because of racy udev behavior it is restored later back
Created attachment 1043212 [details] Udev event trace for the vgremove log Trace shows that change & 2x remove events are recieved by udev. The order of processing and internal udev races result into final wrong udev state. Also it looks like udev in latest 'rawhide' (f23) behaves here much better - so likely races were addressed - but older systems with older udev are fragile. There are possible some improvement of lvm2 code - but their implementation time is unknown. Key fix likely is - that activation of 'COW' LV should be completely avoided and eliminated - just like we already implemented special code for uncaching cached LV where we managed to do all steps within single lock call. So as for now - real bug is in udev - but lvm2 could do a better job and avoid using ADD & DEL for a single device within a single udev cookie.
This bug appears to have been reported against 'rawhide' during the Fedora 23 development cycle. Changing version to '23'. (As we did not run this process for some time, it could affect also pre-Fedora 23 development cycle bugs. We are very sorry. It will help us with cleanup during Fedora 23 End Of Life. Thank you.) More information and reason for this action is here: https://fedoraproject.org/wiki/BugZappers/HouseKeeping/Fedora23
This bug appears to have been reported against 'rawhide' during the Fedora 24 development cycle. Changing version to '24'. More information and reason for this action is here: https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora24#Rawhide_Rebase
This message is a reminder that Fedora 24 is nearing its end of life. Approximately 2 (two) weeks from now Fedora will stop maintaining and issuing updates for Fedora 24. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '24'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 24 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Fedora 24 changed to end-of-life (EOL) status on 2017-08-08. Fedora 24 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. If you are unable to reopen this bug, please file a new report against the current release. If you experience problems, please add a comment to this bug. Thank you for reporting this bug and we are sorry it could not be fixed.
It's supposed this bug is no longer present with FC26 - udev has fixed race in links updating and lvm2 also adding missing udev synchronization point to avoid streaming REMOVE + ADD into a single cookie udev transaction for snapshot creation.