Bug 887357

Summary: workaround for for appliance creation in koji
Product: [Fedora] Fedora Reporter: Matthew Miller <mattdm>
Component: device-mapper-multipathAssignee: Ben Marzinski <bmarzins>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 18CC: agk, agrimm, apevec, awilliam, bmarzins, dhuff, dwysocha, fdinitto, gholms, heinzm, lvm-team, mgoldman, msnitzer, prajnoha, prockai, robatino, virt-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: AcceptedBlocker
Fixed In Version: device-mapper-multipath-0.4.9-36.fc18 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-01-02 16:51:32 EST Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Bug Depends On:    
Bug Blocks: 752661    

Description Matthew Miller 2012-12-14 14:24:20 EST
There's a recent new problem in F18 which causes appliance creation to fail in the koji buildsystem, because loopback devices aren't created properly inside the chroot.

Andy Grimm has created a patch http://fpaste.org/LwiV/ (also, follows) which we'd like to apply to work around the issue.

There may be a deeper issue with device-mapper that is the root cause, but this is a small and relatively unobtrusive fix which should get us back to building images without much implication.

Also nominating as F18 blocker, as functioning EC2 images is a final release criteria and we'll need this and it can't be an update.


----

    --- appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-08-09 03:29:33.000000000 -0700
    +++ appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-12-14 10:36:57.982845602 -0800
    @@ -173,6 +173,21 @@
    if rc != 0:
    raise MountError("Failed to map partitions for '%s'" %
    d['disk'].device)
    + if not os.path.exists(mapperdev):
    + dmsetup = subprocess.Popen(["dmsetup", "ls"],
    + stdout=subprocess.PIPE)
    + dmsetupOutput = dmsetup.communicate()[0].split('\n')
    + for line in dmsetupOutput:
    + m = re.match("%s\s+\((\d+):(\d+)\)" % os.path.basename(mapperdev), line)
    + if not m:
    + continue
    + major, minor = m.groups()[:2]
    + rc = subprocess.call(["/bin/mknod", mapperdev, "b", major, minor])
    + if rc != 0:
    + raise MountError("Failed to map partitions for '%s'" %
    + d['disk'].device)
    + break
    +
    d['mapped'] = True
Comment 1 Andy Grimm 2012-12-14 15:25:27 EST
Upon further review, appliance-creator tasks in koji do bind mount /dev (thanks dgilmore for pointing that out).  Given that, we are back to this being a race condition.  I've verified that I can reproduce it, and sleeping even a fraction of a second (0.1 or less) is enough for the problem to go away.  I've also verified that in doing os.listdir("/dev/mapper") immediately after the kpartx call returns, the /dev/mapper symlink is not present yet.  So this is the same issue as 885385 after all.  Sorry for the confusion.
Comment 2 Andy Grimm 2012-12-14 16:47:25 EST
Let's clarify one more time.  It's a race,  but not the same race as 885385
This one is because kpartx's -s option doesn't work when udev is not running in the chroot.
Comment 3 Ben Marzinski 2012-12-14 17:46:54 EST
device-mapper-multipath packages that reinstate the libdm failback device creation for multipath and kpartx when called in sync mode are available at:

http://koji.fedoraproject.org/koji/taskinfo?taskID=4792365

Let me know if this fixes the problem.
Comment 4 Andy Grimm 2012-12-14 19:05:23 EST
Confirmed that the above build fixes the problem.  Thanks, Ben.
Comment 5 Adam Williamson 2012-12-17 14:01:16 EST
Discussed at 2012-12-17 blocker review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-17/f18final-blocker-review-5.2012-12-17-16.40.log.txt . Accepted as a blocker per criterion "The release must boot successfully as Xen DomU with releases providing a functional, supported Xen Dom0 and widely used cloud providers utilizing Xen. This does not include any issues limited to the release functioning as Xen Dom0" - this bug blocks creation of the EC2 image, which is a major case of that criterion.
Comment 6 Ben Marzinski 2012-12-17 14:52:19 EST
fix added. multipath and kpartx now use the libdm failback device creation when called in sync mode.
Comment 7 Fedora Update System 2012-12-17 15:13:44 EST
device-mapper-multipath-0.4.9-36.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/device-mapper-multipath-0.4.9-36.fc18
Comment 8 Fedora Update System 2012-12-18 16:26:41 EST
Package device-mapper-multipath-0.4.9-36.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing device-mapper-multipath-0.4.9-36.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-20573/device-mapper-multipath-0.4.9-36.fc18
then log in and leave karma (feedback).
Comment 9 Fedora Update System 2013-01-02 16:51:35 EST
device-mapper-multipath-0.4.9-36.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.