Bug 887357

Summary:	workaround for for appliance creation in koji
Product:	[Fedora] Fedora	Reporter:	Matthew Miller <mattdm>
Component:	device-mapper-multipath	Assignee:	Ben Marzinski <bmarzins>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	18	CC:	agk, agrimm, apevec, awilliam, bmarzins, dhuff, dwysocha, fdinitto, gholms, heinzm, lvm-team, mgoldman, msnitzer, prajnoha, prockai, robatino, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:	AcceptedBlocker
Fixed In Version:	device-mapper-multipath-0.4.9-36.fc18	Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-01-02 21:51:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	752661

Description Matthew Miller 2012-12-14 19:24:20 UTC

There's a recent new problem in F18 which causes appliance creation to fail in the koji buildsystem, because loopback devices aren't created properly inside the chroot.

Andy Grimm has created a patch http://fpaste.org/LwiV/ (also, follows) which we'd like to apply to work around the issue.

There may be a deeper issue with device-mapper that is the root cause, but this is a small and relatively unobtrusive fix which should get us back to building images without much implication.

Also nominating as F18 blocker, as functioning EC2 images is a final release criteria and we'll need this and it can't be an update.


----

    --- appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-08-09 03:29:33.000000000 -0700
    +++ appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-12-14 10:36:57.982845602 -0800
    @@ -173,6 +173,21 @@
    if rc != 0:
    raise MountError("Failed to map partitions for '%s'" %
    d['disk'].device)
    + if not os.path.exists(mapperdev):
    + dmsetup = subprocess.Popen(["dmsetup", "ls"],
    + stdout=subprocess.PIPE)
    + dmsetupOutput = dmsetup.communicate()[0].split('\n')
    + for line in dmsetupOutput:
    + m = re.match("%s\s+\((\d+):(\d+)\)" % os.path.basename(mapperdev), line)
    + if not m:
    + continue
    + major, minor = m.groups()[:2]
    + rc = subprocess.call(["/bin/mknod", mapperdev, "b", major, minor])
    + if rc != 0:
    + raise MountError("Failed to map partitions for '%s'" %
    + d['disk'].device)
    + break
    +
    d['mapped'] = True

Comment 1 Andy Grimm 2012-12-14 20:25:27 UTC

Upon further review, appliance-creator tasks in koji do bind mount /dev (thanks dgilmore for pointing that out).  Given that, we are back to this being a race condition.  I've verified that I can reproduce it, and sleeping even a fraction of a second (0.1 or less) is enough for the problem to go away.  I've also verified that in doing os.listdir("/dev/mapper") immediately after the kpartx call returns, the /dev/mapper symlink is not present yet.  So this is the same issue as 885385 after all.  Sorry for the confusion.

Comment 2 Andy Grimm 2012-12-14 21:47:25 UTC

Let's clarify one more time.  It's a race,  but not the same race as 885385
This one is because kpartx's -s option doesn't work when udev is not running in the chroot.

Comment 3 Ben Marzinski 2012-12-14 22:46:54 UTC

device-mapper-multipath packages that reinstate the libdm failback device creation for multipath and kpartx when called in sync mode are available at:

http://koji.fedoraproject.org/koji/taskinfo?taskID=4792365

Let me know if this fixes the problem.

Comment 4 Andy Grimm 2012-12-15 00:05:23 UTC

Confirmed that the above build fixes the problem.  Thanks, Ben.

Comment 5 Adam Williamson 2012-12-17 19:01:16 UTC

Discussed at 2012-12-17 blocker review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-17/f18final-blocker-review-5.2012-12-17-16.40.log.txt . Accepted as a blocker per criterion "The release must boot successfully as Xen DomU with releases providing a functional, supported Xen Dom0 and widely used cloud providers utilizing Xen. This does not include any issues limited to the release functioning as Xen Dom0" - this bug blocks creation of the EC2 image, which is a major case of that criterion.

Comment 6 Ben Marzinski 2012-12-17 19:52:19 UTC

fix added. multipath and kpartx now use the libdm failback device creation when called in sync mode.

Comment 7 Fedora Update System 2012-12-17 20:13:44 UTC

device-mapper-multipath-0.4.9-36.fc18 has been submitted as an update for Fedora 18.
https://admin.fedoraproject.org/updates/device-mapper-multipath-0.4.9-36.fc18

Comment 8 Fedora Update System 2012-12-18 21:26:41 UTC

Package device-mapper-multipath-0.4.9-36.fc18:
* should fix your issue,
* was pushed to the Fedora 18 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing device-mapper-multipath-0.4.9-36.fc18'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-20573/device-mapper-multipath-0.4.9-36.fc18
then log in and leave karma (feedback).

Comment 9 Fedora Update System 2013-01-02 21:51:35 UTC

device-mapper-multipath-0.4.9-36.fc18 has been pushed to the Fedora 18 stable repository.  If problems still persist, please make note of it in this bug report.