There's a recent new problem in F18 which causes appliance creation to fail in the koji buildsystem, because loopback devices aren't created properly inside the chroot. Andy Grimm has created a patch http://fpaste.org/LwiV/ (also, follows) which we'd like to apply to work around the issue. There may be a deeper issue with device-mapper that is the root cause, but this is a small and relatively unobtrusive fix which should get us back to building images without much implication. Also nominating as F18 blocker, as functioning EC2 images is a final release criteria and we'll need this and it can't be an update. ---- --- appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-08-09 03:29:33.000000000 -0700 +++ appliance-tools-006.2.orig/appcreate/partitionedfs.py 2012-12-14 10:36:57.982845602 -0800 @@ -173,6 +173,21 @@ if rc != 0: raise MountError("Failed to map partitions for '%s'" % d['disk'].device) + if not os.path.exists(mapperdev): + dmsetup = subprocess.Popen(["dmsetup", "ls"], + stdout=subprocess.PIPE) + dmsetupOutput = dmsetup.communicate()[0].split('\n') + for line in dmsetupOutput: + m = re.match("%s\s+\((\d+):(\d+)\)" % os.path.basename(mapperdev), line) + if not m: + continue + major, minor = m.groups()[:2] + rc = subprocess.call(["/bin/mknod", mapperdev, "b", major, minor]) + if rc != 0: + raise MountError("Failed to map partitions for '%s'" % + d['disk'].device) + break + d['mapped'] = True
Upon further review, appliance-creator tasks in koji do bind mount /dev (thanks dgilmore for pointing that out). Given that, we are back to this being a race condition. I've verified that I can reproduce it, and sleeping even a fraction of a second (0.1 or less) is enough for the problem to go away. I've also verified that in doing os.listdir("/dev/mapper") immediately after the kpartx call returns, the /dev/mapper symlink is not present yet. So this is the same issue as 885385 after all. Sorry for the confusion.
Let's clarify one more time. It's a race, but not the same race as 885385 This one is because kpartx's -s option doesn't work when udev is not running in the chroot.
device-mapper-multipath packages that reinstate the libdm failback device creation for multipath and kpartx when called in sync mode are available at: http://koji.fedoraproject.org/koji/taskinfo?taskID=4792365 Let me know if this fixes the problem.
Confirmed that the above build fixes the problem. Thanks, Ben.
Discussed at 2012-12-17 blocker review meeting: http://meetbot.fedoraproject.org/fedora-bugzappers/2012-12-17/f18final-blocker-review-5.2012-12-17-16.40.log.txt . Accepted as a blocker per criterion "The release must boot successfully as Xen DomU with releases providing a functional, supported Xen Dom0 and widely used cloud providers utilizing Xen. This does not include any issues limited to the release functioning as Xen Dom0" - this bug blocks creation of the EC2 image, which is a major case of that criterion.
fix added. multipath and kpartx now use the libdm failback device creation when called in sync mode.
device-mapper-multipath-0.4.9-36.fc18 has been submitted as an update for Fedora 18. https://admin.fedoraproject.org/updates/device-mapper-multipath-0.4.9-36.fc18
Package device-mapper-multipath-0.4.9-36.fc18: * should fix your issue, * was pushed to the Fedora 18 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing device-mapper-multipath-0.4.9-36.fc18' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-20573/device-mapper-multipath-0.4.9-36.fc18 then log in and leave karma (feedback).
device-mapper-multipath-0.4.9-36.fc18 has been pushed to the Fedora 18 stable repository. If problems still persist, please make note of it in this bug report.