Bug 1116158 - systemd-215-1.fc21 breaks rawhide buildroot
Summary: systemd-215-1.fc21 breaks rawhide buildroot
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mock
Version: rawhide
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miroslav Suchý
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-03 22:45 UTC by Kevin Fenzi
Modified: 2014-10-10 16:04 UTC (History)
14 users (show)

Fixed In Version: mock-1.1.41-3.fc19
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2014-07-20 03:24:38 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
[PATCH] backend: do not touch mtab (1.07 KB, patch)
2014-07-03 23:49 UTC, Igor Gnatenko
no flags Details | Diff
[PATCH v2] backend: do not touch mtab (1.56 KB, patch)
2014-07-03 23:52 UTC, Igor Gnatenko
no flags Details | Diff
[PATCH v3] backend: do not touch mtab (1.68 KB, patch)
2014-07-04 07:37 UTC, Igor Gnatenko
no flags Details | Diff
[PATCH v3] backend: do not touch mtab (1.69 KB, patch)
2014-07-04 07:39 UTC, Igor Gnatenko
no flags Details | Diff

Description Kevin Fenzi 2014-07-03 22:45:33 UTC
This may well be a mock or fedpkg bug thats exposed by a change in systemd, but filing here first since the change is in systemd. 

fedpkg sources (run in rawhide mock roots) fails with this systemd version in the build root. With: 

ERROR: [Errno 2] No such file or directory: '/var/lib/mock/f21-build-2179824-398929/root/etc/mtab'
Traceback (most recent call last):
  File "/usr/sbin/mock", line 306, in <module>
    def do_buildsrpm(config_opts, chroot, options, args):
  File "/usr/sbin/mock", line 579, in main
    chroot.chroot(args, options)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 832, in chroot
    self._setupFiles()
  File "<peak.util.decorators.rewrap wrapping mockbuild.backend._setupFiles at 0x0214A050>", line 3, in _setupFiles
    def _setupFiles(self): return __decorated(self)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/backend.py", line 552, in _setupFiles
    mockbuild.util.touch(item)
  File "<peak.util.decorators.rewrap wrapping mockbuild.util.touch at 0x02104B18>", line 3, in touch
    def touch(fileName): return __decorated(fileName)
  File "/usr/lib/python2.7/site-packages/mockbuild/trace_decorator.py", line 70, in trace
    result = func(*args, **kw)
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 77, in touch
    fo = open(fileName, 'w')
IOError: [Errno 2] No such file or directory: '/var/lib/mock/f21-build-2179824-398929/root/etc/mtab'

See http://koji.fedoraproject.org/koji/taskinfo?taskID=7105860 for example.

mtab has been added to systemd's tmpfiles.d/etc.conf:  

L+ /etc/mtab - - - - ../proc/self/mounts

I've untagged this systemd from the buildroot until we sort this out.

Comment 1 Lennart Poettering 2014-07-03 22:52:40 UTC
Why would mock touch /etc/mtab? THis is really wrong.

Comment 2 Igor Gnatenko 2014-07-03 23:49:50 UTC
Created attachment 914599 [details]
[PATCH] backend: do not touch mtab

I have not tested it, because I can't reproduce this bug in my local machine.

I've read systemd docs.
L, L+

    Create a symlink if it does not exist yet. If suffixed with + and a file already exists where the symlink is to be created, it will be removed and be replaced by the symlink. If the argument is omitted, symlinks to files with the same name residing in the directory /usr/share/factory/ are created.


because after creating fstab/yumlog we're going to mount dirs including /proc I think that's correctly to don't create mtab now.

Comment 3 Igor Gnatenko 2014-07-03 23:52:56 UTC
Created attachment 914600 [details]
[PATCH v2] backend: do not touch mtab

oh, I forget to commit making link /proc/self/mounts -> /proc/mounts

Comment 4 Lennart Poettering 2014-07-04 01:17:53 UTC
From your patch:

os.symlink("/proc/self/mounts", self.makeChrootPath("/proc/mounts"))

This looks wrong. You probably mean /etc/mtab, not /proc/mounts, right?

Comment 5 Igor Gnatenko 2014-07-04 07:36:47 UTC
(In reply to Lennart Poettering from comment #4)
> From your patch:
> 
> os.symlink("/proc/self/mounts", self.makeChrootPath("/proc/mounts"))
> 
> This looks wrong. You probably mean /etc/mtab, not /proc/mounts, right?
yes. I never should write patches at 3 A.M.

Comment 6 Igor Gnatenko 2014-07-04 07:37:14 UTC
Created attachment 914650 [details]
[PATCH v3] backend: do not touch mtab

Comment 7 Igor Gnatenko 2014-07-04 07:39:04 UTC
Created attachment 914651 [details]
[PATCH v3] backend: do not touch mtab

os.symlink("/proc/mounts", self.makeChrootPath('etc', 'mtab'))

now it's so. I think that's correctly

If we found etc/mtab in chroot - deleting it.

Comment 8 Miroslav Suchý 2014-07-04 13:43:17 UTC
Mock touch it because in past time (still in RHEL-5, RHEL-6) this file was owned by setup.rpm, which is not installed in buildroot.
In recent Fedoras this file is owned by util-linux, which does
  ln -sf /proc/mounts %{buildroot}/etc/mtab
so after _setupFiles(), which prepares chroot touch it, it is in Fedoras overwritten during installation of util-linux.

If we symlink that file directly - instead of touching it. Then there is no problem. In fact, it make situation even better, because 'df' will start working in el-5 chroots.

This part is IMO redundant.
+            if os.path.isfile(self.makeChrootPath('etc', 'mtab')):
+                os.remove(self.makeChrootPath('etc', 'mtab'))
But we will need this for transition period for el5/6 chroots created by old mock.

But anyway this is *NOT* fix to the problem (but it makes world better).
The core of the problem is that

    getLog().debug("touching file: %s" % fileName)
    fo = open(fileName, 'w')
in 
  File "/usr/lib/python2.7/site-packages/mockbuild/util.py", line 77, in touch

Is unable to create that file. IMHO only reason why this fail is that <chroot>/etc does not exist in that time. Which is weird.

I tried to reproduce it:
 mock -r fedora-rawhide-x86_64 --clean
 mock -r fedora-rawhide-x86_64 --init
 mock -r fedora-rawhide-x86_64 --intall fedpkg
 mock -r fedora-rawhide-x86_64 --shell
    fedpkg clone --anonymous util-linux
    cd util-linux
    fedpkg sources
It works for me. But I'm still getting systemd-214-5.fc21. I manually download version 215 of systemd and systemd-libs and install it there by "rpm -Uvh" and:
    fedpkg clone --anonymous util-linux
    cd util-linux
    fedpkg sources
still works for me.

Minimal manual reproducer would be handy. Is somebody able to reproduce it? I mean outside of Koji.

Comment 9 Lennart Poettering 2014-07-04 13:52:12 UTC
Hmm, what's the precise state of this bug? Can we undo the untagging of the systemd build?

Comment 10 Kevin Fenzi 2014-07-04 16:22:14 UTC
(In reply to Lennart Poettering from comment #9)
> Hmm, what's the precise state of this bug? Can we undo the untagging of the
> systemd build?

No.

If the fix is in mock, we need to make updated mock builds and update all the builders with it. Also, it would be probibly polite to give a few days for maintainers to get the new mock version so they could actually build rawhide packages locally. Also copr and assorted other places would need to update to be able to build rawhide packages. 

Short term, perhaps we could drop the mtab handling in systemd until fixed mock rolls out?

Comment 11 Miroslav Suchý 2014-07-05 09:38:45 UTC
The fix is not in Mock. Because I could not reproduce it (and therefore test it) and Igor claimed the same in #2.
Kevin do you have reproducer? I mean outside of Koji.

Comment 12 Kevin Fenzi 2014-07-05 16:53:49 UTC
You need to have systemd-215 in your repos when you make the chroot. 

I reproduced it here just fine with your steps, but the --shell step ends in: 

$ mock -r fedora-rawhide-x86_64 --shell
No handlers could be found for logger "trace.__main__"
INFO: mock.py version 1.1.39 starting...
Start: init plugins
INFO: selinux enabled
Finish: init plugins
Start: run
Start: lock buildroot
Start: device setup
Finish: device setup
ERROR: [Errno 2] No such file or directory: '/var/lib/mock/fedora-rawhide-x86_64/root/etc/mtab'
Finish: lock buildroot

Comment 13 Zbigniew Jędrzejewski-Szmek 2014-07-05 17:02:34 UTC
It's probably because the link is created as 'etc/mtab -> ../proc/self/mounts', not 'etc/mtab -> /proc/self/mounts'. I'll disable it for now in systemd, so we can test the new release of systemd without waiting for mock.

Comment 14 Igor Gnatenko 2014-07-05 17:53:34 UTC
(In reply to Zbigniew Jędrzejewski-Szmek from comment #13)
> It's probably because the link is created as 'etc/mtab ->
> ../proc/self/mounts', not 'etc/mtab -> /proc/self/mounts'. I'll disable it
> for now in systemd, so we can test the new release of systemd without
> waiting for mock.

If so - I think my patch to mock should fix the problem.

Comment 15 Kevin Fenzi 2014-07-06 19:37:58 UTC
related: /etc/resolv.conf handling seems to break livecd-tools: https://bugzilla.redhat.com/show_bug.cgi?id=1116651

Comment 16 Miroslav Suchý 2014-07-17 10:16:18 UTC
Committed as
* 2a32c85 make /etc/mtab symlink to /proc/self/mounts [BZ# 1116158]
in msuchy-work branch.

Comment 17 Igor Gnatenko 2014-07-17 10:53:53 UTC
seems you have not pushed it ;)

Comment 18 Miroslav Suchý 2014-07-17 11:00:07 UTC
Indeed. Done.

Comment 19 Igor Gnatenko 2014-07-17 11:21:37 UTC
(In reply to Miroslav Suchý from comment #16)
> Committed as
> * 2a32c85 make /etc/mtab symlink to /proc/self/mounts [BZ# 1116158]
> in msuchy-work branch.
heh, again. you comitted my patch not from my name ;)

Comment 20 Miroslav Suchý 2014-07-17 11:28:59 UTC
That's because I clicked 'Raw unified' instead of 'View'. Damned BZ. Sorry. Definitely not intentional.

Comment 21 Igor Gnatenko 2014-07-17 11:33:01 UTC
(In reply to Miroslav Suchý from comment #20)
> That's because I clicked 'Raw unified' instead of 'View'. Damned BZ. Sorry.
> Definitely not intentional.
no problem. btw, because u're working in non-master branch and not released version you just can amend commit.
git rebase -i 2a32c851078c1b6b6d4dcb4989bd9be794df3f67^
mark this commit as 'edit'
git commit --amend --no-edit --author="Igor Gnatenko <i.gnatenko.brain>"
git rebase --continue
git push origin msuchy-work -f

Comment 22 Fedora Update System 2014-07-18 02:13:22 UTC
mock-1.1.40-2.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.40-2.fc20

Comment 23 Fedora Update System 2014-07-18 02:13:48 UTC
mock-1.1.40-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.40-2.fc19

Comment 24 Fedora Update System 2014-07-18 02:14:10 UTC
mock-1.1.40-2.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.40-2.el6

Comment 25 Fedora Update System 2014-07-18 17:45:29 UTC
mock-1.1.41-1.fc20 has been submitted as an update for Fedora 20.
https://admin.fedoraproject.org/updates/mock-1.1.41-1.fc20

Comment 26 Fedora Update System 2014-07-18 17:45:55 UTC
mock-1.1.41-1.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.41-1.fc19

Comment 27 Fedora Update System 2014-07-18 17:46:22 UTC
mock-1.1.41-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/mock-1.1.41-1.el6

Comment 28 Fedora Update System 2014-07-19 05:58:12 UTC
Package mock-1.1.41-1.fc20:
* should fix your issue,
* was pushed to the Fedora 20 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing mock-1.1.41-1.fc20'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-8492/mock-1.1.41-1.fc20
then log in and leave karma (feedback).

Comment 29 Fedora Update System 2014-07-20 03:24:38 UTC
mock-1.1.41-1.fc20 has been pushed to the Fedora 20 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 30 Fedora Update System 2014-07-22 18:07:57 UTC
mock-1.1.41-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 31 Fedora Update System 2014-09-16 16:46:57 UTC
mock-1.1.41-2.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.41-2.fc19

Comment 32 Fedora Update System 2014-09-30 09:00:12 UTC
mock-1.1.41-3.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/mock-1.1.41-3.fc19

Comment 33 Fedora Update System 2014-10-10 16:04:20 UTC
mock-1.1.41-3.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.