Bug 797438

Summary: systemd always fails systemd-remount-api-vfs.service
Product: [Fedora] Fedora Reporter: Michal Jaegermann <michal>
Component: util-linuxAssignee: Karel Zak <kzak>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: rawhideCC: johannbg, jonathan, kzak, metherid, mluscon, mschmidt, notting, plautrba, systemd-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: util-linux-2.21-2.fc17 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-03-21 18:52:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
fstab as used on my rawhide installation none

Description Michal Jaegermann 2012-02-25 20:05:57 UTC
Description of problem:

After a new boot one consistently gets:

# systemctl --failed | cat
UNIT			    LOAD   ACTIVE SUB	 JOB DESCRIPTION
systemd-...-api-vfs.service loaded failed failed     Remount API VFS

In logs it is possible to find messages like these:


 systemd-remount-api-vfs[301]: mount: mount: can't find /sys in /etc/fstabmount:
 systemd-remount[301]: /bin/mount for /sys exited with exit status 1.
 systemd-remount[301]: /bin/mount for /dev/shm exited with exit status 1.
 [301]: /bin/mount for /proc exited with exit status 1.
 [301]: /bin/mount for /dev/pts exited with exit status 1.
 systemd[1]: systemd-remount-api-vfs.service: main process exited, code=exited, status=1
 systemd-remount-api-vfs[301]: can't find /dev/shm in /etc/fstab
 systemd-remount-api-vfs[301]: can't find /proc in /etc/fstab
 systemd-remount-api-vfs[301]: mount: can't find /dev/pts in /etc/fstab
 systemd[1]: Unit systemd-remount-api-vfs.service entered failed state.

"/etc/fstabmount"?  That looks like something which was glued together.  In any case things like /sys and /dev/pts and others are actually present in /etc/fstab.  Moreover all these "failed" mounts show up, not that surprisingly, in the running system.

Version-Release number of selected component (if applicable):
systemd-43-1.fc17.x86_64

How reproducible:
On every (re)boot.

Additional info:
'cat' above was needed to remove a garbage produced by 'systemctl --failed' which looks like that:

systemd-r...t-api-vfs.service loaded ESC[1;31mfailed failedESC[0m     Remount API VFS

Dumping into 'less' random escape codes is not a very good practice.

Comment 1 Kay Sievers 2012-02-26 13:34:08 UTC
Please attach /etc/fstab.

Comment 2 Michal Jaegermann 2012-02-26 21:51:51 UTC
Created attachment 565926 [details]
fstab as used on my rawhide installation

> Please attach /etc/fstab.

Here we go.

After a boot, and with all these errors from systemd, one can find, among others, in /etc/mtab:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0

Comment 3 Kay Sievers 2012-02-27 00:31:57 UTC
/etc/mtab is required to be a symlink to /proc/mounts. A regular file is not
supported by systemd systems, and systemd logs an error if it finds one.

The former /etc/mtab file was replaced by a symlink by the util-linux package,
you might have replaced it with a file, but that is not expected to work
correctly.

Also, are you possibly running your own kernel? If yes, can you reproduce
the failure with the Fedora kernel?

Comment 4 Michal Jaegermann 2012-02-27 01:41:51 UTC
(In reply to comment #3)
> /etc/mtab is required to be a symlink to /proc/mounts. A regular file is not
> supported by systemd systems, and systemd logs an error if it finds one.

Sigh!
# ls -l /etc/mtab
lrwxrwxrwx 1 root root 12 Feb 25 11:58 /etc/mtab -> /proc/mounts
# rpm -qf /etc/mtab
util-linux-2.21-1.fc18.x86_64
Happy?

What this has to do with the issue on hand?

> Also, are you possibly running your own kernel?

What on earth gave you that impression?  All kernels are Fedora kernels.
If you think that this is important at this moment I am running
3.3.0-0.rc4.git3.1.fc18.x86_64.

Comment 5 Kay Sievers 2012-02-27 03:18:21 UTC
(In reply to comment #4)
> lrwxrwxrwx 1 root root 12 Feb 25 11:58 /etc/mtab -> /proc/mounts
>
> What this has to do with the issue on hand?

You mentioned mtab in the earlier comment; it's just a known issue to
confuse mount because we never clean it up on boot these days.

> > Also, are you possibly running your own kernel?
> 
> What on earth gave you that impression?

I just tried your fstab entries here and the remount service only fails when
I use a kernel without the proper devpts options enabled, so I asked. With the
fedora kernel I can not reproduce your problem here.

Comment 6 Michal Jaegermann 2012-02-27 05:45:39 UTC
(In reply to comment #5)

> I just tried your fstab entries here and the remount service only fails when
> I use a kernel without the proper devpts options enabled, so I asked. With the
> fedora kernel I can not reproduce your problem here.

I have only Fedora kernels on that rawhide installation and I am seeing that FAILED consistently. OTOH all these "can't find" end up actually mounted so they are found by something even if 'systemctl --failed' reports what it reports.
Any debugging hints?

Comment 7 Michal Schmidt 2012-02-27 14:07:57 UTC
systemd-remount-api-vfs just spawns commands like this:
/bin/mount /sys -o remount

The message "mount: can't find /sys in /etc/fstab" comes from /bin/mount itself.

It seems to be a regression in util-linux-2.21. Looks like it does not like the "none" in fstab (because it finishes successfully when I replace it with "gaga"). Reassigning.

Comment 8 Michal Jaegermann 2012-02-27 19:01:53 UTC
(In reply to comment #7)
 
> It seems to be a regression in util-linux-2.21. Looks like it does not like the
> "none" in fstab (because it finishes successfully when I replace it with
> "gaga"). Reassigning.

I was thinking for a moment about changing fstab so these "can't find" entries will look closer to mtab patterns.  Only that my fstab is a result of a series of updates going way back so the same likely will happen on production systems.
Also what I have now in fstab is apparently valid but systemd still ends up unhappy (apparently because mount was complaining).

Comment 9 Karel Zak 2012-02-27 23:04:19 UTC
Fixed by upstream commit 772cce373f693096c2ae6f156306acf9e20c09e8, Fedora package will be updated ASAP. Thanks.

Comment 10 Fedora Update System 2012-03-02 12:28:56 UTC
util-linux-2.21-2.fc17 has been submitted as an update for Fedora 17.
https://admin.fedoraproject.org/updates/util-linux-2.21-2.fc17

Comment 11 Fedora Update System 2012-03-05 21:00:26 UTC
Package util-linux-2.21-2.fc17:
* should fix your issue,
* was pushed to the Fedora 17 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing util-linux-2.21-2.fc17'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2012-2976/util-linux-2.21-2.fc17
then log in and leave karma (feedback).

Comment 12 Michal Jaegermann 2012-03-05 23:19:51 UTC
(In reply to comment #11)
> Package util-linux-2.21-2.fc17:
> * should fix your issue,
Indeed, it does that.  Corresponding rawhide packages are still missing.

Comment 13 Fedora Update System 2012-03-21 18:52:35 UTC
util-linux-2.21-2.fc17 has been pushed to the Fedora 17 stable repository.  If problems still persist, please make note of it in this bug report.