Description of problem: nfs file systems mounted in my fstab are mounted before the network is started. It delays my boot and makes the file systems unmounted. Version-Release number of selected component (if applicable): $ rpm -q NetworkManager NetworkManager-0.8.997-6.git20110328.fc15.x86_64 $ rpm -q systemd systemd-20-2.fc15.x86_64 Expected results: Network has to be available prior the NFS are mounted.
Created attachment 488663 [details] Trimmed down /var/log/messages
Created attachment 488664 [details] /etc/fstab
Please use the parameter "systemd.log_level=debug" when reporting systemd bugs. This issue may be the same as bug 691075. Does the same workaround help? But before you try that, could you retest with systemd-21-2.fc15 from Koji?
Created attachment 488724 [details] systemd.debug_level=debug I saw #691075 before, but I was not sure, and still not sure if it is the same or not. I have tested the systemd-21-2.fc15 from Koji but it was complete failure. It never reached X and moreover it did not stored anything at all in my /var/log/messages. Not sure if there is some pre-requirement? I have attached the messages log with detailed loging, but just with the systemd-20 output. HTH
(In reply to comment #4) > I have tested the systemd-21-2.fc15 from Koji but it was complete > failure. It never reached X and moreover it did not stored anything at all in > my /var/log/messages I smell a SELinux problem. Try with "enforcing=0".
Created attachment 489021 [details] enforcing=0 systemd.log_level=debug Yes, "enforcing=0" made the trick with new systemd. However, I don't want to use my system with disabled SELinux, so I hope you will make sure that this is not an issue. Nevertheless, the update does not solve my issue. Please see attached log.
Try the workaround from bug 691075: systemctl enable network.service
(In reply to comment #7) > Try the workaround from bug 691075: > systemctl enable network.service This is even worse. 1) System didn't want to restart. 2) System didn't want to start. No log what so ever. I have only log, probably when I enabled the network service: Mar 31 15:42:48 dhcp-25-40 systemd[1]: Reloading. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit hdapsd entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-mirror.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-archive.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-scratch.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-globalsync.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-engarchive2.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-redhat.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Reloading. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit hdapsd entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-mirror.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-archive.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-scratch.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-globalsync.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-engarchive2.mount entered failed state. Mar 31 15:42:48 dhcp-25-40 systemd[1]: Unit mnt-redhat.mount entered failed state.
(In reply to comment #8) > (In reply to comment #7) > > Try the workaround from bug 691075: > > systemctl enable network.service > > This is even worse. > 1) System didn't want to restart. > 2) System didn't want to start. No log what so ever. Sorry, but this description of the situation is too vague. Have you tried waiting at least 3 minutes for the boot? That's the default timeout for most services. Aren't there any hints if you boot with "systemd.log_level=debug systemd.log_target=kmsg" and without "rhgb quiet"? (and of course, keep using "enforcing=0" to avoid the known bug)
Well, I cannot say if I waited 3 minutes or not, because you are telling me that I should wait after the test, not before. I can't know every timeout. But what I know is that I had for each mount red "failed" on my screen, so I suppose that it failed and waiting for 3 minutes doesn't make sense. But I will try again and let you know.
Created attachment 489335 [details] network.service enabled 1 Failures when network.service is enabled, with "rhgb quiet".
Created attachment 489336 [details] network.service enabled 2 Failures when network.service is enabled, with "rhgb quiet" - after some time
Created attachment 489337 [details] network.service enabled 3 Failures when network.service is enabled, with removed "rhgb quiet".
Created attachment 489338 [details] network.service enabled 4 Failures when network.service is enabled, with removed "rhgb quiet" - after some time.
What I have noticed, that my root file system is, for some reason, mounted RO, when network.service is enabled. Therefore I have no log, because it can't be written.
Thanks for the screenshots. The symptom with ro filesystem is the same as in bug 692573. Did you by any change disable SELinux in /etc/selinux/config (as opposed to passing "selinux=0" or "enforcing=0" to the kernel)? If it's not that, then try this: Since gettys are being started according to one of the screenshots, could you login, do a "mount -o remount,rw /" and then save the dmesg (provided that you ensure systemd is logging there using "systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M"). Another useful piece of information would be "systemctl dump".
Another idea: Try commenting out /home from /etc/fstab temporarily. The fact that it is an encrypted volume may be significant.
(In reply to comment #16) > Thanks for the screenshots. The symptom with ro filesystem is the same as in > bug 692573. Did you by any change disable SELinux in /etc/selinux/config (as > opposed to passing "selinux=0" or "enforcing=0" to the kernel)? I did not touch that configuration file. It contains: SELINUX=enforcing SELINUXTYPE=targeted But this problem appear just after enabling network.service and enabling network.service is just workaround as far as I understand it, not solution. So I am wondering what is the point? My original problem was little bit different ... > If it's not that, then try this: > Since gettys are being started according to one of the screenshots, could you > login, do a "mount -o remount,rw /" and then save the dmesg (provided that you > ensure systemd is logging there using "systemd.log_level=debug > systemd.log_target=kmsg log_buf_len=1M"). Another useful piece of information > would be "systemctl dump". Save dmesg? How should I do that?
Created attachment 489362 [details] systemctl dump Dump in default state, i.e. network.service disabled and systemd-20-1.fc15.x86_64
(In reply to comment #18) > But this problem appear just after enabling network.service and enabling > network.service is just workaround as far as I understand it, not solution. Yeah, but it is supposed to work. What you're seeing after enabling network.service seems like even a more serious problem than the original one. > Save dmesg? How should I do that? dmesg > dmesg.txt
Created attachment 489372 [details] a modified NetworkManager.service unit file
Created attachment 489373 [details] NetworkManager-wait-online.service unit file
So to go back to the original problem... I have attached two unit files. Used together they should make network.target fulfilled only after a connection is made, thus making sure the NFS mounts can immediately succeed. So you won't need network.service then. Put the two files in /etc/systemd/system/. I would still like to find out what is so horribly wrong when network.service is enabled on your system. I'll look at the dump you attached when I have more time.
I have recently updated to the latest systemd: $ rpm -q systemd systemd-22-1.fc15.x86_64 but it did not helped. So I had used the workaround you provided and that did the trick. So I hope you will release the fix soon. I am going to revert the changes and see what more I can investigate about the network.service.
Created attachment 489982 [details] network.service enabled debug
Created attachment 489983 [details] network.service enabled dump
So currently my system can boot when network.service is enabled. However, it will not help with my mounting issue. Please see logs attached.
Michal's units look good, however I'd like to keep them independent from each other. I have now made the appropriate change and sent this upstream. With this in place people who want to delay network.target after the links are up in order to NFS things after it can do so now by doing "systemctl enable NetworkManager-wait-online.service". This unit is disabled by default however, in order not to delay bootup based on whether a network is around or not.
Hmm, I decided not to make them independent after all so that people can rely on NetworkManager-wait-online.service and nothing else to get the network up.
It is still not working with latest update: $ rpm -q systemd systemd-24-1.fc15.x86_64
The unit files are in NetworkManager upstream. Vít, the network service is failing on your system (the script exits with code 1). Could you describe your network configuration? I see there are several ifcfg files mentioned in the log: ifcfg-wlan0 ifcfg-em1 ifcfg-VitaD-Link ifcfg-Auto_RH_WLAN_TRUST Which interfaces do you expect to come up on boot? And which ones actually are up after boot? And through which interface are the NFS mounts supposed to be reached? Can you attach the ifcfg-* files?
Created attachment 490810 [details] Network configuration scripts I am attaching my network configuration scripts. Only em1 should be up at boot. The wireless is disabled at boot and the other configuration scripts are either VPN or wireless profiles. Of course I would like to see the NFS mounted as soon as some network is available and the mounts are reachable, but I am not sure if that is feasible or if it is not beyond scope of this issue.
Created attachment 491113 [details] network.service unit for debugging I'd like to see what the network script complains about. Could you save the attached unit file as /etc/systemd/system/network.service, run "systemctl enable network.service" and reboot? It should fail exactly the same as before, but the stdout and stderr of the network script will be saved in syslog.
Vit, if you run "systemctl enable NetworkManager-wait-online.service" and reboot, do things work, then?
Created attachment 491798 [details] messages with network.service Michal, please see attached log. produced while I used your network.service files. The nfs was not mounted.
Created attachment 491801 [details] messages with bridge What is interesting, I had by a coincidence in my network settings added settings for KVM bridge. In this case, there was some 1 minute delay during boot, but the NFS was *mounted*! What is even more interesting for me is that the log stated something like: Apr 13 16:33:47 dhcp-25-40 systemd[1]: mnt-mirror.mount mount process exited, code=exited status=32 Apr 13 16:33:47 dhcp-25-40 systemd[1]: Unit mnt-mirror.mount entered failed state.
(In reply to comment #34) > Vit, if you run "systemctl enable NetworkManager-wait-online.service" and > reboot, do things work, then? $ sudo systemctl enable NetworkManager-wait-online.service [sudo] password for vondruch: Couldn't find NetworkManager-wait-online.service. $ rpm -q systemd systemd-24-1.fc15.x86_64 $ rpm -q NetworkManager NetworkManager-0.8.998-2.git20110406.fc15.x86_64 Do I need some updated systemd?
No, just an updated NM.
NetworkManager-0.8.998-2.git20110406.fc15 is the latest version built so far, but it does not include NetworkManager-wait-online.service yet.
(In reply to comment #35) > Created attachment 491798 [details] > messages with network.service > > Michal, please see attached log. produced while I used your network.service > files. The nfs was not mounted. So here we can see what the classic network service complains about (half Czech, half English): Apr 13 16:33:47 dhcp-25-40 network[1125]: Aktivování rozhraní loopback: [ OK ] Apr 13 16:33:47 dhcp-25-40 NetworkManager[1011]: <warn> connection /org/freedesktop/NetworkManager/Settings/1 failed to activate: (2) Device not managed by NetworkManager or unavailable Apr 13 16:33:47 dhcp-25-40 network[1125]: Aktivuji rozhraní Auto_RH_WLAN_TRUST: Chyba: Selhala aktivace připojení: Device not managed by NetworkManager or unavailable Apr 13 16:33:47 dhcp-25-40 network[1125]: [SELHALO] Apr 13 16:33:47 dhcp-25-40 NetworkManager[1011]: <warn> connection /org/freedesktop/NetworkManager/Settings/2 failed to activate: (2) Device not managed by NetworkManager or unavailable Apr 13 16:33:47 dhcp-25-40 network[1125]: Aktivuji rozhraní VitaD-Link: Chyba: Selhala aktivace připojení: Device not managed by NetworkManager or unavailable Apr 13 16:33:47 dhcp-25-40 network[1125]: [SELHALO] Apr 13 16:33:47 dhcp-25-40 systemd[1]: network.service: main process exited, code=exited, status=1 Apr 13 16:33:47 dhcp-25-40 systemd[1]: Unit network.service entered failed state. Apr 13 16:33:47 dhcp-25-40 systemd[1]: mnt-mirror.mount mount process exited, code=exited status=32 (For the benefit of non-speakers of Czech: Aktivuji rozhraní = [I am] activating interface Chyba: Selhala aktivace připojení = Error: Connection activation failed) So it managed to activate only loopback. It tried and failed to connect using Auto_RH_WLAN_TRUST and VitaD-Link. It tried to activate these two connections because they have ONBOOT=yes in their config files. ifcfg-em1 also has ONBOOT=yes, but the network script did not try to activate it. I believe this is because BOOTPROTO=dhcp is missing. NetworkManager does not mind, but the classic network scripts do not know what to do about such an interface. Therefore network.service fails for you without managing to do anything useful. The finish of network.service unblocks the NFS mount services. And because no network interface is up, they must fail. You have two options: 1. add BOOTPROTO=dhcp to ifcfg-em1. This will make network.service bring the interface up. network.service will still show up as failed (because of the two failing wireless connections), but it will not finish before bringing em1 up and the NFS mounts will have a chance to succeed. 2. use NetworkManager-wait-online.service (when it's available in NM) to delay network.target and thus the NFS mounts to the moment NM succeeds in making a connection (via em1 in your case).
(In reply to comment #34) > Vit, if you run "systemctl enable NetworkManager-wait-online.service" and > reboot, do things work, then? Yes, it works. Will you make it default? I hope so. $ rpm -q systemd systemd-25-1.fc15.x86_64 $ rpm -q NetworkManager NetworkManager-0.8.998-3.git20110419.fc15.x86_64
No, we don't want to make this the default, as this slows down boot if somebody pulled the network plug.
Note that it wasn't the default in prior releases, FWIW.
Actually I don't know how did it worked in F14, but it worked. I am just asking the same functionality for F15, otherwise NFS mounts has no sense.
Experienced same problem and also same fix using the wait.online.service does the trick.
-- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Think of "systemctl enable NetworkManager-wait-online.service" as an equivalent of adding NETWORKWAIT=yes to /etc/sysconfig/network in F14. Is there anything left to solve in this BZ?
(In reply to comment #47) > Think of "systemctl enable NetworkManager-wait-online.service" as an equivalent > of adding NETWORKWAIT=yes to /etc/sysconfig/network in F14. > > Is there anything left to solve in this BZ? I did not need to set NETWORKWAIT=yes in F14. It Just Worked(tm) in F14. I believe the "netfs" SysVInit script was responsible (which has Bill's name in it!).
Thanks, that explains it. The netfs initscript has this: case "$1" in start) [ ! -f /var/lock/subsys/network ] && ! nm-online -x >/dev/null 2>&1 && exit 0 "nm-online -x" waits for up to 30 seconds for the completion of the connection if NetworkManager is running and trying to connect.
netfs has a NetworkManager dispatcher script that re-runs to mount the filesystem after NM gains an IP. In *theory*, this should all work. If netfs is run before NM is online, or NM starts connecting, it will exit. The NM-dispatcher script will re-run after a connection is established, and mount the mount. If netfs is run while NM is connecting, nm-online should wait, and the scrit will continue. If netfs is run after NM has connected, it should just work.
The dispatcher script does not work as expected here. It looks like $1 is empty. The script wants the interface name in that parameter. A bug in NM-dispatcher?
The dispatcher is broken: bug 703321
I was getting all my NFS mounts failing on a new fedora 15 install, and I found this bug in a search. I did the recommended "systemctl enable NetworkManager-wait-online.service" and rebooted, and the boot.log still shows all the NFS mounts failing, but I actually got 12 mounts (of the 15 I have specified). So it works better, but still doesn't work. The mounts that fail all look like this when I ask about them: [root@tomh ~]# systemctl status userland.mount userland.mount - /userland Loaded: loaded Active: failed since Thu, 26 May 2011 10:13:12 -0400; 6min ago Where: /userland What: userland:/userland Process: 862 ExecMount=/bin/mount /userland (code=exited, status=32) CGroup: name=systemd:/system/userland.mount What I really want to happen is for the mounts to simply be backgrounded, not for them to delay the boot. Maybe I should just disable the NFS mounting service and run a script in the background in rc.local to background each mount in a loop till it really gets mounted :-).
I have the same problem: the shared directories are not mounted via nfs4 during boot, and in my case, the network interfaces are NOT controlled by NetworkManager. I use nfs4 with kerberos authentication. In fedora 14 worked perfectly, after upgrading to fedora 15 the problems started. Putting "mount -a" in /etc/rc.local the shared directories are mounted correctly. Why??
Well, after a recent update to 2 F15 systems, this is now working correctly with just the following in my fstab... misty:/myhome /home/egreshko/misty nfs4 rw 0 0 Can anyone else on the CC: list verify?
I changed some kernel boot options at the same time I updated to a new kernel and I noticed the NFS error messages no longer appeared in the boot.log, but I wasn't sure if it was the boot option change or the new kernel. Maybe it was the kernel.
(In reply to comment #55) > Well, after a recent update to 2 F15 systems, this is now working correctly > with just the following in my fstab... > > > misty:/myhome /home/egreshko/misty nfs4 rw 0 0 > > Can anyone else on the CC: list verify? It looks like it was the NM dispatcher regression mentioned in comment # 52 which was fixed in NM
Now I am totally confused..... I had to re-install the F15 systems due to a disk issue. I can only test on one of the systems at this time....but it is back to not working. The only difference between the original system and the one now is 64 v.s. 32 bit.
To add to the above..... The one system that fails is on "real" hardware. I now have a second F15 system as a Vbox VM and it succeeds. Both are updated.... Both have NetworkManager-0.8.9997-2.git20110531.fc15.i686 installed.
(In reply to comment #57) > It looks like it was the NM dispatcher regression mentioned in comment # 52 > which was fixed in NM This didn't solve it for me. I've had the NM version installed since Sun Jun 05 and the problem keeps occurring. The problem is random, though. It's only happened twice in the last four system boots since my last yum update. Could there be some kind of race condition?
I've got this problem too, the solutions above don't seem to work, and two points strike me. - My system has been upgraded multiple times, but its origin pre-dates NetworkManager, and I don't think any of the network interfaces profiles made it into there. I'm still using /etc/sysconfig/network-scripts/ifcfg-*. They do have NM_CONTROLLED=yes in them, but there are no references to those interfaces under /etc/NetworkManager. Does this explain the problem? - My eth0 interface seems to be quite slow to start, and that might be another factor? [ 9.199206] r8169 0000:04:00.0: eth0: RTL8102e at 0xffffc90001224000, 6c:62:6d:9e:ae:c0, XID 04c00000 IRQ 43 [ 21.183268] r8169 0000:04:00.0: eth0: link down [ 21.183677] r8169 0000:04:00.0: eth0: link down [ 21.184877] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 22.801993] r8169 0000:04:00.0: eth0: link up [ 22.803024] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready I've "given up" and just stuck a 'mount -a' at the end of /etc/rc.local, which I'm hoping will work around this issue. K.
(In reply to comment #0) > Description of problem: > nfs file systems mounted in my fstab are mounted before the network is started. > It delays my boot and makes the file systems unmounted. > I've got this problem too, my solution is add comment in fstab(comment=systemd.automount). [root@test ~]# tail -n 2 /etc/fstab linux:/home/STUDENTS /home/STUDENTS nfs rsize=8192,wsize=8192,comment=systemd.automount 0 0 linux:/home/PREPODS /home/PREPODS nfs rsize=8192,wsize=8192,comment=systemd.automount 0 0
Yes, people keep saying that is a solution, but why would any sane system require me to add a comment when the dadgum type field is already "nfs", which pretty obviously indicates it needs the network. There are probably only about 87,631,233 lines of nfs mountpoints in fstabs around the world, so making everyone modify them all to provide information that is already possible to obtain from the existing line is more than a little silly.
AFAIK, isn't this all just a manifestation of https://bugzilla.redhat.com/show_bug.cgi?id=633774#c14? (Essentially, the dispatcher script that NM uses to restart netfs isn't getting run right.)
It can't all be NM related, because I have NM disabled, and network enabled and a static IP defined, yet I still have the problem.
Tom - does merely adding: # Required-Start: $network to the LSB section of netfs fix it?
I don't know what an "LSB section" is, but I put that comment right after the Provides: comment that was already in the netfs script and did not detect any change in the behavior of NFS mounts, still got errors in the boot.log, still worked OK when mounting in background from rc.local. I've actually decided I like the rc.local script better anyway. I'm thinking about changing all my machines to start a loop in the background trying to mount each NFS filesystem and change the fstab to say "noauto" on everything. All the files will eventually get mounted, yet nothing NFS related will delay the boot - the best of both worlds (unless there is an NFS system I actually need to access at boot time, of course). I really do get the impression that something in the boot believes the network is "up", even though it isn't really functional yet. I had to move my stunnel startup to rc.local as well rather than using the stunnel init.d script I've used for years. It would always believe it started OK, but the sockets it opened to talk to the predefined servers would never actually work. Doing a "service stunnel restart" in rc.local makes it work OK.
I'm having a similar problem mounting CIFS shares. I am also using a background script started from rc.local. I end up with my shares mounted by the time I can get to a command prompt to check them. I do have SELinux disabled, but I haven't attempted to determine if that has something to do with this. Certainly seems to be some bugs here.
*** Bug 716339 has been marked as a duplicate of this bug. ***
*** Bug 710771 has been marked as a duplicate of this bug. ***
This package has changed ownership in the Fedora Package Database. Reassigning to the new owner of this component.
Is this still a problem or can this bug be closed?
Works for me in F16.
This bug was still present for me in FC16 x86_64. It was a reinstall over an existing (fairly heavily modified) system from the netboot CD, so might not have been 100% "clean". However, the "comment=systemd.automount" alone didn't fix it. After doing that, I changed the last field of /etc/fstab for those filesystems from 2 to 0 (which should not have had any effect on the mounting order, surely?), and applied the aforementioned fix in the LSB section (*), and then the problem went away. *) "Comment 66 Bill Nottingham 2011-06-24 17:06:56 EDT". The LSB section resides between the lines: ### BEGIN INIT INFO ... and: ### END INIT INFO ... you can in principle add the suggested line anywhere between those lines: # Required-Start: $network What this does, I believe, is instructs the init system (whatever it's called nowadays) to order the init scripts differently to the esoteric symlink type method and that makes things work. See the 'redhat-lsb' RPM for more details.
this bug is so annoying... I'm on Fedora 16 64bit I this bug is still there even today! Here's what I in /etc/fstab did as a workaround: IP:/share /home/user/share nfs defaults,timeo=1 0 0 just add timeo=1 option, after GUI is loaded and I login share is there and mounted. This may not work for everybody but it does the trick for me (:
Nick, you have made sure you have the service NetworkManager-wait-online.service enabled, as described in the comments above, haven't you? Our F16 (64bit) systems now works just fine, and that was a vital part.
Hey Göran, I didn't tried this approach, I just got these two files https://bugzilla.redhat.com/attachment.cgi?id=489373&action=edit https://bugzilla.redhat.com/attachment.cgi?id=489372&action=edit and put them into /etc/systemd/system/ then systemctl enable NetworkManager-wait-online.service and got Warning: unit files do not carry install information. No operation executed. Am I doing something wrong? (; rpm -q NetworkManager NetworkManager-0.9.2-1.fc16.x86_64 rpm -q systemd systemd-37-13.fc16.x86_64 btw why these are not part of the base install if they are so important? Thanks in advance!
These service files ARE in current versions of NetworkManager. You should have a file /lib/systemd/system/NetworkManager-wait-online.service on your system already. It belongs to the NetworkManager package you have. So remove the things two attachments to this Bugzilla you downloaded again. Then after that do systemctl daemon-reload systemctl enable NetworkManager-wait-online.service (I'm not really sure if the "daemon-reload" is necessary, but it won't hurt. I just want to be sure systemd discovers you removed the things you put in /etc, and start using the default version in /lib again.)
Confirmed! Success from first time. I even disabled that service after that just to check if this is what does the trick and got stuck again for a minute on the same place as before during boot. So thats it, all works great now. 10x (;
Good to hear! :-)