Created attachment 603065 [details] syslog from ypbind failure Description of problem: After booting, ypbind is not running. The log indicates the NIS server is not responding, but there is no issue with restarting ypbind after the system has booted. Version-Release number of selected component (if applicable): ypbind-1.35-5.fc16.x86_64 How reproducible: Every time Steps to Reproduce: 1. reboot system 2. ypbind has entered failed state 3. Actual results: ybind failed Expected results: ypbind running Additional info:
I have two NIS servers (master+slave) so the yp.conf says domain rlent.pnet broadcast
Thanks for reporting. Could you, please, attach more verbose syslog with "log_buf_len=1M systemd.log_level=debug systemd.log_target=kmsg" as boot parameters to see ordering of services and systemd events during boot?
Created attachment 603259 [details] More verbose boot log
Oh, there is still no systemd-related info. Can you attach output of dmesg, please?
Created attachment 603410 [details] dmesg output immediately after reboot rebooted and ran dmesg immediately upon login.
I don't see any problem in services order. But it is weird, that it is working at all after manual restart, which shouldn't actually. broadcast mode is simply broken in ypbind in F16, because of bug #781880 in libtirpc. Does it really work for you after manual restart with the same configuration (domain rlent.pnet broadcast)? Another idea I currently have is to increase NISTIMEOUT in /etc/sysconfig/ypbind and see if it helps.
I seem to be having the same basic issue, but on Fedora 17. I am also using ypbind-1.35-5.fc17.x86_64 My yp.conf has: domain lucasarts server deathstar So, not broadcast. Restarting ypbind after boot works every time. I have discovered that if I remove rghb and quiet from the bootup, ypbind will bind successfully. I think this is due to the fact that scrolling is really slow in that mode and ypbind has more time to bind. This was working correctly until ~1 month ago. Do you still have a copy of the previous version of ypbind that I could use to test?
Created attachment 605265 [details] dmesg output on failed boot Attaching the dmesg from Fedora 17 with additional kernel arguments requested above.
(In reply to comment #7) > I have discovered that if I remove rghb and quiet from the bootup, ypbind > will bind successfully. I think this is due to the fact that scrolling is > really slow in that mode and ypbind has more time to bind. What about extending NISTIMEOUT in /etc/sysconfig/ypbind? Does it help? > This was working correctly until ~1 month ago. Do you still have a copy of > the previous version of ypbind that I could use to test? Feel free to test previous builds at: http://koji.fedoraproject.org/koji/packageinfo?packageID=372
I spent about an hour working on this issue this morning. Here are my results. Setting NISTIMEOUT=60 in /etc/sysconfig/ypbind made no difference. Down-rev'ing ypbind to version ypbind-1.35-2.fc17.x86_64 resolves the issue. With -5, every boot failed to bind. With -2, every bind at boot ( out of 5 tests ) was successful.
Thank you for your time, Michael. There were basically two changes made during builds -2 and -5. The first was a small fix of kill command in case binding was not successful, which I suspect to caused this issue actually. The second one was a bit complicated and was connected with bug #829487. In case ypbind-1.35-3.fc17.x86_64 fails in your environment as well, it means it was introduced by the first change (i.e. ypbind is not ready in 60s so it is killed). You can also check this without reinstalling ypbind by commenting-out a kill command in /usr/lib/ypbind/ypbind-post-waitbind: diff --git a/ypbind-post-waitbind b/ypbind-post-waitbind index f0cdb7f..7799cf0 100755 --- a/ypbind-post-waitbind +++ b/ypbind-post-waitbind @@ -48,7 +48,7 @@ else "NIS server for domain `domainname` is not responding." logger -t ypbind \ "Killing ypbind with PID $MAINPID." - kill -SIGTERM $MAINPID || : + #kill -SIGTERM $MAINPID || : fi exit $retval It would mean that your network set-up is really slow and even NISTIMEOUT=60 is too short. Then you can try NISTIMEOUT=180 for example, which could help.
Here are my results from testing: -2 installed +kill commented: Boot up/bind is successful and fast. -5 installed +kill commented: Boot up/bind is successful, but VERY slow. It probably takes at least 2x the time to boot. The network isn't slow. This is a very small network with 1 master and 3 NIS slaves, gig network, i5/i7 computers. Binding with previous versions is instantaneous and binding on restart is also instantaneous. Michael
I'd like to add to the discussion... I've been having ypbind problems since I installed Fedora 16 (F16) over 6 months ago, and in fact, opened 815978 back in April. I have spent many hours trying to get ypbind to reliably start at boot. I should note that it ALWAYS starts after the machine has completed booting, usually within 1 second. I have *never* had a problem with ypbind after the box has booted, and it ALWAYS starts very quickly. All of my problems are at boot time. I have a network of 30 Fedora boxes, and 10 CentOS boxes. I have 2 YP servers, both running CentOS. The CentOS boxes, and my 2 Solaris boxes, never have ypbind issues (unless the network goes down). Until yesterday, I always use broadcast when starting ypbind, and have never had a /etc/sysconfig/ypbind file (I see no references to it in the man pages.). ypbind fails on most of my boxes (but not all) at boot time, and even on the boxes it does succeed on, it's not reliable. I'll add attachments to this ticket with excerpts of a couple of my logs. In summary, I think that the problems arise from a combination of NetworkManager taking a long time to start up, broadcast not working at boot time (but it seems to work later) and a timer expiring in ypbind. After reading this ticket, I was able to get ypbind to start reliably (sample size is 5) by doing 2 things (both are needed): 1) Don't use broadcast in /etc/yp.conf, and instead specify the FQDN of one of my YP servers 2) Create a /etc/sysconfig/ypbind file and put "NISTIMEOUT=120" in it. (Note, 90 is not large enough for my network) I'd like for broadcast to work, since I have 2 YP servers on my network, and I want the box to find the one that's up. I'd also like for the /etc/sysconfig/ypbind file to be documented, if indeed, that's a supported way to specify options to ypbind. What I find is that it takes something like 88s to 90s for ypbind to complete on many of my boxes. Often times, it takes over 50s for NetworkManager to successfully activate. I have seen ypbind killing itself after 50s or so. I am running: ypbind.x86_64 3:1.35-5.fc16 @updates
Forgot to ask... After looking at my attachments, do you think a bug should be opened up against NetworkManager?? I am running: NetworkManager.x86_64 1:0.9.4-6.git20120521.fc16 @updates
Created attachment 606358 [details] Log of ypbind failing at boot using broadcast Excerpt of /var/log/messages: ypbind using broadcast with no /etc/sysconfig/ypbind file ypbind kills itself after 51s.
Created attachment 606359 [details] Using broadcast with 90s timeout specified in /etc/sysconfig/ypbind excerpt from /var/log/messages ypbind dies (times out) after 90s (90s was specified in /etc/sysconfig/ypbind) using broadcast (in /etc/yp.conf). NetworkManager fails activation.
I too had no problems with ypbind until I upgraded to Fedora 16 several months ago. Previously, this host had been running Fedora 14. I've tried setting NISTIMEOUT=180 with no effect. ypbind always starts immediately without any issues *after* boot. My network is small, only three hosts currently use NIS, although more use NFS. THe network is gigabit ethernet, so I don't think network speed is an issue. I have some vague recollection of disabling NetworkManager in Fedora 14. Unfortunately, that seems nearly impossible to do with Fedora 16 since several services seems to not recognize that the network is available unless it is running.
Created attachment 606394 [details] Attached are the output of ypbind-post-waitbind from -2 and -5 at boot Log contains a working boot from -2 and a failed boot from -5
I just tried adding a sleep 5 into ypbind-pre-setdomain just to see if this was a timing issue. With it, ypbind is successful at boot up even with version -5
Guys, thank you all for your responses. (In reply to comment #13) > I'd like for broadcast to work, since I have 2 YP servers on my network, and > I want the box to find the one that's up. Let's hope bug #781880 will be solved soon, it should fix broadcast issue. Until then you can downgrade to: libtirpc-0.2.2-0.fc16.x86_64 rpcbind-0.2.0-11.fc16.x86_64 > I'd also like for the > /etc/sysconfig/ypbind file to be documented, if indeed, that's a supported > way to specify options to ypbind. That will be added. > What I find is that it takes something like 88s to 90s for ypbind to > complete on many of my boxes. Often times, it takes over 50s for > NetworkManager to successfully activate. I have seen ypbind killing itself > after 50s or so. 45s is default timeout, so it would make sense. (In reply to comment #14) > Forgot to ask... After looking at my attachments, do you think a bug should > be opened up against NetworkManager?? As I understand it, network set up took much less time in previous versions, so it sounds like a regression for me and it'd make sense to report it against NetworkManager. (In reply to comment #16) > NetworkManager fails activation. It really seems like NetworkManager has serious problems. (In reply to comment #17) > I too had no problems with ypbind until I upgraded to Fedora 16 several > months ago. Previously, this host had been running Fedora 14. I've tried > setting NISTIMEOUT=180 with no effect. ypbind always starts immediately > without any issues *after* boot. > > My network is small, only three hosts currently use NIS, although more use > NFS. THe network is gigabit ethernet, so I don't think network speed is an > issue. Now I realized, that systemd timeout cuts off ypbind after 90s, which is default value. If NetworkManager is really buggy and network set up (I mean time to preparing network to use, not network bandwidth) is slower than 90s, it could be the problem. (In reply to comment #19) > I just tried adding a sleep 5 into ypbind-pre-setdomain just to see if this > was a timing issue. With it, ypbind is successful at boot up even with > version -5 This is a real mystery for me, I don't have a clue why this should help. Anyway, do you use localhost (any host from 127.0.0.0/8) as a NIS server?
(In reply to comment #20) > As I understand it, network set up took much less time in previous versions, > so it sounds like a regression for me and it'd make sense to report it > against NetworkManager. Or if the real problem is in slow dhcp, then reporting against that component would be better.
I've realized there is a better way to tell systemd to start ypbind after network is really up, which is to use NetworkManager-wait-online.service. So if you can test it in your scenario, please, add a line: After=NetworkManager-wait-online.service ... into your ypbind.service unit file and enable that service: # systemctl enable NetworkManager-wait-online.service
(In reply to comment #19) > I just tried adding a sleep 5 into ypbind-pre-setdomain just to see if this > was a timing issue. With it, ypbind is successful at boot up even with > version -5 Cau you provide dmesg output of that successful boot up to compare it with the one in comment #8, please?
Created attachment 606641 [details] dmesg output on successful bootup (with sleep) dmesg output on successful bootup (with sleep)
Honza, when adding the After line above, should it replace the entire existing line, or just get appeneded to the exiting entries for After= ? Moving the sleep to 1 works as well, but 0 fails. My ypbind.conf is attached above. As you can see, it is not using broadcast and it only binding to one server.
(In reply to comment #25) > Honza, when adding the After line above, should it replace the entire > existing line, or just get appeneded to the exiting entries for After= ? You need to keep the other services as well, so you can have either: After=syslog.target network.target rpcbind.service ypserv.service NetworkManager-wait-online.service or After=syslog.target network.target rpcbind.service ypserv.service After=NetworkManager-wait-online.service ... both has the same meaning. > Moving the sleep to 1 works as well, but 0 fails. Now I'd suspect rpcbind, that it states it's ready too early. There is a bug against rpcbind, that adds rpcbind.socket but doesn't support socket activation (bug #747363). However, I can't say it is related to this issue. We need some more info about ypbind, that behaves somehow differently if no waiting is performed. You can try to change an argument in ypbind.service to start in debug mode and provide ypbind related messages logged in /var/log/messages after then: -ExecStart=/usr/sbin/ypbind -n $OTHER_YPBIND_OPTS +ExecStart=/usr/sbin/ypbind -d $OTHER_YPBIND_OPTS Comparing such messages from successful and unsuccessful boot can say even more.
Created attachment 607346 [details] ypbind -d output Attached is the output from ypbind in debug mode. I noticed in the first failure that it complains about host name resolution. I changed yp.conf to include only the ip address and now it binds successfully at bootup, but it takes 4 seconds which seems long. Still seems like not everything is fully ready when ypbind is started
I have attached the ypbind -d output above. Please note that it is complaining about a DNS lookup. Switching to IP seems to work. Enabling NetworkManager-wait-online.service also seems to resolve the issue for me. Could someone else test the networkManager-wait-online.service and see if it helps?
I upgrade to F17 yesterday then downgraded ypbind to the -2 package. After seeing that last request to try NetworkManager-wait-online.service with the -5, I upgraded. I can confirm it works for me; ypbind is running once I get a GUI or console to log into.
Honza et al: Sorry for dropping out of sight for a few weeks -- extremely busy on other work... I finally had to make revisiting this issue a priority, since it was causing enough other grief... The bad news: while my comments dated 2012-08-22 13:36:09 EDT do help the situation, but unfortunately, do not completely fix the issue. The good news is that Honza's comments on 2012-08-23 06:13:31 EDT do seem to completely fix the problem (there's still an issue of ypbind not starting at boot time using broadcast mode, but I think that's documented in another ticket). I tried rebooting my system 10X (but only on my box, not different servers), and the following seems to work! I did the following: 1) Edited /lib/systemd/system/ypbind.service and modified the existing "After" line to look like (added NetworkManager-wait-online.service): After=syslog.target network.target rpcbind.service ypserv.service NetworkManager-wait-online.service (BTW, this just seems to make sense that the Network needs to be up before ypbind has a prayer of starting up successfully. I would strongly recommend that this change be added to the standard Fedora Distribution.) (If you add this to the After line, how do you ensure that all systems have NetworkManager-wait-online.service enabled?? Something to think about...) 2) As Honza indicated, you need to (as root): systemctl enable NetworkManager-wait-online.service 3) To get rid of my previous attempt to fix this problem, which didn't always work, I: rm -f /etc/sysconfig/ypbind 4) There remains one issue of /sbin/dhclient-script overwriting /etc/yp.conf at boot time. dhclient-script kept overwriting my yp.conf file to use broadcast mode, which we know has problems. To fix this issue, I added the following line to the appropriate subnet in my DHCP server's /etc/dhcpd.conf file: option nis-servers 192.168.1.XX, 192.168.1.YY; (you'll need to provide the correct values for XX and YY for your network) This change causes dhclient-script to put the following in your yp.conf file: domain my.domain server 192.168.1.XX domain my.domain server 192.168.1.YY which will fix the problem of trying to use broadcast mode and failing... BTW, you don't want to use a FQDN in yp.conf, since sometimes I saw DNS lookup failures (NetworkManager hadn't finished starting up (see below)). I intend to open a new ticket on NetworkManager. I always get the following when I boot: Failed to start Network Manager Wait Online However, it takes long enough for NetworkManager-wait-online.service to fail that it gives ypbind enough wait time so that it succeeds. Now, I see ypbind taking 0 seconds to start up at boot, when before, it was taking around 90s (and failing)... From my /var/log/messages (SUCCESS!!): Sep 19 17:14:03 myserver ypbind-pre-setdomain[946]: Setting NIS domain: 'my.domain' (environment variable) Sep 19 17:14:03 myserver ypbind: Binding NIS service Sep 19 17:14:06 myserver ypbind: Binding NIS service Sep 19 17:14:06 myserver ypbind: Binding took 0 seconds Sep 19 17:14:06 myserver ypbind: NIS domain: my.domain, NIS server: mynisserver FYI, I am now running: ypbind.x86_64 3:1.35-5.fc16 @updates BTW, on one of my test runs, I saw the following strange behavior (from boot.log): Starting NIS/YP (Network Information Service) Clients to NIS Domain Binder... Stopping NIS/YP (Network Information Service) Clients to NIS Domain Binder... Starting NIS/YP (Network Information Service) Clients to NIS Domain Binder... Started NIS/YP (Network Information Service) Clients to NIS Domain Binder Why would ypbind be stopped, then restarted?? Strange...
(In reply to comment #30) > (BTW, this just seems to make sense that the Network needs to be up before > ypbind has a prayer of starting up successfully. I would strongly recommend > that this change be added to the standard Fedora Distribution.) I've added that ordering into the last update, which is currently in testing repo: https://admin.fedoraproject.org/updates/FEDORA-2012-14750/ypbind-1.36-6.fc18 > (If you add this to the After line, how do you ensure that all systems have > NetworkManager-wait-online.service enabled?? Something to think about...) This service is intended to be turned on only in case someone has problems during start, but it shouldn't be needed in a general case. However, I've seen some intentions to have this service on by default in F18, so maybe it will actually happen. Until then you need to do, what you did: systemctl enable NetworkManager-wait-online.service > BTW, on one of my test runs, I saw the following strange behavior (from > boot.log): > > Starting NIS/YP (Network Information Service) Clients to NIS Domain Binder... > Stopping NIS/YP (Network Information Service) Clients to NIS Domain Binder... > Starting NIS/YP (Network Information Service) Clients to NIS Domain Binder... > Started NIS/YP (Network Information Service) Clients to NIS Domain Binder > > Why would ypbind be stopped, then restarted?? Strange... This is actually no mystery. DHCP client runs some scripts located at /etc/dhcp/dhclient.d/* after DHCP gets ready. There is nis.sh, that actually backups, generate or restores /etc/yp.conf, which you mentioned above. After changing /etc/yp.conf ypbind has to be restarted to load new configuration, which is usually done after DHCP resolution obviously. Anyway, from the last comments and with NetworkManager-wait-online.service included in the default ypbind.service file I have a feeling this bug should be solved. All user should do after updating to the new version is the following: systemctl enable NetworkManager-wait-online.service Updates for F16 and F17 will follow.
ypbind-1.36-6.fc17 has been submitted as an update for Fedora 17. https://admin.fedoraproject.org/updates/ypbind-1.36-6.fc17
ypbind-1.36-6.fc16 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/ypbind-1.36-6.fc16
Honza, many thanks!
Package ypbind-1.36-6.fc16: * should fix your issue, * was pushed to the Fedora 16 testing repository, * should be available at your local mirror within two days. Update it with: # su -c 'yum update --enablerepo=updates-testing ypbind-1.36-6.fc16' as soon as you are able to. Please go to the following url: https://admin.fedoraproject.org/updates/FEDORA-2012-14965/ypbind-1.36-6.fc16 then log in and leave karma (feedback).
The issues to strike again on Fedora 20, at least with certain hardware combination. It has been intermittent, but aggravating lately up to a total denial of service. I followed Ray Mikkelson's advice (comment #30) which seems to solve the issue - the target system is still under observation, but has behaved well for at least some 15 reboots. Before it used to hang 3 out of 5 times...
(In reply to David Ramirez from comment #36) > The issues to strike again on Fedora 20 The issue means that ypbind was slow on boot time (due the slow network set-up), so it failed to start, right? > I followed Ray Mikkelson's advice (comment #30) The one with enabling NetworkManager-wait-online.service?
(In reply to Honza Horak from comment #37) > (In reply to David Ramirez from comment #36) > > The issues to strike again on Fedora 20 > > The issue means that ypbind was slow on boot time (due the slow network > set-up), so it failed to start, right? > > > I followed Ray Mikkelson's advice (comment #30) > > The one with enabling NetworkManager-wait-online.service? Hi - thanks for following up - Yes, I meant enabling the NetworkManager-wait-online.service (steps 1 and 2 on comment #30 ) What can influence ybind to be slow here? I have some other 70 clients to the NIS server working well under Fedora 17 or CentOS, so I don't think it is a slow server issue; may be a particular condition on the target hardware (?). My problem system has behaved well so far after that fix...
(In reply to David Ramirez from comment #38) > What can influence ybind to be slow here? This is not that ypbind would be slow, but getting network un and working is probably slow. ypbind just waits for network to be properly working and doesn't start until then. That is actually desired behaviour, since if ypbind would wait for network in the background, people wouldn't be able to log in using NIS account. So the question should be, what can influence the network set up to be slow? > I have some other 70 clients to the NIS server working well under Fedora 17 > or CentOS, so I don't think it is a slow server issue; may be a particular > condition on the target hardware (?). Yeah, may be. > My problem system has behaved well so far after that fix... Great, this is pretty much everything we can do about it, AFAICT.