Bug 1133579
| Summary: | rhel-osp-installer: foreman-discovery-image-7.0 dhclient crashes after some time (ISO) | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Omri Hochman <ohochman> |
| Component: | rhel-osp-installer | Assignee: | Lukas Zapletal <lzap> |
| Status: | CLOSED ERRATA | QA Contact: | Omri Hochman <ohochman> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 5.0 (RHEL 6) | CC: | aberezin, dnavale, lzap, mburns, ohochman, rhos-maint, sclewis, yeylon |
| Target Milestone: | z1 | ||
| Target Release: | Installer | ||
| Hardware: | x86_64 | ||
| OS: | All | ||
| Whiteboard: | |||
| Fixed In Version: | foreman-discovery-image-7.0-20140905.0.2.el7sat | Doc Type: | Bug Fix |
| Doc Text: |
In some rare cases, foreman discovery image uploaded facts to the Foreman application too early when the network was not fully initialized. Due to this, discovered hosts had no network information, therefore, it was not possible to provision them.
With this update, the image waits 30 seconds ensuring the network is initialized, before it uploads its first facts to the server.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2014-10-02 12:56:31 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Omri Hochman
2014-08-25 13:45:32 UTC
Omri, can you please check if you are able to reboot the 2nd/3rd hosts using this API command: curl -3 -H "Accept:application/json" -H "Content-Length:0" -k -X PUT http://DISCOVERED_HOST_IP:8443/bmc/ignored/chassis/power/cycle Also what do you see on terminals? Were those hosts discovered? Any kind of error? Also if you provide rootpw kernel command line option, you should be able to log on. If you enconter this behavior, can you please check if "foreman-proxy" service is running and if not or investigate syslog. Thanks. with curl: curl: (7) couldn't connect to host It seems that networking dies on the discovered host. pings to the host also don't work after some time failing with "Destination Host Unreachable" Having hard time to reproduce. Can you please:
- add this to the kernel command line (TFTP root pxelinux.cfg/default):
rootpw=RHhwCLrQXB8zE
- then boot a node and switch over to tty2 and login as root/redhat (you will be prompted to change your password)
- once the network goes down, are you able to tell something from the syslog?
This looks like some missing dependency (so library) or something. I am trying to reproduce with dhclient on the foreground to see the error message. All I know is it returns -1. The "find-missing-libs" script does not show anything particular. It looks like dhclient should have all it's dependencies installed. A couple things that were mentioned here: This has been noticed by a few people on normal RHEL 7 hosts. It seems like it might be something to do with NetworkManager. Lukas, how hard would it be to switch to using network instead? One other possible solution is to drastically increase the dhcp lease time (on the server side). That would be a temporary stopgap imo. Thanks. There is a regression (reported in Fedora) in NetworkManager - it no longer restarts failed dhclients. When it dies it dies. https://bugzilla.redhat.com/show_bug.cgi?id=1022954 Obviously we can't wait until this is resolved upstream-downstream. I am not sure if we start using sysvinit script (network.service) it will help - the faulty dhclient is still there and when it dies - there is nothing to restart it. I like the lease time workaround but this is rather workaround for QA phase than for our customers. I need to know what is wrong but I wonder if this is not a generic issue (of dhclient) which makes this really challenging. Other workaround might be to restart dhclient regularly but NetworkManager spawns it with it's own configuration files which makes this hard. Maybe a combintion - try to get rid of NetworkManager and then if it does not help, I can create some daemon watchdog. Or if you have other ideas... I am all open. If the bug is really NM failing, then switching to the old network service might just solve it. Yes, the lease time issue is a temporary solution I don't have any other solutions at the moment. It seems to me like we could just write ifcfg files for each nic (you do this anyway iirc), with the NM_CONTROLLED=no and disable NM and enable network and see if that helps us. Upstream changes: https://github.com/theforeman/ovirt-node-plugin-foreman/pull/9 Unable to reproduce : foreman-discovery-image-7.0-20140905.0.2.el7sat.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHBA-2014-1350.html |