Bug 1158773
Summary: | openshift-routing-daemon always return success even it is not started. | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | Johnny Liu <jialiu> |
Component: | Node | Assignee: | chris alfonso <calfonso> |
Status: | CLOSED ERRATA | QA Contact: | libra bugs <libra-bugs> |
Severity: | medium | Docs Contact: | |
Priority: | high | ||
Version: | 2.2.0 | CC: | adellape, bleanhar, hbrock, jokerman, libra-onpremise-devel, mmccomas |
Target Milestone: | --- | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | rubygem-openshift-origin-routing-daemon-0.17.1.7-1 | Doc Type: | Bug Fix |
Doc Text: |
Previously, starting the openshift-routing-daemon service always returned successfully, which could be confusing to administrators under certain conditions. For example, if the routing daemon was configured to use nginx but nginx was not yet installed, the openshift-routing-daemon service would still start successfully, then report "no instances running" when checking the service's status. This was due to a bug in the routing daemon in which the exit code was not reported correctly. This bug fix updates the routing daemon to handle several edge cases, and the service now reports exit codes correctly. After applying this update the openshift-routing-daemon service must be restarted.
|
Story Points: | --- |
Clone Of: | Environment: | ||
Last Closed: | 2014-12-10 13:24:46 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Johnny Liu
2014-10-30 08:23:11 UTC
I've opened a PR upstream, which upon merge, I'll pull into OSE 2.2.z https://github.com/openshift/origin-server/pull/5923 This has been merged, and built in brew. The next puddle build should contain rubygem-openshift-origin-routing-daemon-0.17.1.5-1.el6op. I'll update this with the puddle link once it exits. Re-test this bug with rubygem-openshift-origin-routing-daemon-0.17.1.6-1.el6op.noarch, FAIL. 1. Make sure there is no /opt/rh/nginx14/root/etc/nginx/conf.d dir existing. 2. # /etc/init.d/openshift-routing-daemon start; echo $?; /etc/init.d/openshift-routing-daemon status Starting openshift-routing-daemon 0 openshift-routing-daemon is not running No "FAIL" is shown, and the exist value is still "0". ==> /var/log/openshift-routing-daemon.output <== /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.17.1.6/lib/openshift/routing/models/nginx.rb:46:in `open': No such file or directory - /opt/rh/nginx16/root/etc/nginx/conf.d (Errno::ENOENT) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.17.1.6/lib/openshift/routing/models/nginx.rb:46:in `entries' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.17.1.6/lib/openshift/routing/models/nginx.rb:46:in `get_pool_names' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.17.1.6/lib/openshift/routing/controllers/simple.rb:136:in `pools' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.17.1.6/lib/openshift/routing/daemon.rb:118:in `initialize' from /etc/init.d/openshift-routing-daemon:63:in `new' from /etc/init.d/openshift-routing-daemon:63:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `block in start_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call_as_daemon' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:219:in `start_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:255:in `start' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:188:in `block in run_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:187:in `run_proc' from /etc/init.d/openshift-routing-daemon:62:in `block in <main>' from /etc/init.d/openshift-routing-daemon:45:in `block (2 levels) in locked' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.28.3.0/lib/openshift-origin-common/utils/path_utils.rb:93:in `block in flock' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.28.3.0/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.28.3.0/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' from /etc/init.d/openshift-routing-daemon:44:in `block in locked' from /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' from /etc/init.d/openshift-routing-daemon:43:in `locked' from /etc/init.d/openshift-routing-daemon:61:in `<main>' I added a few more checks around start,stop,restart https://github.com/openshift/origin-server/pull/5970 I'll merge this to enterprise-server once it's merged upstream. This has been merged and built for enterprise-server. It will be in the next puddle build. Verified this bug with rubygem-openshift-origin-routing-daemon-0.20.2.1-1.el6op.noarch, and PASS. # /etc/init.d/openshift-routing-daemon start; echo $?; /etc/init.d/openshift-routing-daemon status; ps -ef|grep rout Starting openshift-routing-daemon pid-file for killed process 8796 found (/var/run/openshift-routing-daemon.pid), deleting. FAIL 1 openshift-routing-daemon is not running root 8821 26536 0 13:54 pts/1 00:00:00 grep rout Found openshift-routing-daemon can not be started with rubygem-openshift-origin-routing-daemon-0.20.2.1-1.el6op.noarch, so assign it back, this bug is blocking testing. # /etc/init.d/openshift-routing-daemon stop openshift-routing-daemon is not running # /etc/init.d/openshift-routing-daemon start Starting openshift-routing-daemon pid-file for killed process 8961 found (/var/run/openshift-routing-daemon.pid), deleting. FAIL # /etc/init.d/openshift-routing-daemon status openshift-routing-daemon is not running ==> /var/log/openshift-routing-daemon.log <== I, [2014-11-25T14:05:37.468178 #8982] INFO -- : *** below you find the most recent exception thrown, this will be likely (but not certainly) the exception that made the application exit abnormally *** E, [2014-11-25T14:05:37.468487 #8982] ERROR -- : exception reentered (fatal) I, [2014-11-25T14:05:37.468594 #8982] INFO -- : *** below you find all exception objects found in memory, some of them may have been thrown in your application, others may just be in memory because they are standard exceptions *** E, [2014-11-25T14:05:37.468729 #8982] ERROR -- : No such process (Errno::ESRCH) /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pid.rb:14:in `kill' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pid.rb:14:in `running?' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pidfile.rb:43:in `block in find_files' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pidfile.rb:41:in `delete_if' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pidfile.rb:41:in `find_files' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application_group.rb:63:in `find_applications' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application_group.rb:55:in `setup' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:65:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:188:in `block in run_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:187:in `run_proc' /etc/init.d/openshift-routing-daemon:81:in `block in <main>' /etc/init.d/openshift-routing-daemon:45:in `block (2 levels) in locked' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:94:in `block in flock' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' /etc/init.d/openshift-routing-daemon:44:in `block in locked' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /etc/init.d/openshift-routing-daemon:43:in `locked' /etc/init.d/openshift-routing-daemon:80:in `<main>' E, [2014-11-25T14:05:37.469092 #8982] ERROR -- : No such file or directory - /var/run/openshift-routing-daemon.pid (Errno::ENOENT) /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pidfile.rb:100:in `delete' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/pidfile.rb:100:in `cleanup' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:189:in `block (2 levels) in start_proc' E, [2014-11-25T14:05:37.469197 #8982] ERROR -- : closed stream (IOError) /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `close' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' /etc/init.d/openshift-routing-daemon:44:in `block in locked' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /etc/init.d/openshift-routing-daemon:43:in `locked' /etc/init.d/openshift-routing-daemon:80:in `<main>' E, [2014-11-25T14:05:37.469515 #8982] ERROR -- : No such file or directory - /var/lock/openshift-routing-daemon.lock (Errno::ENOENT) /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1406:in `unlink' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1406:in `block in remove_file' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1411:in `platform_support' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:1405:in `remove_file' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:785:in `remove_file' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:563:in `block in rm' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:562:in `each' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:562:in `rm' /opt/rh/ruby193/root/usr/share/ruby/fileutils.rb:585:in `rm_f' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:96:in `ensure in block in flock' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:97:in `block in flock' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' /etc/init.d/openshift-routing-daemon:44:in `block in locked' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /etc/init.d/openshift-routing-daemon:43:in `locked' /etc/init.d/openshift-routing-daemon:80:in `<main>' E, [2014-11-25T14:05:37.469668 #8982] ERROR -- : undefined method `[]' for nil:NilClass (NoMethodError) /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:48:in `block (2 levels) in read_config' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:43:in `instance_eval' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:43:in `block in read_config' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:42:in `map' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:42:in `read_config' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:165:in `initialize' /etc/init.d/openshift-routing-daemon:82:in `new' /etc/init.d/openshift-routing-daemon:82:in `block (2 levels) in <main>' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `block in start_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call_as_daemon' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:219:in `start_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:255:in `start' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:188:in `block in run_proc' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions' /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:187:in `run_proc' /etc/init.d/openshift-routing-daemon:81:in `block in <main>' /etc/init.d/openshift-routing-daemon:45:in `block (2 levels) in locked' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:94:in `block in flock' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' /etc/init.d/openshift-routing-daemon:44:in `block in locked' /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' /etc/init.d/openshift-routing-daemon:43:in `locked' /etc/init.d/openshift-routing-daemon:80:in `<main>' E, [2014-11-25T14:05:37.470512 #8982] ERROR -- : stream closed (IOError) E, [2014-11-25T14:05:37.470638 #8982] ERROR -- : failed to allocate memory (NoMemoryError) E, [2014-11-25T14:05:37.470760 #8982] ERROR -- : stack level too deep (SystemStackError) E, [2014-11-25T14:05:37.471001 #8982] ERROR -- : exception reentered (fatal) ==> /var/log/openshift-routing-daemon.output <== /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:48:in `block (2 levels) in read_config': undefined method `[]' for nil:NilClass (NoMethodError) from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:43:in `instance_eval' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:43:in `block in read_config' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:42:in `map' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:42:in `read_config' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-routing-daemon-0.20.2.1/lib/openshift/routing/daemon.rb:165:in `initialize' from /etc/init.d/openshift-routing-daemon:82:in `new' from /etc/init.d/openshift-routing-daemon:82:in `block (2 levels) in <main>' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:215:in `block in start_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/daemonize.rb:192:in `call_as_daemon' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:219:in `start_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/application.rb:255:in `start' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/controller.rb:69:in `run' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:188:in `block in run_proc' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `call' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons/cmdline.rb:105:in `catch_exceptions' from /opt/rh/ruby193/root/usr/share/gems/gems/daemons-1.0.10/lib/daemons.rb:187:in `run_proc' from /etc/init.d/openshift-routing-daemon:81:in `block in <main>' from /etc/init.d/openshift-routing-daemon:45:in `block (2 levels) in locked' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:94:in `block in flock' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `open' from /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-common-1.29.1.1/lib/openshift-origin-common/utils/path_utils.rb:88:in `flock' from /etc/init.d/openshift-routing-daemon:44:in `block in locked' from /opt/rh/ruby193/root/usr/share/ruby/timeout.rb:69:in `timeout' from /etc/init.d/openshift-routing-daemon:43:in `locked' from /etc/init.d/openshift-routing-daemon:80:in `<main>' After dig into this bug, the issue found in comment 10 is caused by some new code, so it is a new bug, I opened a new bug - BZ#1167625 for tracking it, and verified this bug according to comment 9. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2014-1979.html |