Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1.install aeolus-all/aeolus-configure 2.[root@dell-per905-01 ~]# aeolus-check-services Checking mongod ... Success: mongod (pid 8107) is running... Checking iwhd ... Success: iwhd (pid 8299) is running... Checking postgresql ... Success: postmaster (pid 8024) is running... Checking httpd ... Success: httpd (pid 8139) is running... Checking qpidd ... Success: qpidd (pid 8597) is running... Checking deltacloud-core ... Success: deltacloud-core (pid 8522) is running... Checking libvirtd ... Success: libvirtd (pid 8200) is running... Checking aeolus-conductor ... Success: (pid 8414) is running... Checking conductor-dbomatic ... FAILURE: dbomatic dead but pid file exists Checking imagefactory ... Success: imagefactory (pid 8668) is running... [root@dell-per905-01 ~]# ps -auxf |grep dbomatic Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ root 9306 0.0 0.0 103228 844 pts/0 S+ 00:30 0:00 \_ grep dbomatic dbomatic.log: 011-12-18 23:52:05 INFO DBOmatic starting up 2011-12-18 23:52:09 ERROR /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/dependencies.rb:304:in `depend_on': No such file to load -- lib/image 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/activesupport-3.0.10/lib/active_support/dependencies.rb:216:in `require_dependency' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/engine.rb:138:in `eager_load!' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/engine.rb:137:in `each' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/engine.rb:137:in `eager_load!' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/engine.rb:135:in `each' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/engine.rb:135:in `eager_load!' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:108:in `eager_load!' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application/finisher.rb:41 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/initializable.rb:25:in `instance_exec' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/initializable.rb:25:in `run' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/initializable.rb:50:in `run_initializers' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/initializable.rb:49:in `each' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/initializable.rb:49:in `run_initializers' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:134:in `initialize!' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `send' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/gems/1.8/gems/railties-3.0.10/lib/rails/application.rb:77:in `method_missing' 2011-12-18 23:52:09 ERROR from /usr/share/aeolus-conductor/config/environment.rb:23 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require' 2011-12-18 23:52:09 ERROR from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require' 2011-12-18 23:52:09 ERROR from /usr/share/aeolus-conductor/dbomatic/dbomatic:274 Additional info: [root@dell-per905-01 ~]# rpm -qa | grep aeolus rubygem-aeolus-cli-0.2.0-3.el6.noarch rubygem-rack-mount-0.7.1-3.aeolus.el6.noarch rubygem-aeolus-image-0.2.0-1.el6.noarch rubygem-arel-2.0.10-0.aeolus.el6.noarch aeolus-conductor-0.8.0-0.20111216150144gitb062bc3.el6.noarch aeolus-configure-2.5.0-0.20111216121225git990f43d.el6.noarch aeolus-conductor-doc-0.8.0-0.20111216150144gitb062bc3.el6.noarch aeolus-all-0.8.0-0.20111216150144gitb062bc3.el6.noarch rubygem-ZenTest-4.3.3-2.aeolus.el6.noarch aeolus-conductor-daemons-0.8.0-0.20111216150144gitb062bc3.el6.noarch
I have successfully reproduced via jenkins test. See http://hudson.rhq.lab.eng.bos.redhat.com:8080/hudson/view/DEV-CloudForms/job/ImageFactory-KocaTesting2/100/console
I'm seeing this also. This appears to be a duplicate of a newer bug already filed (in POST) by Matt Wagner (bug#771505). I recommend just duplicating this bug against the newer one since a fix is already POSTED.
And not only conductor-dbomatic. Now I'm getting error for deltacloud-core. Maybe it is just me. See automatic test in hudson: http://hudson.rhq.lab.eng.bos.redhat.com:8080/hudson/view/DEV-CloudForms/job/ImageFactory-KocaTesting2/173/console. # aeolus-check-services |grep -1 -i fail Checking deltacloud-core ... FAILURE: deltacloud-core dead but subsys locked -- Checking conductor-dbomatic ... FAILURE: dbomatic is stopped # tail -30 /var/log/deltacloud-core/mock.log >> Writing PID to /var/run/deltacloud/deltacloud-core-mock.pid >> Changing process privilege to nobody:nobody thin server (localhost:3002) [deltacloud-mock][15357]: >> Thin web server (v1.3.1 codename Triple Espresso) thin server (localhost:3002) [deltacloud-mock][15357]: >> Debugging ON thin server (localhost:3002) [deltacloud-mock][15357]: >> Maximum connections set to 1024 thin server (localhost:3002) [deltacloud-mock][15357]: >> Listening on localhost:3002, CTRL+C to stop thin server (localhost:3002) [deltacloud-mock][15357]: >> Exiting! thin server (localhost:3002) [deltacloud-mock][15357]: /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:572:in `start_tcp_server' thin server (localhost:3002) [deltacloud-mock][15357]: : thin server (localhost:3002) [deltacloud-mock][15357]: no acceptor thin server (localhost:3002) [deltacloud-mock][15357]: ( thin server (localhost:3002) [deltacloud-mock][15357]: RuntimeError thin server (localhost:3002) [deltacloud-mock][15357]: ) thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:572:in `start_server' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/backends/tcp_server.rb:16:in `connect' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/backends/base.rb:53:in `start' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `call' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/backends/base.rb:61:in `start' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/server.rb:159:in `start' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/controllers/controller.rb:86:in `start' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/runner.rb:185:in `send' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/runner.rb:185:in `run_command' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/lib/ruby/gems/1.8/gems/thin-1.3.1/lib/thin/runner.rb:151:in `run!' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/share/deltacloud-core/bin/deltacloudd:226 thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/bin/deltacloudd:5:in `load' thin server (localhost:3002) [deltacloud-mock][15357]: from /usr/bin/deltacloudd:5
It might be more sane to keep this as one issue per bug. Since the conductor-dbomatic issue is already tracked (with fix POSTed) in bug#771505, I recommend repurposing this bug to address your deltacloud-core service problem. Please change the subject and component as needed.
Updating owner and assigning out.
This 'bug' appears only when there is another instance of Deltacloud API running on the same machine using the same port. In this case I think Deltacloud API should refuse to start or/and don't lock the subsys.
OK seems like I figured out what is wrong. Currently we're using 'SIGQUIT' to kill thin in 'proper way'. Mean it should finish all open connections and then just quit. However it seems like this for some 'unknown' reason doesn't kill thin and thin is still running. This behavior is weird, because I tested SIGQUIT few weeks ago on RHEL6.1 and F16 and this signal was ending thin correctly. The fix for now can be replacing SIGQUIT with SIGTERM which will kill thin immediately. I don't think this is the 'cleanest' way to end thin process but it works :-) The patch for init script (/etc/init.d/deltacloud-core) is one-char :-) - 64 kill -3 `cat $PIDFILE` + 64 kill -9 `cat $PIDFILE` If QA confirm that this fix is working for them, I'll push this fix to brew.
+ important note: Updating certificate-based repositories. Installed Packages Name : rubygem-thin Arch : x86_64 Version : 1.2.11 Release : 10.el6 Size : 249 k Repo : installed From repo : rhel-aeolus-testing Summary : A thin and fast web server URL : http://code.macournoyer.com/thin/ License : (GPLv2 or Ruby) and BSD and MIT Description : Thin is a Ruby web server that glues together three of the best Ruby : libraries in web history. : The Mongrel parser, the root of Mongrel speed and security, : Event Machine, a network I/O library with extremely high scalability and : Rack, a minimal interface between webservers and Ruby frameworks. [root@qeblade22 ~]# gem list | grep thin thin (1.3.1, 1.2.11) [root@qeblade22 ~]# thin --version thin 1.3.1 codename Triple Espresso You're using 'thin' installed using 'gem install' not the one available in our repository. I don't think this is somehow related, but the init script was tested and proved working with the version in aeolus-testing repository.
(In reply to comment #7) > OK seems like I figured out what is wrong. Currently we're using 'SIGQUIT' to > kill thin in 'proper way'. Mean it should finish all open connections and then > just quit. However it seems like this for some 'unknown' reason doesn't kill > thin and thin is still running. > This behavior is weird, because I tested SIGQUIT few weeks ago on RHEL6.1 and > F16 and this signal was ending thin correctly. > The fix for now can be replacing SIGQUIT with SIGTERM which will kill thin > immediately. I don't think this is the 'cleanest' way to end thin process but > it works :-) > > The patch for init script (/etc/init.d/deltacloud-core) is one-char :-) > > - 64 kill -3 `cat $PIDFILE` > + 64 kill -9 `cat $PIDFILE` > > If QA confirm that this fix is working for them, I'll push this fix to brew. From QA point of view this fix is working.
(In reply to comment #8) > + important note: > ok, just a note that for some reason I have the same stuff on my F15 box too, but there is no issue with deltacloud-core service. # yum info rubygem-thin Installed Packages Name : rubygem-thin Arch : x86_64 Version : 1.2.11 Release : 10.fc15 Size : 249 k Repo : installed From repo : updates Summary : A thin and fast web server URL : http://code.macournoyer.com/thin/ License : (GPLv2 or Ruby) and BSD and MIT Description : Thin is a Ruby web server that glues together three of the best Ruby : libraries in web history. : The Mongrel parser, the root of Mongrel speed and security, : Event Machine, a network I/O library with extremely high scalability and : Rack, a minimal interface between webservers and Ruby frameworks. # gem list | grep thin thin (1.3.1, 1.2.11) # thin --version thin 1.3.1 codename Triple Espresso
I'm moving to on_qa whayutin looking into it
this bug is referring to two subsystems of cloud engine.. deltacloud and dbomatic.. James correctly stated.. 771505 tracks the dbomatic. I'm not seening any issues w/ deltacloud at the moment.. For sanity sake.. I'm going to close this bug.. If we still see issues w/ deltacloud... lets open a new clean bug. *** This bug has been marked as a duplicate of bug 771505 ***