Bug 903388
| Summary: | Apparent (non-fatal) tomcat race condition during configuration | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Retired] Subscription Asset Manager | Reporter: | Og Maciel <omaciel> | ||||||||||||
| Component: | katello-configure | Assignee: | Ivan Necas <inecas> | ||||||||||||
| Status: | CLOSED ERRATA | QA Contact: | SAM QE List <sam-qe-list> | ||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||
| Priority: | unspecified | ||||||||||||||
| Version: | 1.2 | CC: | bkearney, jdornak, liliu, mmccune, sgao, sthirugn, xdmoon | ||||||||||||
| Target Milestone: | rc | Keywords: | Triaged | ||||||||||||
| Target Release: | --- | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | Environment: | ||||||||||||||
| Last Closed: | 2013-10-01 11:04:08 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Attachments: |
|
||||||||||||||
When I tried to deploy SAM from RHN(Product CDN) with RHEL build RHEL6.4-20130116.3, the installation process(yum install -y katello-headpin-all) was completed, but failed to run the command: katello-configure --deployment=sam as follows:
[root@rhel6 ~]# katello-configure --deployment=sam
Starting Katello configuration
The top-level log file is [/var/log/katello/katello-configure-20130124-024315/main.log]
Creating Candlepin database user
############################################################ ... OK
Creating Katello database user
############################################################ ... OK
Creating Katello database
############################################################ ... OK
err: /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of '/usr/sbin/service-wait tomcat6 restart' returned 4: at /usr/share/katello/install/puppet/modules/candlepin/manifests/service.pp:18
Populating Katello database schema
############################################################ ... OK
Initializing Katello data
###########################################################
Failed, please check [/var/log/katello/katello-configure/db_seed.log]
Report errors using # katello-debug tool.
[1]Log attached in file db_seed.log.
[2]Result of running "rpm -qa | grep java"
[root@samserv ~]# rpm -qa | grep java
jboss-javaee-poms-5.0.1-2.9.ep5.el6.noarch
tzdata-java-2012i-2.el6.noarch
java-1.7.0-openjdk-1.7.0.9-2.3.3.2.el6_3.x86_64
glassfish-javamail-1.4.4-6_redhat_1.ep6.el6.1.noarch
java_cup-0.10k-5.el6.x86_64
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64
[root@samserv ~]#
[3] Run command "katello-configure --deployment=sam" again, it still failed with following ERROR.
[root@samserv ~]# katello-configure --deployment=sam
Starting Katello configuration
The top-level log file is [/var/log/katello/katello-configure-20130124-213028/main.log]
Creating Candlepin database user
############################################################ ... OK
Creating Katello database user
############################################################ ... OK
Creating Katello database
############################################################ ... OK
Initializing Katello data
###########################################################
Failed, please check [/var/log/katello/katello-configure/db_seed.log]
Report errors using # katello-debug tool.
Created attachment 687166 [details]
sam log
Hi Og, Since the newest SAM1.2 has been pushed to QA CDN and content test is PASS, I installed SAM1.2 from QA CDN on RHEL6.4-RC-1.0-Server-x86_64, the deployment are succcessful, but the error message methioned in this bug still exists when running "katello-configure --deployment=sam" as bellow: [root@samserv ~]# katello-configure --deployment=sam Starting Katello configuration The top-level log file is [/var/log/katello/katello-configure-20130208-110500/main.log] Creating Candlepin database user ############################################################ ... OK err: /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of '/usr/sbin/service-wait tomcat6 restart' returned 4: at /usr/share/katello/install/puppet/modules/candlepin/manifests/service.pp:18 Creating Katello database user ############################################################ ... OK Creating Katello database ############################################################ ... OK Populating Katello database schema ############################################################ ... OK Initializing Katello data ############################################################ ... OK [root@samserv ~]# Pls see details in attachment samdeployrc.log Created attachment 694992 [details]
samdeployrc.log
Hey Shang Gao, Developers have not mentioned any concerns about this (apparently) non-fatal issue that seem to only affect newer versions of rhel. I will chat with them once again today. Ditto comment 10: Tested the below version in RHEL6.4-20130130.0 Server x86_64: * candlepin-0.7.24-1.el6_3.noarch * candlepin-selinux-0.7.24-1.el6_3.noarch * candlepin-tomcat6-0.7.24-1.el6_3.noarch * elasticsearch-0.19.9-5.el6_3.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.2.1-1h.el6_3.noarch * katello-cli-1.2.1-12h.el6_3.noarch * katello-cli-common-1.2.1-12h.el6_3.noarch * katello-common-1.2.1-18h.el6_4.noarch * katello-configure-1.2.3-6h.el6_4.noarch * katello-glue-candlepin-1.2.1-18h.el6_4.noarch * katello-headpin-1.2.1-18h.el6_4.noarch * katello-headpin-all-1.2.1-18h.el6_4.noarch * katello-selinux-1.2.1-2h.el6_3.noarch * thumbslug-0.0.28-1.el6_3.noarch * thumbslug-selinux-0.0.28-1.el6_3.noarch # katello-configure --deployment=sam Starting Katello configuration The top-level log file is [/var/log/katello/katello-configure-20130308-114642/main.log] Creating Katello database user ############################################################ ... OK Creating Katello database ############################################################ ... OK Creating Candlepin database user ############################################################ ... OK err: /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of '/usr/sbin/service-wait tomcat6 restart' returned 4: at /usr/share/katello/install/puppet/modules/candlepin/manifests/service.pp:17 Populating Katello database schema ############################################################ ... OK Initializing Katello data ############################################################ ... OK *** Bug 920210 has been marked as a duplicate of this bug. *** There should be API polling in katello-configure to check if candlepin is up. this sounds like a regression or some new side effect. Does anyone have katello-debug output or at lest the /var/log/katello/katello-configure/main.log file? Attaching katello-debug and a TGZ of /var/log/katello/* Created attachment 730818 [details]
katello-debug
Created attachment 730819 [details]
/var/log/katello/* files
I've identified some changes in /etc/init.d/tomcat6 init script between 6.3 and 6.4 that might cause this, looking deeper into it, it doesn't seem to be service-wait issue. The cause of the issue seems to be running `service tomcat6 start` following `service tomcat6 restart`, restart fails in this case with exit code 4 service tomcat6 start; echo $?; service tomcat6 restart; echo $? Investigating cause and finding solution (perhpas better than sleeping for some time somewhere) So it was service-wait issue at the end: the waiting for service didn't count on SSL not ready. Using curl as additional check fixes it. The proposed fix is here: https://github.com/Katello/katello/pull/1836 This did not happen to me in any of the recent sam 1.3 snaps. Version Tested: * candlepin-0.8.26-1.el6sam.noarch * candlepin-cert-consumer-cloud-qe-19.idm.lab.bos.redhat.com-1.0-1.noarch * candlepin-scl-1-5.el6_4.noarch * candlepin-scl-quartz-2.1.5-5.el6_4.noarch * candlepin-scl-rhino-1.7R3-1.el6_4.noarch * candlepin-scl-runtime-1-5.el6_4.noarch * candlepin-selinux-0.8.26-1.el6sam.noarch * candlepin-tomcat6-0.8.26-1.el6sam.noarch * elasticsearch-0.19.9-8.el6sat.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.4.2-2.el6sat.noarch * katello-cli-1.4.3-11.el6sat.noarch * katello-cli-common-1.4.3-11.el6sat.noarch * katello-common-1.4.3-15.el6sam_splice.noarch * katello-configure-1.4.4-4.el6sat.noarch * katello-glue-candlepin-1.4.3-15.el6sam_splice.noarch * katello-glue-elasticsearch-1.4.3-15.el6sam_splice.noarch * katello-headpin-1.4.3-15.el6sam_splice.noarch * katello-headpin-all-1.4.3-15.el6sam_splice.noarch * katello-selinux-1.4.4-2.el6sat.noarch * thumbslug-0.0.34-1.el6sam.noarch * thumbslug-selinux-0.0.34-1.el6sam.noarch Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1390.html |
Created attachment 686283 [details] catalina.out Description of problem: We have seen cases where tomcat6 is slow to start and the configuration process shows an error: Creating Candlepin database user ############################################################ ... OK err: /Stage[main]/Candlepin::Service/Service[tomcat6]: Failed to call refresh: Could not restart Service[tomcat6]: Execution of '/usr/sbin/service-wait tomcat6 restart' returned 4: at /usr/share/katello/install/puppet/modules/candlepin/manifests/service.pp:18 The configuration process does however finish successfully. Version-Release number of selected component (if applicable): * candlepin-0.7.23-1.el6_3.noarch * candlepin-tomcat6-0.7.23-1.el6_3.noarch * elasticsearch-0.19.9-5.el6_3.noarch * katello-candlepin-cert-key-pair-1.0-1.noarch * katello-certs-tools-1.2.1-1h.el6_3.noarch * katello-cli-1.2.1-12h.el6_3.noarch * katello-cli-common-1.2.1-12h.el6_3.noarch * katello-common-1.2.1-15h.el6_3.noarch * katello-configure-1.2.3-3h.el6_3.noarch * katello-glue-candlepin-1.2.1-15h.el6_3.noarch * katello-headpin-1.2.1-15h.el6_3.noarch * katello-headpin-all-1.2.1-15h.el6_3.noarch * katello-selinux-1.2.1-2h.el6_3.noarch * thumbslug-0.0.28-1.el6_3.noarch * thumbslug-selinux-0.0.28-1.el6_3.noarch * tomcat6-6.0.24-48.el6.noarch How reproducible: Hard to reproduce which lead me to think there is a race condition. Steps to Reproduce: 1. Install and configure latest sam code. 2. 3. Actual results: Expected results: Additional info: