Red Hat Bugzilla – Bug 1253432
Some upgrades are failing on RHEL 6 with an error about tomcat
Last modified: 2017-02-23 14:47:55 EST
Description of problem: Stopping tomcat6: waiting for processes 2787 to exit killing 2787 which did not stop after 60 seconds [WARNING] [ OK ] Starting tomcat6: [ OK ] Stopping httpd: [ OK ] Starting httpd: [ OK ] Starting foreman-tasks: [ OK ] Some services failed: tomcat6 [ERROR 2015-08-12 12:22:53 main] Upgrade step restart_services failed. Check logs for more information. [DEBUG 2015-08-12 12:22:53 main] Exit with status code: 1 (signal was 1) [ERROR 2015-08-12 12:22:53 main] Repeating errors encountered during run: [ERROR 2015-08-12 12:22:53 main] mongod is stopped *[ERROR 2015-08-12 12:22:53 main] <NilClass> nil* [ERROR 2015-08-12 12:22:53 main] httpd is stopped
I am not able to reproduce this on my own systems, not matter what I do to tomcat. If anyone encounters this, it would be very helpful to see what step it's failing on, and what state tomcat is in. This script gathers that data: #!/bin/bash time service tomcat6 stop echo "Exit code: " $? time service tomcat6 start echo "Exit code: " $? # maximum time to wait (in seconds) WAIT_MAX=${WAIT_MAX:-30} TOMCAT_PORT=${TOMCAT_PORT:-8443} TOMCAT_SERV_PORT=${TOMCAT_SERV_PORT:-8005} TOMCAT_TEST_URL=${TOMCAT_TEST_URL:-https://localhost:$TOMCAT_PORT/candlepin/status} wait_for_url() { echo $(date) time /usr/bin/wget --timeout=1 --tries=$WAIT_MAX --retry-connrefused -qO- --no-check-certificate $1 echo "wget exit code ${?}" time /usr/bin/curl -ks --retry $WAIT_MAX --retry-delay 1 $1 echo "" echo "curl exit code ${?}" if ! [ $? = '0' ]; then RETVAL=5 fi } time wait_for_url $TOMCAT_TEST_URL echo "wait_for_url exit code ${?}" time service-wait tomcat6 restart echo "service-wait exit code ${?}"
Customer experiencing the issue in (Case 01491540) ran Tomcat stop/start script from Comment #1. Results.... Stopping tomcat6: waiting for processes 28198 to exit [ OK ] real 0m3.344s user 0m0.311s sys 0m0.049s Exit code: 0 Starting tomcat6: [ OK ] real 0m0.078s user 0m0.018s sys 0m0.017s Exit code: 0 Thu Aug 13 14:38:31 EDT 2015 real 0m0.014s user 0m0.012s sys 0m0.002s wget exit code 4 real 0m0.004s user 0m0.000s sys 0m0.002s curl exit code 0 real 0m0.019s user 0m0.012s sys 0m0.004s wait_for_url exit code 0 Stopping tomcat6: waiting for processes 28938 to exit killing 28938 which did not stop after 60 seconds [WARNING] [ OK ] Starting tomcat6: [ OK ] real 1m1.994s user 0m0.496s sys 0m0.390s service-wait exit code 5
As a workaround I commented out the tomcat6 call in /usr/sbin/service-wait. That allows me to upgrade the Satellite.
Created redmine issue http://projects.theforeman.org/issues/11353 from this bug
Thanks for those who supplied the output of that, it helped understand what was going wrong. There's a brief window where tomcat's listening on 8443, but not responding 200 to /candlepin/status. Sometimes, we end up calling wget in that window, which exits immediately with a failure. wget does not obey the --tries there in this case. It's generally reproducible if you do this as one command, you'll see wget does NOT retry: service tomcat6 stop; service tomcat6 start; /usr/bin/wget --timeout=1 --tries=30 --retry-connrefused -qO- --no-check-certificate https://localhost:8443/candlepin/status; echo $?
Upstream bug assigned to stbenjam@redhat.com
Moving to POST since upstream bug http://projects.theforeman.org/issues/11353 has been closed
Ok, I was trying to verify this bz. I installed sat6.0.8 and populated some content (along with capsule/provisioning conf) and upgraded the server with snap17. Upgrade is completed successfully. [root@cloud-qe-9 yum.repos.d]# katello-installer --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_mongo... Upgrade Step: migrate_pulp... Upgrade Step: start_httpd... Upgrade Step: migrate_candlepin... Upgrade Step: migrate_foreman... Upgrade Step: Running installer... Installing Done [100%] [..................................................................] The full log is at /var/log/katello-installer/katello-installer.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade Step: errata_import (this may take a while) ... Upgrade Step: update_gpg_urls (this may take a while) ... Upgrade Step: update_repository_metadata (this may take a while) ... Katello upgrade completed!
I'm curious to know if there is any other way to know whether the original issue is really fixed with new snap ? I can see the changes in `/usr/share/katello/script/service-wait` but still would like to double check. thanks
Based on comments 13 and 15. Moving this to verified. Thanks
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:1688