Bug 1418410 - Upgrading to Satellite 6.2.7 fails with Some services failed to start: tomcat due to java 1.8 becoming default
Summary: Upgrading to Satellite 6.2.7 fails with Some services failed to start: tomcat...
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Candlepin
Version: 6.2.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: Barnaby Court
QA Contact: Katello QA List
URL:
Whiteboard:
Depends On:
Blocks: Sat6_Upgrades
TreeView+ depends on / blocked
 
Reported: 2017-02-01 18:48 UTC by Kathryn Dixon
Modified: 2020-07-16 09:10 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-04-10 18:40:34 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2898631 0 None None None 2017-02-03 15:41:36 UTC

Description Kathryn Dixon 2017-02-01 18:48:45 UTC
Description of problem:

[root@satellite ~]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Redirecting to /bin/systemctl stop  foreman-tasks.service

Redirecting to /bin/systemctl stop  httpd.service

Redirecting to /bin/systemctl stop  pulp_workers.service

Redirecting to /bin/systemctl stop  foreman-proxy.service

Redirecting to /bin/systemctl stop  pulp_streamer.service

Redirecting to /bin/systemctl stop  pulp_resource_manager.service

Redirecting to /bin/systemctl stop  pulp_celerybeat.service

Redirecting to /bin/systemctl stop  smart_proxy_dynflow_core.service

Redirecting to /bin/systemctl stop  tomcat.service

Redirecting to /bin/systemctl stop  squid.service

Redirecting to /bin/systemctl stop  qdrouterd.service

Redirecting to /bin/systemctl stop  qpidd.service

Success!

Upgrade Step: start_databases...
Redirecting to /bin/systemctl start  mongod.service

Redirecting to /bin/systemctl start  postgresql.service

Success!

Upgrade Step: update_http_conf...

Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: migrate_pulp...
29717

Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image]
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Found the following type definitions that were not present in the update collection [node, repository]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 23
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_rpm.plugins.migrations is up to date at version 32
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.

Upgrade Step: start_httpd...
Redirecting to /bin/systemctl start  httpd.service

Success!

Upgrade Step: start_qpidd...
Redirecting to /bin/systemctl start  qpidd.service

Redirecting to /bin/systemctl start  qdrouterd.service

Success!

Upgrade Step: start_pulp...
Redirecting to /bin/systemctl start  pulp_celerybeat.service

Redirecting to /bin/systemctl start  pulp_resource_manager.service

Redirecting to /bin/systemctl start  pulp_workers.service

Success!

Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Some services failed to start: tomcat

Upgrade step start_tomcat failed. Check logs for more information.


Version-Release number of selected component (if applicable):


How reproducible: 50%


Steps to Reproduce:
1. yum update the satellite
2. upgrade satellite to 6.2.7
3.

Actual results: 

Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service

Some services failed to start: tomcat


Expected results: upgrade to complete 


Additional info:

When checking # alternatives --config java default on all of these failing is 1.8 .. when we switch this to 1.7 upgrade will complete. Why is the default being changed to 1.8?

Comment 3 Craig Donnelly 2017-02-01 19:40:16 UTC
A typical traceback to key you in on this exact issue and workaround is the following from foreman/production.log:

2017-02-01 16:18:24 [katello/cp_proxy] [E] OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `block in connect'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start'
 | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:852:in `start'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'

Comment 4 Craig Donnelly 2017-02-01 19:44:07 UTC
Looking at alternatives data from sosreport, you will see something similar to the following:

[/sos_commands/java/alternatives*]
<snip>
java - status is auto.
 link currently points to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.121-2.6.8.0.el7_3.x86_64/jre/bin/java - family java-1.7.0-openjdk.x86_64 priority 1700121
</snip>

Comment 10 Barnaby Court 2017-04-10 18:40:34 UTC
I was unable to reproduce this & we have not had a case since 3/10. Likely a JVM issue but without a reproducer I don't have the information required to fix it. Please re-open if a reproducer can be reliably created.


Note You need to log in before you can comment on or make changes to this bug.