Description of problem: [root@satellite ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Redirecting to /bin/systemctl stop foreman-tasks.service Redirecting to /bin/systemctl stop httpd.service Redirecting to /bin/systemctl stop pulp_workers.service Redirecting to /bin/systemctl stop foreman-proxy.service Redirecting to /bin/systemctl stop pulp_streamer.service Redirecting to /bin/systemctl stop pulp_resource_manager.service Redirecting to /bin/systemctl stop pulp_celerybeat.service Redirecting to /bin/systemctl stop smart_proxy_dynflow_core.service Redirecting to /bin/systemctl stop tomcat.service Redirecting to /bin/systemctl stop squid.service Redirecting to /bin/systemctl stop qdrouterd.service Redirecting to /bin/systemctl stop qpidd.service Success! Upgrade Step: start_databases... Redirecting to /bin/systemctl start mongod.service Redirecting to /bin/systemctl start postgresql.service Success! Upgrade Step: update_http_conf... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: migrate_pulp... 29717 Attempting to connect to localhost:27017 Attempting to connect to localhost:27017 Write concern for Mongo connection: {} Loading content types. Loading type descriptors [] Parsing type descriptors Validating type descriptor syntactic integrity Validating type descriptor semantic integrity Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module Loading unit model: docker_blob = pulp_docker.plugins.models:Blob Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest Loading unit model: docker_image = pulp_docker.plugins.models:Image Loading unit model: docker_tag = pulp_docker.plugins.models:Tag Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory Loading unit model: iso = pulp_rpm.plugins.db.models:ISO Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile Updating the database with types [] Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image] Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image] Found the following type definitions that were not present in the update collection [node, repository] Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Migration package pulp.server.db.migrations is up to date at version 23 Migration package pulp_docker.plugins.migrations is up to date at version 2 Migration package pulp_puppet.plugins.migrations is up to date at version 5 Migration package pulp_rpm.plugins.migrations is up to date at version 32 Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module Loading unit model: docker_blob = pulp_docker.plugins.models:Blob Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest Loading unit model: docker_image = pulp_docker.plugins.models:Image Loading unit model: docker_tag = pulp_docker.plugins.models:Tag Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory Loading unit model: iso = pulp_rpm.plugins.db.models:ISO Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile Database migrations complete. Upgrade Step: start_httpd... Redirecting to /bin/systemctl start httpd.service Success! Upgrade Step: start_qpidd... Redirecting to /bin/systemctl start qpidd.service Redirecting to /bin/systemctl start qdrouterd.service Success! Upgrade Step: start_pulp... Redirecting to /bin/systemctl start pulp_celerybeat.service Redirecting to /bin/systemctl start pulp_resource_manager.service Redirecting to /bin/systemctl start pulp_workers.service Success! Upgrade Step: migrate_candlepin... Migrating candlepin database Liquibase Update Successful Upgrade Step: start_tomcat... Redirecting to /bin/systemctl start tomcat.service Some services failed to start: tomcat Upgrade step start_tomcat failed. Check logs for more information. Version-Release number of selected component (if applicable): How reproducible: 50% Steps to Reproduce: 1. yum update the satellite 2. upgrade satellite to 6.2.7 3. Actual results: Upgrade Step: start_tomcat... Redirecting to /bin/systemctl start tomcat.service Some services failed to start: tomcat Expected results: upgrade to complete Additional info: When checking # alternatives --config java default on all of these failing is 1.8 .. when we switch this to 1.7 upgrade will complete. Why is the default being changed to 1.8?
A typical traceback to key you in on this exact issue and workaround is the following from foreman/production.log: 2017-02-01 16:18:24 [katello/cp_proxy] [E] OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect' | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `block in connect' | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout' | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call' | /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout' | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect' | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start' | /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:852:in `start' | /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit' | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
Looking at alternatives data from sosreport, you will see something similar to the following: [/sos_commands/java/alternatives*] <snip> java - status is auto. link currently points to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.121-2.6.8.0.el7_3.x86_64/jre/bin/java - family java-1.7.0-openjdk.x86_64 priority 1700121 </snip>
I was unable to reproduce this & we have not had a case since 3/10. Likely a JVM issue but without a reproducer I don't have the information required to fix it. Please re-open if a reproducer can be reliably created.