Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem:
[root@satellite ~]# satellite-installer --scenario satellite --upgrade
Upgrading...
Upgrade Step: stop_services...
Redirecting to /bin/systemctl stop foreman-tasks.service
Redirecting to /bin/systemctl stop httpd.service
Redirecting to /bin/systemctl stop pulp_workers.service
Redirecting to /bin/systemctl stop foreman-proxy.service
Redirecting to /bin/systemctl stop pulp_streamer.service
Redirecting to /bin/systemctl stop pulp_resource_manager.service
Redirecting to /bin/systemctl stop pulp_celerybeat.service
Redirecting to /bin/systemctl stop smart_proxy_dynflow_core.service
Redirecting to /bin/systemctl stop tomcat.service
Redirecting to /bin/systemctl stop squid.service
Redirecting to /bin/systemctl stop qdrouterd.service
Redirecting to /bin/systemctl stop qpidd.service
Success!
Upgrade Step: start_databases...
Redirecting to /bin/systemctl start mongod.service
Redirecting to /bin/systemctl start postgresql.service
Success!
Upgrade Step: update_http_conf...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: migrate_pulp...
29717
Attempting to connect to localhost:27017
Attempting to connect to localhost:27017
Write concern for Mongo connection: {}
Loading content types.
Loading type descriptors []
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Updating the database with types []
Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image]
Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image]
Found the following type definitions that were not present in the update collection [node, repository]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 23
Migration package pulp_docker.plugins.migrations is up to date at version 2
Migration package pulp_puppet.plugins.migrations is up to date at version 5
Migration package pulp_rpm.plugins.migrations is up to date at version 32
Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module
Loading unit model: docker_blob = pulp_docker.plugins.models:Blob
Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest
Loading unit model: docker_image = pulp_docker.plugins.models:Image
Loading unit model: docker_tag = pulp_docker.plugins.models:Tag
Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata
Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution
Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup
Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory
Loading unit model: iso = pulp_rpm.plugins.db.models:ISO
Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment
Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM
Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM
Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM
Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile
Database migrations complete.
Upgrade Step: start_httpd...
Redirecting to /bin/systemctl start httpd.service
Success!
Upgrade Step: start_qpidd...
Redirecting to /bin/systemctl start qpidd.service
Redirecting to /bin/systemctl start qdrouterd.service
Success!
Upgrade Step: start_pulp...
Redirecting to /bin/systemctl start pulp_celerybeat.service
Redirecting to /bin/systemctl start pulp_resource_manager.service
Redirecting to /bin/systemctl start pulp_workers.service
Success!
Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful
Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start tomcat.service
Some services failed to start: tomcat
Upgrade step start_tomcat failed. Check logs for more information.
Version-Release number of selected component (if applicable):
How reproducible: 50%
Steps to Reproduce:
1. yum update the satellite
2. upgrade satellite to 6.2.7
3.
Actual results:
Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start tomcat.service
Some services failed to start: tomcat
Expected results: upgrade to complete
Additional info:
When checking # alternatives --config java default on all of these failing is 1.8 .. when we switch this to 1.7 upgrade will complete. Why is the default being changed to 1.8?
A typical traceback to key you in on this exact issue and workaround is the following from foreman/production.log:
2017-02-01 16:18:24 [katello/cp_proxy] [E] OpenSSL::SSL::SSLError: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A
| /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect'
| /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `block in connect'
| /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:89:in `block in timeout'
| /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `call'
| /opt/rh/rh-ruby22/root/usr/share/ruby/timeout.rb:99:in `timeout'
| /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:923:in `connect'
| /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:863:in `do_start'
| /opt/rh/rh-ruby22/root/usr/share/ruby/net/http.rb:852:in `start'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.37/lib/restclient_ext/request.rb:44:in `transmit'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
Looking at alternatives data from sosreport, you will see something similar to the following:
[/sos_commands/java/alternatives*]
<snip>
java - status is auto.
link currently points to /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java
/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.121-2.6.8.0.el7_3.x86_64/jre/bin/java - family java-1.7.0-openjdk.x86_64 priority 1700121
</snip>
I was unable to reproduce this & we have not had a case since 3/10. Likely a JVM issue but without a reproducer I don't have the information required to fix it. Please re-open if a reproducer can be reliably created.
Description of problem: [root@satellite ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Redirecting to /bin/systemctl stop foreman-tasks.service Redirecting to /bin/systemctl stop httpd.service Redirecting to /bin/systemctl stop pulp_workers.service Redirecting to /bin/systemctl stop foreman-proxy.service Redirecting to /bin/systemctl stop pulp_streamer.service Redirecting to /bin/systemctl stop pulp_resource_manager.service Redirecting to /bin/systemctl stop pulp_celerybeat.service Redirecting to /bin/systemctl stop smart_proxy_dynflow_core.service Redirecting to /bin/systemctl stop tomcat.service Redirecting to /bin/systemctl stop squid.service Redirecting to /bin/systemctl stop qdrouterd.service Redirecting to /bin/systemctl stop qpidd.service Success! Upgrade Step: start_databases... Redirecting to /bin/systemctl start mongod.service Redirecting to /bin/systemctl start postgresql.service Success! Upgrade Step: update_http_conf... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: migrate_pulp... 29717 Attempting to connect to localhost:27017 Attempting to connect to localhost:27017 Write concern for Mongo connection: {} Loading content types. Loading type descriptors [] Parsing type descriptors Validating type descriptor syntactic integrity Validating type descriptor semantic integrity Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module Loading unit model: docker_blob = pulp_docker.plugins.models:Blob Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest Loading unit model: docker_image = pulp_docker.plugins.models:Image Loading unit model: docker_tag = pulp_docker.plugins.models:Tag Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory Loading unit model: iso = pulp_rpm.plugins.db.models:ISO Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile Updating the database with types [] Found the following type definitions that were not present in the update collection [node, puppet_module, docker_tag, repository, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, drpm, distribution, rpm, srpm, docker_image] Updating the database with types [puppet_module, drpm, erratum, docker_blob, docker_manifest, yum_repo_metadata_file, package_group, package_category, iso, package_environment, docker_tag, distribution, rpm, srpm, docker_image] Found the following type definitions that were not present in the update collection [node, repository] Content types loaded. Ensuring the admin role and user are in place. Admin role and user are in place. Beginning database migrations. Migration package pulp.server.db.migrations is up to date at version 23 Migration package pulp_docker.plugins.migrations is up to date at version 2 Migration package pulp_puppet.plugins.migrations is up to date at version 5 Migration package pulp_rpm.plugins.migrations is up to date at version 32 Loading unit model: puppet_module = pulp_puppet.plugins.db.models:Module Loading unit model: docker_blob = pulp_docker.plugins.models:Blob Loading unit model: docker_manifest = pulp_docker.plugins.models:Manifest Loading unit model: docker_image = pulp_docker.plugins.models:Image Loading unit model: docker_tag = pulp_docker.plugins.models:Tag Loading unit model: erratum = pulp_rpm.plugins.db.models:Errata Loading unit model: distribution = pulp_rpm.plugins.db.models:Distribution Loading unit model: package_group = pulp_rpm.plugins.db.models:PackageGroup Loading unit model: package_category = pulp_rpm.plugins.db.models:PackageCategory Loading unit model: iso = pulp_rpm.plugins.db.models:ISO Loading unit model: package_environment = pulp_rpm.plugins.db.models:PackageEnvironment Loading unit model: drpm = pulp_rpm.plugins.db.models:DRPM Loading unit model: srpm = pulp_rpm.plugins.db.models:SRPM Loading unit model: rpm = pulp_rpm.plugins.db.models:RPM Loading unit model: yum_repo_metadata_file = pulp_rpm.plugins.db.models:YumMetadataFile Database migrations complete. Upgrade Step: start_httpd... Redirecting to /bin/systemctl start httpd.service Success! Upgrade Step: start_qpidd... Redirecting to /bin/systemctl start qpidd.service Redirecting to /bin/systemctl start qdrouterd.service Success! Upgrade Step: start_pulp... Redirecting to /bin/systemctl start pulp_celerybeat.service Redirecting to /bin/systemctl start pulp_resource_manager.service Redirecting to /bin/systemctl start pulp_workers.service Success! Upgrade Step: migrate_candlepin... Migrating candlepin database Liquibase Update Successful Upgrade Step: start_tomcat... Redirecting to /bin/systemctl start tomcat.service Some services failed to start: tomcat Upgrade step start_tomcat failed. Check logs for more information. Version-Release number of selected component (if applicable): How reproducible: 50% Steps to Reproduce: 1. yum update the satellite 2. upgrade satellite to 6.2.7 3. Actual results: Upgrade Step: start_tomcat... Redirecting to /bin/systemctl start tomcat.service Some services failed to start: tomcat Expected results: upgrade to complete Additional info: When checking # alternatives --config java default on all of these failing is 1.8 .. when we switch this to 1.7 upgrade will complete. Why is the default being changed to 1.8?