Bug 1340122
| Summary: | [3.0.0 RC5] UPG failed due to Candlepin/Pulp Issues (Multiple Content Host Entries linked to Host-Collections inside DB) | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Justin Sherrill <jsherril> |
| Component: | Content Management | Assignee: | Justin Sherrill <jsherril> |
| Status: | CLOSED ERRATA | QA Contact: | Sachin Ghai <sghai> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.2.0 | CC: | bbuckingham, ehelms, sghai |
| Target Milestone: | Unspecified | Keywords: | Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| URL: | http://projects.theforeman.org/issues/15121 | ||
| Whiteboard: | |||
| Fixed In Version: | rubygem-katello-3.0.0.38-1,rubygem-katello-3.0.0.38-1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-07-27 11:20:54 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1335807 | ||
Created from redmine issue http://projects.theforeman.org/issues/15121 Steps to reproduce would be: on 6.1, register two systems with the same hostname (or the same system twice after a sub-man clean). Associate both systems to a host collection upgrade to 6.2 Verified with upgrade performed from sat6.1.9 -> sat6.2 GA snap17 I created two content hosts with same and added them in a host-collection. ~]# hammer -u admin -p changeme content-host list --organization-id 1 | grep bzhost1 19f78156-1928-495e-855e-1ec74d4d01b2 | bzhost1.satqe.lab.eng.rdu2.redhat.com | 0 1464d7fd-23a3-42f2-97d5-f89dda61685f | bzhost1.satqe.lab.eng.rdu2.redhat.com | 0 ]# hammer -u admin -p changeme host-collection content-hosts --id 3 --organization-id 1 -------------------------------------|-------------------------------------- ID | NAME -------------------------------------|-------------------------------------- 19f78156-1928-495e-855e-1ec74d4d01b2 | bzhost1.satqe.lab.eng.rdu2.redhat.com 1464d7fd-23a3-42f2-97d5-f89dda61685f | bzhost1.satqe.lab.eng.rdu2.redhat.com -------------------------------------|-------------------------------------- Later, I performed the upgrade and upgrade was completed successfully. [root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade Upgrading... Upgrade Step: stop_services... Upgrade Step: start_databases... Upgrade Step: update_http_conf... Upgrade Step: migrate_pulp... Upgrade Step: fix_pulp_httpd_conf... Upgrade Step: start_httpd... Upgrade Step: start_qpidd... Upgrade Step: start_pulp... Upgrade Step: migrate_candlepin... Upgrade Step: start_tomcat... Upgrade Step: fix_katello_settings_file... Upgrade Step: migrate_foreman... Upgrade Step: remove_nodes_distributors... Upgrade Step: Running installer... Installing Done [100%] [..................................................................] The full log is at /var/log/foreman-installer/satellite.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade Step: import_package_groups (this may take a while) ... Upgrade Step: import_rpms (this may take a while) ... Upgrade Step: import_distributions (this may take a while) ... Upgrade Step: import_puppet_modules (this may take a while) ... Upgrade Step: import_subscriptions (this may take a while) ... Upgrade Step: elasticsearch_message... Elasticsearch has been removed as a dependency. The database files can be removed manually with #rm -rf /var/lib/elasticsearch. Some packages are no longer needed and can be removed: #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4 Upgrade Step: remove_docker_v1_content (this may take a while) ... Upgrade Step: update_puppet_repository_distributors (this may take a while) ... Upgrade Step: remove_gutterball... Katello upgrade completed! Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:1501 |
Dear all, I think my UPG is failing due to the below which is related to the new "Host Unification" Feature. Please let me know if the Infos are enough or if you need the logs as well in order to supply a fix... *"foreman-installer --scenario katello --upgrade"* Upgrade Step: Running installer... /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] Installing Done [100%] [...............................] Something went wrong! Check the log for ERROR-level output The full log is at /var/log/foreman-installer/katello.log Upgrade Step: restart_services... Upgrade Step: db_seed... Upgrade step db_seed failed. Check logs for more information. --- Run `rake db:migrate` to update your database then try again. Upgrade step db_seed failed. Check logs for more information. --- Multiple content hosts with the hostname centos-6-clone1.cesystems.at found, unregistering all except last registered. Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13 Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13 Multiple content hosts with the hostname centos-7-clone1.cesystems.at found, unregistering all except last registered. Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6 Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6 Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at Found host with hostname centos-6-genericcloud-ovirt.cesystems.at. Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at. Content Host d3078602-3cb0-43b7-b326-c3534c8e0e13 doesn't have candlepin information, unregistering. Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13 Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 Exception when destroying candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"} (DELETE /candlepin/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13): {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"} Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 404 NotFound | application/json 528 bytes | Pulp consumer not found for consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 proceeding. Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13 Content Host ece0be43-0e4d-423a-967c-8d53fa4c99d6 doesn't have candlepin information, unregistering. Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6 Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 Exception when destroying candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"} (DELETE /candlepin/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6): {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"} Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 [katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json" | \n# => 404 NotFound | application/json 528 bytes | Pulp consumer not found for consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 proceeding. Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6 Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at Found host with hostname centos-6-genericcloud-ovirt.cesystems.at. Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at. *"GUI => Monitor => Tasks"* Actions::Katello::Host::Destroy {"host"=>{"id"=>251, "name"=>"centos-6-clone1.cesystems.at"}, "pool_ids"=> ["8a2180484c14ecee014c1503fe820033", "8a2180484ca12e87014ca2277340001a", "8a2180484c14ecee014c15052fd4003d", "8a2180484ca12e87014ca21b43930002"]} ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections" DETAIL: Key (id)=(157) is still referenced from table "katello_system_host_collections". : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1 --- Actions::Katello::Host::Destroy {"host"=>{"id"=>252, "name"=>"centos-7-clone1.cesystems.at"}, "pool_ids"=> ["8a2180484ca12e87014ca21bdf580009", "8a2180484ca12e87014ca22908890023", "8a2180484c14ecee014c1503fe820033", "8a2180484c14ecee014c15052fd4003d"]} ActiveRecord::InvalidForeignKey: PG::Error: ERROR: update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections" DETAIL: Key (id)=(158) is still referenced from table "katello_system_host_collections". : DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1 *"hammer ping"* candlepin: Status: ok Server Response: Duration: 30ms candlepin_auth: Status: ok Server Response: Duration: 32ms pulp: Status: ok Server Response: Duration: 35ms foreman_tasks: Status: ok Server Response: Duration: 17ms *"foreman-rake db:migrate"* == 20150930183738 MigrateContentHosts: migrating ============================== rake aborted! StandardError: An error has occurred, this and all later migrations canceled: and so on.. --- May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=d3078602-3cb0-43b7-b326-c3534c8e0e13 May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=ece0be43-0e4d-423a-967c-8d53fa4c99d6