Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1340122 - [3.0.0 RC5] UPG failed due to Candlepin/Pulp Issues (Multiple Content Host Entries linked to Host-Collections inside DB)
Summary: [3.0.0 RC5] UPG failed due to Candlepin/Pulp Issues (Multiple Content Host En...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Content Management
Version: 6.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: Unspecified
Assignee: Justin Sherrill
QA Contact: Sachin Ghai
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks: 1335807
TreeView+ depends on / blocked
 
Reported: 2016-05-26 13:28 UTC by Justin Sherrill
Modified: 2019-09-26 17:32 UTC (History)
3 users (show)

Fixed In Version: rubygem-katello-3.0.0.38-1,rubygem-katello-3.0.0.38-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-27 11:20:54 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 15121 0 None None None 2016-05-26 14:02:31 UTC

Description Justin Sherrill 2016-05-26 13:28:36 UTC
Dear all,

I think my UPG is failing due to the below which is related to the new "Host Unification" Feature.
Please let me know if the Infos are enough or if you need the logs as well in order to supply a fix...

*"foreman-installer --scenario katello --upgrade"*

Upgrade Step: Running installer...
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
 /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing             Done                                               [100%] [...............................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/katello.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade step db_seed failed. Check logs for more information.
---
Run `rake db:migrate` to update your database then try again.
Upgrade step db_seed failed. Check logs for more information.
---
Multiple content hosts with the hostname centos-6-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Multiple content hosts with the hostname centos-7-clone1.cesystems.at found, unregistering all except last registered.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.
Content Host d3078602-3cb0-43b7-b326-c3534c8e0e13 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: d3078602-3cb0-43b7-b326-c3534c8e0e13
Removing Candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
Exception when destroying candlepin consumer d3078602-3cb0-43b7-b326-c3534c8e0e13:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"} (DELETE /candlepin/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13): {"displayMessage":"Unit d3078602-3cb0-43b7-b326-c3534c8e0e13 has been deleted","requestUuid":"45c6734d-6a90-466c-858b-b0329f10a416","deletedId":"d3078602-3cb0-43b7-b326-c3534c8e0e13"}
Removing Pulp consumer d3078602-3cb0-43b7-b326-c3534c8e0e13
katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/d3078602-3cb0-43b7-b326-c3534c8e0e13/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json"
 | \n# => 404 NotFound | application/json 528 bytes
 |
Pulp consumer not found for consumer d3078602-3cb0-43b7-b326-c3534c8e0e13 proceeding.
Removing system d3078602-3cb0-43b7-b326-c3534c8e0e13
Content Host ece0be43-0e4d-423a-967c-8d53fa4c99d6 doesn't have candlepin information, unregistering.
Unregistering content host with UUID: ece0be43-0e4d-423a-967c-8d53fa4c99d6
Removing Candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
Exception when destroying candlepin consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6:Katello::Resources::Candlepin::Consumer: 410 Gone {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"} (DELETE /candlepin/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6): {"displayMessage":"Unit ece0be43-0e4d-423a-967c-8d53fa4c99d6 has been deleted","requestUuid":"35253242-d511-4f8f-9064-69501f3f8c76","deletedId":"ece0be43-0e4d-423a-967c-8d53fa4c99d6"}
Removing Pulp consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6
[katello/pulp_rest] [I] RestClient.delete "https://katello.cesystems.at/pulp/api/v2/consumers/ece0be43-0e4d-423a-967c-8d53fa4c99d6/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "accept"=>"application/json", "content_type"=>"application/json"
 | \n# => 404 NotFound | application/json 528 bytes
 |
Pulp consumer not found for consumer ece0be43-0e4d-423a-967c-8d53fa4c99d6 proceeding.
Removing system ece0be43-0e4d-423a-967c-8d53fa4c99d6
Processing content host afe352b7-7aaf-48cc-89c5-ca8e4cbeadab centos-6-genericcloud-ovirt.cesystems.at
Found host with hostname centos-6-genericcloud-ovirt.cesystems.at.
Creating content facet for host centos-6-genericcloud-ovirt.cesystems.at.

*"GUI => Monitor => Tasks"*
Actions::Katello::Host::Destroy
{"host"=>{"id"=>251, "name"=>"centos-6-clone1.cesystems.at"},
 "pool_ids"=>
  ["8a2180484c14ecee014c1503fe820033",
   "8a2180484ca12e87014ca2277340001a",
   "8a2180484c14ecee014c15052fd4003d",
   "8a2180484ca12e87014ca21b43930002"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR:  update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL:  Key (id)=(157) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1   
---
Actions::Katello::Host::Destroy
{"host"=>{"id"=>252, "name"=>"centos-7-clone1.cesystems.at"},
 "pool_ids"=>
  ["8a2180484ca12e87014ca21bdf580009",
   "8a2180484ca12e87014ca22908890023",
   "8a2180484c14ecee014c1503fe820033",
   "8a2180484c14ecee014c15052fd4003d"]}
ActiveRecord::InvalidForeignKey: PG::Error: ERROR:  update or delete on table "katello_systems" violates foreign key constraint "system_host_collections_system_id_fk" on table "katello_system_host_collections"
DETAIL:  Key (id)=(158) is still referenced from table "katello_system_host_collections".
: DELETE FROM "katello_systems" WHERE "katello_systems"."id" = $1   

*"hammer ping"*
candlepin:
    Status:          ok
    Server Response: Duration: 30ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 32ms
pulp:
    Status:          ok
    Server Response: Duration: 35ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 17ms
    
*"foreman-rake db:migrate"*    
== 20150930183738 MigrateContentHosts: migrating ==============================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
and so on..
---
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=d3078602-3cb0-43b7-b326-c3534c8e0e13
May 22 10:00:02 katello pulp: pulp.server.webservices.middleware.exception:INFO: Missing resource(s): consumer=ece0be43-0e4d-423a-967c-8d53fa4c99d6

Comment 1 Justin Sherrill 2016-05-26 13:28:38 UTC
Created from redmine issue http://projects.theforeman.org/issues/15121

Comment 2 Justin Sherrill 2016-05-26 13:49:53 UTC
Steps to reproduce would be:

on 6.1, register two systems with the same hostname (or the same system twice after a sub-man clean).
Associate both systems to a host collection
upgrade to 6.2

Comment 4 Sachin Ghai 2016-06-28 09:00:58 UTC
Verified with upgrade performed from sat6.1.9 -> sat6.2 GA snap17

I created two content hosts with same and added them in a host-collection.

~]# hammer -u admin -p changeme content-host list --organization-id 1 | grep bzhost1
19f78156-1928-495e-855e-1ec74d4d01b2 | bzhost1.satqe.lab.eng.rdu2.redhat.com             | 0                 
1464d7fd-23a3-42f2-97d5-f89dda61685f | bzhost1.satqe.lab.eng.rdu2.redhat.com             | 0     

]# hammer -u admin -p changeme host-collection content-hosts --id 3 --organization-id 1
-------------------------------------|--------------------------------------
ID                                   | NAME                                 
-------------------------------------|--------------------------------------
19f78156-1928-495e-855e-1ec74d4d01b2 | bzhost1.satqe.lab.eng.rdu2.redhat.com
1464d7fd-23a3-42f2-97d5-f89dda61685f | bzhost1.satqe.lab.eng.rdu2.redhat.com
-------------------------------------|--------------------------------------



Later, I performed the upgrade and upgrade was completed successfully.

Comment 5 Sachin Ghai 2016-06-28 09:01:23 UTC
[root@qe-sat6-upgrade-rhel7 ~]# satellite-installer --scenario satellite --upgrade 
Upgrading...
Upgrade Step: stop_services...
Upgrade Step: start_databases...
Upgrade Step: update_http_conf...
Upgrade Step: migrate_pulp...
Upgrade Step: fix_pulp_httpd_conf...
Upgrade Step: start_httpd...
Upgrade Step: start_qpidd...
Upgrade Step: start_pulp...
Upgrade Step: migrate_candlepin...
Upgrade Step: start_tomcat...
Upgrade Step: fix_katello_settings_file...
Upgrade Step: migrate_foreman...
Upgrade Step: remove_nodes_distributors...
Upgrade Step: Running installer...
Installing             Done                                               [100%] [..................................................................]
  The full log is at /var/log/foreman-installer/satellite.log
Upgrade Step: restart_services...
Upgrade Step: db_seed...
Upgrade Step: import_package_groups (this may take a while) ...
Upgrade Step: import_rpms (this may take a while) ...
Upgrade Step: import_distributions (this may take a while) ...
Upgrade Step: import_puppet_modules (this may take a while) ...
Upgrade Step: import_subscriptions (this may take a while) ...
Upgrade Step: elasticsearch_message...
Elasticsearch has been removed as a dependency.  The database files can be removed manually with #rm -rf /var/lib/elasticsearch.  Some packages are no longer needed and can be removed:  #rpm -e ruby193-rubygem-tire tfm-rubygem-tire elasticsearch sigar-java sigar snappy-java lucene4-contrib lucene4
Upgrade Step: remove_docker_v1_content (this may take a while) ...
Upgrade Step: update_puppet_repository_distributors (this may take a while) ...
Upgrade Step: remove_gutterball...
Katello upgrade completed!

Comment 6 Bryan Kearney 2016-07-27 11:20:54 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1501


Note You need to log in before you can comment on or make changes to this bug.