Bug 1228165 - [upgrade] Could not find Content Host with exact name <capsule_fqdn> (Katello::Errors::CapsuleContentMissingConsumer): On synchronising contents from upgraded sat6 to upgraded capsule
Summary: [upgrade] Could not find Content Host with exact name <capsule_fqdn> (Katello...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 6
Classification: Red Hat
Component: Upgrades
Version: 6.1.0
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: Unspecified
Assignee: Stephen Benjamin
QA Contact: Sachin Ghai
URL: http://projects.theforeman.org/issues...
Whiteboard:
Depends On:
Blocks: sat61-release-notes 1195450
TreeView+ depends on / blocked
 
Reported: 2015-06-04 10:36 UTC by Sachin Ghai
Modified: 2017-02-23 19:57 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 16:03:00 UTC


Attachments (Terms of Use)
production.logs on syncing capsule contents (68.23 KB, text/plain)
2015-06-04 10:36 UTC, Sachin Ghai
no flags Details
logs from production.log (41.05 KB, text/plain)
2015-07-13 11:22 UTC, Sachin Ghai
no flags Details
CV promotion failed after sat6 upgrade (68.54 KB, image/png)
2015-07-13 12:49 UTC, Sachin Ghai
no flags Details
foreman-debug from rhel71 (619.54 KB, application/x-xz)
2015-07-13 12:57 UTC, Sachin Ghai
no flags Details


Links
System ID Priority Status Summary Last Updated
Foreman Issue Tracker 11123 None None None 2016-04-22 16:56:17 UTC

Description Sachin Ghai 2015-06-04 10:36:14 UTC
Created attachment 1034659 [details]
production.logs on syncing capsule contents

Description of problem:
I was trying upgrade from sat6.0.8 to sat6.1 snap7. I successfully upgraded capsule and satellite server.

But when I ran following command on satellite 6.1 server to re-sync contents on capsule, I got:

[root@sat6-qe-rhel66 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
Could not synchronize capsule content:
  Could not find Content Host with exact name 'sat6-capsule-rhel66.usersys.redhat.com', verify the Capsule is registered with that name.


[root@sat6-qe-rhel66 ~]# hammer -u admin -p changeme capsule list
---|----------------------------------------|----------------------------------------------------
ID | NAME                                   | URL                                                
---|----------------------------------------|----------------------------------------------------
2  | sat6-capsule-rhel66.usersys.redhat.com | https://sat6-capsule-rhel66.usersys.redhat.com:9090
1  | sat6-qe-rhel66.usersys.redhat.com      | https://sat6-qe-rhel66.usersys.redhat.com:9090     
---|----------------------------------------|----------------------------------------------------


[root@sat6-qe-rhel66 ~]# hammer -u admin -p changeme ping
candlepin:      
    Status:          ok
    Server Response: Duration: 21ms
candlepin_auth: 
    Status:          ok
    Server Response: Duration: 28ms
pulp:           
    Status:          ok
    Server Response: Duration: 27ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 29ms
elasticsearch:  
    Status:          ok
    Server Response: Duration: 41ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 0ms


[root@sat6-qe-rhel66 ~]#  hammer -u admin -p changeme organization list
---|----------------------|----------------------|------------
ID | NAME                 | LABEL                | DESCRIPTION
---|----------------------|----------------------|------------
1  | Default_Organization | Default_Organization |            
---|----------------------|----------------------|------------

Version-Release number of selected component (if applicable):
upgrade sat6.0.8 -> sat6.1 snap7

How reproducible:
always

Steps to Reproduce:
1. upgrade sat6.0.8 server to sat6.1 GA snap7
2. upgrade sat6.0.8 capsule to sat6.1 capsule
3. sync contents from sat6.1 server to capsule

Actual results:


Expected results:


Additional info:
I've updated the /etc/hosts file with ip and hostnames.. I can ping capsule node from sat6 via hostname and vice-versa.

Comment 2 Stephen Benjamin 2015-06-04 13:11:59 UTC
It looks like the migration didn't work.  There's this:
   db/migrate/20150423134004_add_content_host_id_to_smart_proxy.rb 

Which is supposed to add the content_host to the smart proxy, and that didn't happen for some reason on your Satellite.

Comment 3 Stephen Benjamin 2015-06-05 20:31:42 UTC
Post-upgrade I didn't encounter this issue, my Capsule had the Content Host associated.

However, deleting the Content Host associated with the Capsule, and later reregistering it will cause this problem, so its something we need to prevent from happening.

Comment 4 Stephen Benjamin 2015-06-10 14:42:55 UTC
I've now done a bunch of upgrades with SNAP 7 through various bug testing and never hit this, so I don't think it's an upgrade issue, or if it was, its been corrected.

The situation could still happen if you unregister the content host that the capsule is attached to.  You can fix it in `foreman-rake console` with this:

User.current = User.anonymous_admin
 
SmartProxy.all.each do |proxy|
  content_host = ::Katello::System.where(:name => proxy.name).order("created_at DESC").first
 
  if content_host
    proxy.content_host_id = content_host.id
    proxy.save!
    puts "Capsule #{proxy} has Content Host #{content_host}"
  else
    puts "No Content Host Found for #{proxy}"
  end
end 


Maybe that could be turned into a rake task.


The permanent solution isn't clear to me, there's a few things we could do. Maybe we should delete the SmartProxy object if a user deletes the relevant ContentHost for a Capsule.

Comment 5 Sachin Ghai 2015-07-13 11:20:46 UTC
This issue is still reproducible on rhel66 with sat6.1.1 snap12.

here is the summary:
========================


While trying upgrade from sat6.0.8 to sat6.1.1 snap12, found a weird issue where content-view promotion ( Library -> dev) failed. Please note that CV publish was successful. I've updated the CV with 6.1.1 capsule and sat-tools repos. So I was trying to generate new version of CV with these repos.

Here is the error I found in production.log:

---
2015-07-13 03:25:17 [E] Could not find Content Host with exact name 'cloud-qe-20.idmqe.lab.eng.bos.redhat.com', verify the Capsule is registered with that name. (Katello::Errors::CapsuleContentMissingConsumer)
--

here 'cloud-qe-20.idmqe.lab.eng.bos.redhat.com' is the node which was configured as external capsule. I skipped the promotion task from dynflow and resume it again.

Later,  I tried capsule-upgrade and it was completed successfully (with workaround[1]). But when I initiated capsule sync, it got failed with same error:

--
[root@cloud-qe-17 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
Could not synchronize capsule content:
  Could not find Content Host with exact name 'cloud-qe-20.idmqe.lab.eng.bos.redhat.com', verify the Capsule is registered with that name.
--

I unregister and re-register the capsule node again and re-initiated sync. this time sync successfully completed.

----
[root@cloud-qe-17 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]
-----

Comment 6 Sachin Ghai 2015-07-13 11:22:54 UTC
Created attachment 1051372 [details]
logs from production.log

Comment 7 Sachin Ghai 2015-07-13 12:49:26 UTC
Created attachment 1051398 [details]
CV promotion failed after sat6 upgrade

Comment 8 Sachin Ghai 2015-07-13 12:54:54 UTC
This issue is reproducible on rhel71 too.

steps performed:
===================

- I installed sat6.0.8 with external capsule and provisioned a content-host via external capsule.

- upgraded sat6.0.8 with sat6.1.1.  (upgrade was completed successfully)
- created custom repos for sat6.1_tools and capsule repo.
- added repos to existing CV and published it
- but when I promoted the new version to next 'dev' env. promotion failed with error:

--
2015-07-13 08:48:02 [E] Could not find Content Host with exact name 'cloud-qe-6.idmqe.lab.eng.bos.redhat.com', verify the Capsule is registered with that name. (Katello::Errors::CapsuleContentMissingConsumer)
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.57/app/lib/katello/capsule_content.rb:48:in `consumer'
/opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.57/app/lib/actions/katello/capsule_content/configure_capsule.rb:19:in `block in plan'
--

Comment 10 Sachin Ghai 2015-07-13 12:57:26 UTC
Created attachment 1051400 [details]
foreman-debug from rhel71

Comment 11 Stephen Benjamin 2015-07-13 13:35:16 UTC
Sachin will open a separate docs BZ for the problem, will leave this one as-is, deleting a content host breaks synchronization.

Comment 14 Sachin Ghai 2015-07-14 12:02:20 UTC
Earlier I was trying satellite6.1 content synchronization (capsule repo and sat-tools repo) after upgrade. Which breaks the content promotion.

As suggested, this time I synced all repos(capsule and sat-tools) before upgrade. Upgrade on Satellite server and external capsule completed successfully.

But when I re-tried the capsule sync after upgrade, I got this:

[root@cloud-qe-12 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
Could not synchronize capsule content:
  Could not find Content Host with exact name 'cloud-qe-15.idmqe.lab.eng.bos.redhat.com', verify the Capsule is registered with that name.

[root@cloud-qe-12 ~]#  hammer -u admin -p changeme capsule list
---|------------------------------------------|------------------------------------------------------
ID | NAME                                     | URL                                                  
---|------------------------------------------|------------------------------------------------------
1  | cloud-qe-12.idmqe.lab.eng.bos.redhat.com | https://cloud-qe-12.idmqe.lab.eng.bos.redhat.com:9090
2  | cloud-qe-15.idmqe.lab.eng.bos.redhat.com | https://cloud-qe-15.idmqe.lab.eng.bos.redhat.com:9090
---|------------------------------------------|------------------------------------------------------
[root@cloud-qe-12 ~]# 

Does this mean we need to re-register the capsule after upgrade ?

Comment 16 Stephen Benjamin 2015-07-15 14:41:13 UTC
Created redmine issue http://projects.theforeman.org/issues/11123 from this bug

Comment 17 Bryan Kearney 2015-07-15 18:04:30 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/11123 has been closed
-------------
Anonymous
Applied in changeset commit:katello|0bc31483f1110205dfa38ce1b0e16ce88dfd0839.

Comment 18 Bryan Kearney 2015-07-20 15:49:25 UTC
commit c0ca95be6ce42a9569010c4d0bc095afb732d89a
Author: Stephen Benjamin <stbenjam@redhat.com>
Date:   Wed Jul 15 10:41:14 2015 -0400

    fixes #11123 - actually assign content_host to capsule

Comment 20 Sachin Ghai 2015-07-23 09:45:57 UTC
Verified with sat6.1.1 GA snap14.

Now I can successfully sync contents to capsule after upgrade.

steps performed:

Upgraded sat server and capsule from  sat6.0.8 -> sat6.1.1 GA snap14
re-run the capsule-sync and it was completed without any error.

[root@cloud-qe-19 ~]# hammer -u admin -p changeme capsule content synchronize --id=2
[............................................................................................................................................] [100%]

Comment 21 Stephen Benjamin 2015-07-27 13:44:03 UTC
A workaround for anyone encountering this until GA:

echo 'User.current = User.anonymous_admin; SmartProxy.all.each { |proxy| proxy.content_host.blank? ? proxy.update_attribute(:content_host, Katello::System.find_by_name(proxy.name)) : nil }' | foreman-rake console

Comment 22 Bryan Kearney 2015-08-12 16:03:00 UTC
This bug was fixed in Satellite 6.1.1 which was delivered on 12 August, 2015.


Note You need to log in before you can comment on or make changes to this bug.