Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1228221

Summary: Capsule Sync failed after Satellite and Capsule upgrade[From 6.0.8 -> 6.1] on RHEL71.
Product: Red Hat Satellite Reporter: Jitendra Yejare <jyejare>
Component: UpgradesAssignee: Stephen Benjamin <stbenjam>
Status: CLOSED DUPLICATE QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.1.0CC: bkearney, jyejare, mmccune, stbenjam
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-06-05 18:29:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Production logs on Satellite and /var/log/messages logs on capsule
none
Satellite Foreman-Debug
none
Capsule Foreman-Debug
none
Katello-service Status logs on Capsule
none
Candlepin errors after capsule sync none

Description Jitendra Yejare 2015-06-04 12:29:42 UTC
Created attachment 1034705 [details]
Production logs on Satellite and /var/log/messages logs on capsule

Description of problem:
Capsule Synchronisation is failing after upgrading Satellite and capsule.

Version-Release number of selected component (if applicable):
Sat 6.1 GA snap 7

How reproducible:

Pre-requisites:
1. Capsule 6.0 registered with Satellite 6.0.8.
2. Capsule 6.0 is synced with Satellite 6.0.8.

Steps to Reproduce:
1. Upgrade Satellite 6.0.8 to Satellite 6.1 with the latest GA snap 7.
2. After finishing satellite upgrade, Upgrade Capsule 6.0 to Capsule 6.1 with the latest GA snap 7.(Sync capsule repo in upgraded Satellite for capsule upgrade)
3. After finishing capsule upgrade, run capsule sync hammer command on Satellite server.
   #hammer -u <user> -p <passwd> capsule content synchronize --id=<capsule_id>
4. Observe foreman/production logs on satellite server and /var/log/messages on capsule.

Actual results:
Capsule sync command executes showing 100%, but /var/log/foreman/production.log shows pulp task error. Also /var/log/messages on capsule clearly shows synchronisation failed. 

Expected results:
Capsule Sync should be successful.

Comment 2 Stephen Benjamin 2015-06-04 17:23:48 UTC
Please always attach a foreman-debug, from both the capsule and satellite.  I really need to see the pulp logs to see what's going on here

Comment 3 Jitendra Yejare 2015-06-05 07:00:05 UTC
Hi Stephen,

Sorry for not adding the foreman-debug logs.

Now, the foreman-debug for Satellite and Capsule is added.

Comment 4 Jitendra Yejare 2015-06-05 07:00:39 UTC
Created attachment 1035041 [details]
Satellite Foreman-Debug

Comment 5 Jitendra Yejare 2015-06-05 07:01:14 UTC
Created attachment 1035042 [details]
Capsule Foreman-Debug

Comment 6 Stephen Benjamin 2015-06-05 13:12:11 UTC
No problem, thanks for the info! Looks like its missing the logs I need though :-(

Can I get the output of `katello-service status` on the capsule?

Thanks!

Comment 7 Jitendra Yejare 2015-06-05 13:30:19 UTC
Created attachment 1035271 [details]
Katello-service Status logs on Capsule

Comment 8 Stephen Benjamin 2015-06-05 18:29:52 UTC
Ah great, thanks! 

Run `capsule-service --upgrade` a second time.

It should work then, and if so, this is a dupe of BZ1228694.

Do reopen if it doesn't fix the problem.

*** This bug has been marked as a duplicate of bug 1228694 ***

Comment 9 Jitendra Yejare 2015-06-08 13:06:38 UTC
Created attachment 1036358 [details]
Candlepin errors after capsule sync

Comment 10 Jitendra Yejare 2015-06-08 13:07:29 UTC
Whether these candlepin errors are harmful ?

Comment 11 Jitendra Yejare 2015-06-08 13:08:45 UTC
Hi Stephen,

As you suggested, I tried capsule upgrade twice and then capsule sync worked well. No stacktrace logs for capsule sync observed.

However, After capsule sync finished to 100%, I have seen some candlepin errors in capsules /var/log/messages.

I am attaching those logs, please have a look. And let if know if anything is needed.

Comment 12 Stephen Benjamin 2015-06-08 13:18:45 UTC
That doesn't look related to the sync at all.

It looks an old client is out there trying to tell Katello about its installed packages via katello-agent, but Katello doesn't recognize it.

Is 1f520c9f-45f5-408d-9434-a48be754994e a UUID for a Content Host you have now?


2015-06-08 08:33:58 [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#enabled_repos as JSON
2015-06-08 08:33:58 [I]   Parameters: {"enabled_repos"=>{"repos"=>[{"baseurl"=>["https://sat6-qe-rhel7.usersys.redhat.com/pulp/repos/Default_Organization/dev/cv_rhel71/custom/sat61_tools/sat61_tools_repo"], "repositoryid"=>"Default_Organization_sat61_tools_sat61_tools_repo"}, {"baseurl"=>["https://sat6-qe-rhel7.usersys.redhat.com/pulp/repos/Default_Organization/dev/cv_rhel71/content/dist/rhel/server/7/7Server/x86_64/os"], "repositoryid"=>"rhel-7-server-rpms"}]}, "id"=>"1f520c9f-45f5-408d-9434-a48be754994e"}
2015-06-08 08:33:58 [E] RestClient::Unauthorized: Katello::Resources::Candlepin::Consumer: 401 Unauthorized {"displayMessage":"Invalid credentials.","requestUuid":"dcb0a29c-30a3-48bf-82a2-01518f02f7f6"} (GET /candlepin/consumers/1f520c9f-45f5-408d-9434-a48be754994e)