Bug 1298107

Summary: Capsule sync fail on missing resource consumer_id
Product: Red Hat Satellite Reporter: Alexander Braverman <abraverm>
Component: PulpAssignee: satellite6-bugs <satellite6-bugs>
Status: CLOSED NOTABUG QA Contact: Katello QA List <katello-qa-list>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.1.3CC: abraverm, jsherril
Target Milestone: Unspecified   
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-24 16:20:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Satellite foreman-debug
none
Capsule foreman-debug
none
Satellite mongo query result none

Description Alexander Braverman 2016-01-13 09:11:59 UTC
Description of problem:
Initiating capsule sync with hammer fail on error (capsule's /var/log/messages):
[ERROR][worker-0] pulp.agent.lib.dispatcher:112 - NotFoundException: RequestException: GET request on /pulp/api/v2/consumers/fdf1e572-4f95-43e8-a101-8d5535ef26b2/bindings/RHEVM-QE-INFRA-qe-xtra-art-extra/ failed with 404 - Missing resource(s): consumer_id=fdf1e572-4f95-43e8-a101-8d5535ef26b2

The sync is initiated with hammer on Satellite instance:
$ hammer capsule content synchronize --id 10
Hammer reports falsely that the command executed successfully. 

Steps to Reproduce:
1. unassociate all environments from the capsule
2. on capsule:
# for s in {pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo # service $s stop; done
# service mongod stop
# mv /var/lib/mongodb{,.bak}
# mkdir /var/lib/mongodb
# restorecon -RvvF /var/lib/mongodb
# service mongod start
# su - apache -s /bin/bash -c /usr/bin/pulp-manage-db 
# for s in {qpidd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do # sudo service $s restart; done
3. attach the environments to the capsule
4. on Satellite, initiate capsule sync:
# hammer capsule content synchronize --id 10

Actual results:
[ERROR][worker-0] pulp.agent.lib.dispatcher:112 - NotFoundException: RequestException: GET request on /pulp/api/v2/consumers/fdf1e572-4f95-43e8-a101-8d5535ef26b2/bindings/RHEVM-QE-INFRA-qe-xtra-art-extra/ failed with 404 - Missing resource(s): consumer_id=fdf1e572-4f95-43e8-a101-8d5535ef26b2

Expected results:
Successful sync with no errors in Capsule logs 

Additional info:
The steps were done as a workaround for BZ-1288855 as suggested at:
https://bugzilla.redhat.com/show_bug.cgi?id=1276911#c17

Comment 1 Alexander Braverman 2016-01-13 09:23:37 UTC
Created attachment 1114345 [details]
Satellite foreman-debug

Comment 2 Alexander Braverman 2016-01-13 09:28:37 UTC
Created attachment 1114346 [details]
Capsule foreman-debug

Comment 3 Justin Sherrill 2016-01-13 17:11:41 UTC
Hey Alexander, 

On IRC, i had mentioned to try to re-register the capsule, but I don't see any evidence that happened.  Maybe it errored out and you didn't realize or it somehow was registered to a different satellite?  

The reasons I don't think it re-registered was that specific capsule is registered in the org 'Global_CI' shows a registered date of 11/17/15 9:27 AM (silly US date format)

Also I don't see any systems having registered today when i look at the dynflow console.  Did it actually re-register properly?

-Justin

Comment 4 Alexander Braverman 2016-01-14 09:37:22 UTC
Hi Justin,

You are right,  I though it was enough to remove it as a capsule and re-run the capsule-install. Anyway, I've re-register it and the error persists but with new consumer id cd9cd848-a037-41d0-bb30-a790e1743908

Thanks,
Alex

Comment 5 Justin Sherrill 2016-01-19 02:16:09 UTC
Hey Alex,

I can't seem to access that satellite anymore via ssh.  Did something change with the configuration.

Can you login and gather this data (no need to be root):

# mongo
> use pulp_database
> db.consumer_bindings.find({"consumer_id": "cd9cd848-a037-41d0-bb30-a790e1743908"})

Thanks,
Justin

Comment 6 Alexander Braverman 2016-01-19 08:52:16 UTC
Created attachment 1116089 [details]
Satellite mongo query result

Comment 7 Alexander Braverman 2016-01-19 08:52:49 UTC
Hi Justin,
Please send me you ssh key and I'll add it to Satellite and the capsule.
I've attached the query result. The query is from Satellite mongo as capsule mongo didn't return a thing.

Comment 10 Justin Sherrill 2016-01-24 16:20:32 UTC
The cause for this was the server_name in server.conf was set as the capsule hostname (for some reason).   Changing that back to the satellite's hostname resolved the issue.