Description of problem: Initiating capsule sync with hammer fail on error (capsule's /var/log/messages): [ERROR][worker-0] pulp.agent.lib.dispatcher:112 - NotFoundException: RequestException: GET request on /pulp/api/v2/consumers/fdf1e572-4f95-43e8-a101-8d5535ef26b2/bindings/RHEVM-QE-INFRA-qe-xtra-art-extra/ failed with 404 - Missing resource(s): consumer_id=fdf1e572-4f95-43e8-a101-8d5535ef26b2 The sync is initiated with hammer on Satellite instance: $ hammer capsule content synchronize --id 10 Hammer reports falsely that the command executed successfully. Steps to Reproduce: 1. unassociate all environments from the capsule 2. on capsule: # for s in {pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do sudo # service $s stop; done # service mongod stop # mv /var/lib/mongodb{,.bak} # mkdir /var/lib/mongodb # restorecon -RvvF /var/lib/mongodb # service mongod start # su - apache -s /bin/bash -c /usr/bin/pulp-manage-db # for s in {qpidd,pulp_celerybeat,pulp_resource_manager,pulp_workers,httpd}; do # sudo service $s restart; done 3. attach the environments to the capsule 4. on Satellite, initiate capsule sync: # hammer capsule content synchronize --id 10 Actual results: [ERROR][worker-0] pulp.agent.lib.dispatcher:112 - NotFoundException: RequestException: GET request on /pulp/api/v2/consumers/fdf1e572-4f95-43e8-a101-8d5535ef26b2/bindings/RHEVM-QE-INFRA-qe-xtra-art-extra/ failed with 404 - Missing resource(s): consumer_id=fdf1e572-4f95-43e8-a101-8d5535ef26b2 Expected results: Successful sync with no errors in Capsule logs Additional info: The steps were done as a workaround for BZ-1288855 as suggested at: https://bugzilla.redhat.com/show_bug.cgi?id=1276911#c17
Created attachment 1114345 [details] Satellite foreman-debug
Created attachment 1114346 [details] Capsule foreman-debug
Hey Alexander, On IRC, i had mentioned to try to re-register the capsule, but I don't see any evidence that happened. Maybe it errored out and you didn't realize or it somehow was registered to a different satellite? The reasons I don't think it re-registered was that specific capsule is registered in the org 'Global_CI' shows a registered date of 11/17/15 9:27 AM (silly US date format) Also I don't see any systems having registered today when i look at the dynflow console. Did it actually re-register properly? -Justin
Hi Justin, You are right, I though it was enough to remove it as a capsule and re-run the capsule-install. Anyway, I've re-register it and the error persists but with new consumer id cd9cd848-a037-41d0-bb30-a790e1743908 Thanks, Alex
Hey Alex, I can't seem to access that satellite anymore via ssh. Did something change with the configuration. Can you login and gather this data (no need to be root): # mongo > use pulp_database > db.consumer_bindings.find({"consumer_id": "cd9cd848-a037-41d0-bb30-a790e1743908"}) Thanks, Justin
Created attachment 1116089 [details] Satellite mongo query result
Hi Justin, Please send me you ssh key and I'll add it to Satellite and the capsule. I've attached the query result. The query is from Satellite mongo as capsule mongo didn't return a thing.
The cause for this was the server_name in server.conf was set as the capsule hostname (for some reason). Changing that back to the satellite's hostname resolved the issue.