Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1330229

Summary: Capsule Sync breaks on rhel6
Product: Red Hat Satellite Reporter: Kedar Bidarkar <kbidarka>
Component: Foreman ProxyAssignee: John Mitsch <jomitsch>
Status: CLOSED NOTABUG QA Contact: Katello QA List <katello-qa-list>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.0CC: bbuckingham, xdmoon
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-04-28 13:47:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
capsule sync breaks
none
capsule foreman-debug
none
satellite6 foreman-debug none

Description Kedar Bidarkar 2016-04-25 16:45:08 UTC
Description of problem:

Capsule Sync breaks on rhel6 OS, with the message "500 Internal Server Error".

2016-04-25 12:34:59 [app] [E] RestClient::InternalServerError: 500 Internal Server Error
 | Body: {"http_request_method": "GET", "exception": ["ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused\n"], "error_message": "localhost:27017: [Errno 111] Connection refused", "_href": "/pulp/api/v2/repositories/Default_Organization-Dev-rhel6_capsule_cv-rhel6_capsule-rhel6_capsule_repo/?details=true", "http_status": 500, "traceback": ["  File \"/usr/lib/python2.6/site-packages/django/core/handlers/base.py\", line 112, in get_response\n    response = wrapped_callback(request, *callback_args, **callback_kwargs)\n", "  File \"/usr/lib/python2.6/site-packages/django/views/generic/base.py\", line 69, in view\n    return self.dispatch(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.6/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n", "  File \"/usr/lib/python2.6/site-packages/pulp/server/webservices/views/decorators.py\", line 241, in _auth_decorator\n    return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)\n", "  File \"/usr/lib/python2.6/site-packages/pulp/server/webservices/views/decorators.py\", line 154, in _verify_auth\n    login = authenticate_user()\n", "  File \"/usr/lib/python2.6/site-packages/pulp/server/webservices/views/decorators.py\", line 30, in check_preauthenticated\n    userid = factory.authentication_manager().check_username_password(username)\n", "  File \"/usr/lib/python2.6/site-packages/pulp/server/managers/auth/authentication.py\", line 95, in check_username_password\n    user = self._check_username_password_local(username, password)\n", "  File \"/usr/lib/python2.6/site-packages/pulp/server/managers/auth/authentication.py\", line 35, in _check_username_password_local\n    user = model.User.objects(login=username).first()\n", "  File \"/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py\", line 264, in first\n    result = queryset[0]\n", "  File \"/usr/lib/python2.6/site-packages/mongoengine/queryset/base.py\", line 161, in __getitem__\n    return queryset._document._from_son(queryset._cursor[key],\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/cursor.py\", line 565, in __getitem__\n    for doc in clone:\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/cursor.py\", line 1097, in next\n    if len(self.__data) or self._refresh():\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/cursor.py\", line 1019, in _refresh\n    self.__read_concern))\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/cursor.py\", line 850, in __send_message\n    **kwargs)\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/mongo_client.py\", line 777, in _send_message_with_response\n    server = topology.select_server(selector)\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/topology.py\", line 141, in select_server\n    address))\n", "  File \"/usr/lib64/python2.6/site-packages/pymongo/topology.py\", line 117, in select_servers\n    self._error_message(selector))\n"]}
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-1.7.0/lib/runcible/base.rb:79:in `block in get_response'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-1.6.7/lib/restclient/request.rb:228:in `process_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rbovirt-0.0.36/lib/restclient_ext/request.rb:50:in `block in transmit'




Version-Release number of selected component (if applicable):
sat62-snap9

How reproducible:


Steps to Reproduce:
1. Add lifecycle-env to the capsule and sync 
2.
3.

Actual results:
Capsule Sync fails

Expected results:
Capsule Sunc should be successful

Additional info:

Comment 1 Kedar Bidarkar 2016-04-25 17:02:14 UTC
Created attachment 1150558 [details]
capsule sync breaks

Comment 2 Kedar Bidarkar 2016-04-25 17:07:29 UTC
Please Note: A workaround was added for https://bugzilla.redhat.com/show_bug.cgi?id=1329640 on the satellite6 for the sync to work.

Comment 4 John Mitsch 2016-04-27 20:21:58 UTC
After updating the packages on the capsule, migrating pulp, and restarting pulp, I was able to sync the repositories onto said capsule from a rhel6 satellite. 

I ran into one bug with the rhel6 kick start repo, but that has been filed here https://bugzilla.redhat.com/show_bug.cgi?id=1329640

Can we move this to VERIFIED?

Comment 5 Kedar Bidarkar 2016-04-28 10:19:42 UTC
Was the workaround for rhel6-ks applied and capsule synced successfully.

If that is so then we can close this bug.


Otherwise currently the issue seen on my setup after the rhel6-ks cdn repo sync thing is this.


Apr 28 06:04:32 qe-capsule-feature-rhel6 pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[683f96ff-01d1-4780-ae4a-45a1db8bce15]
Apr 28 06:04:32 qe-capsule-feature-rhel6 pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[1c12caa5-bce7-4886-b566-49ba941a07b7] succeeded in 2.134045634s: None
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728) Exception caught from plugin during publish for repo [Default_Organization-Dev-rhel6_cv-Red_Hat_Enterprise_Linux_Server-Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_7]
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728) Traceback (most recent call last):
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/server/controllers/repository.py", line 1023, in _do_publish
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 658, in wrap_f
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     return f(*args, **kwargs)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 174, in publish_repo
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     return self._publisher.process_lifecycle()
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 562, in process_lifecycle
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     super(PluginStep, self).process_lifecycle()
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 159, in process_lifecycle
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     step.process()
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 232, in process
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     self._process_block(item=item)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp/plugins/util/publish_step.py", line 291, in _process_block
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     self.process_main(item=item)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 748, in process_main
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     self._publish_distribution_files(unit)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 816, in _publish_distribution_files
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     self._write_pulp_distribution_file(distro_files, xml_file_path)
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)   File "/usr/lib/python2.6/site-packages/pulp_rpm/plugins/distributors/yum/publish.py", line 835, in _write_pulp_distribution_file
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728)     old_files = [old_element.text for old_element in old_xml_root.iter('file')]
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.controllers.repository:ERROR: (4272-37728) AttributeError: iter
Apr 28 06:04:33 qe-capsule-feature-rhel6 pulp: pulp.server.async.tasks:INFO: Task failed : [33bc9dbd-7a2b-4a2e-900f-5e25b898c40a]

Comment 6 Kedar Bidarkar 2016-04-28 10:29:41 UTC
Created attachment 1151802 [details]
capsule foreman-debug

Comment 7 Kedar Bidarkar 2016-04-28 10:48:57 UTC
Created attachment 1151806 [details]
satellite6 foreman-debug

Comment 8 John Mitsch 2016-04-28 12:22:46 UTC
(In reply to Kedar Bidarkar from comment #5)

> pulp.server.controllers.repository:ERROR: (4272-37728) Exception caught from
> plugin during publish for repo
> [Default_Organization-Dev-rhel6_cv-Red_Hat_Enterprise_Linux_Server-
> Red_Hat_Enterprise_Linux_6_Server_Kickstart_x86_64_6_7]

Kedar,

It looks like this error has to do with the rhel6 kickstart repo again, I think that is again related to 1329640 bug.

-John

Comment 9 Kedar Bidarkar 2016-04-28 13:00:46 UTC
Looks like Capsule sync fails for rhel6-ks repo. 

Also a workaround was applied for rhel6-ks sync failure in satellite6 as per mentioned in comment 2.

Discussed the same with john Mitsch and it looks like this issue is seen due to the bug https://bugzilla.redhat.com/show_bug.cgi?id=1329640.
So, Will close this bug and will wait for the bug https://bugzilla.redhat.com/show_bug.cgi?id=1329640 to get fixed for testing the capsule syncs with rhel6-ks.

In short: Capsule sync fails for rhel6 if rhel6-ks repo is to be synced to capsule and we need this to be tested again after https://bugzilla.redhat.com/show_bug.cgi?id=1329640 is fixed.


If this issue is seen again after https://bugzilla.redhat.com/show_bug.cgi?id=1329640   is fixed, this bug will be re-opened.