Description of problem: Fresh default installation of pulp-server acting as a node to another default pulp-server both RHEL 7. Followed instructions: http://pulp-user-guide.readthedocs.org/en/latest/installation.html http://pulp-user-guide.readthedocs.org/en/latest/nodes.html Default certs and verify_ssl=False Consumer Client works on the node server and can use yum. pulp-admin and pulp-consumer works on both servers. pulp-admin node sync run --node-id repo fails every time, tried every thing including fresh installations with same result. Also tried pulp-consumer update --keys and restarting goferd - same results. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Install two RHEL 7 Servers with default installation of pulp-server and create a repo on one. 2. Configure one as a node to the other using current documentation and try to sync the repo. 3. Actual results: Node sync failure Expected results: Node sync success Additional info: Log from messages: Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred: AuthenticationFailed Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last): Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__ Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__ Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 279, in wsgi Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.handle_with_processors() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return handler() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 28, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 422, in _delegate Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return f() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 430, in <lambda> Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: f = lambda: self._delegate_sub_application(pat, what) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 455, in _delegate_sub_application Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return app.handle_with_processors() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return handler() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 28, in process Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 420, in _delegate Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return handle_class(cls) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/web/application.py", line 396, in handle_class Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: return tocall(*args) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 192, in _auth_decorator Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: userid, is_consumer = authenticate_user() Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 111, in oauth_authentication Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT) Sep 15 10:42:05 repo pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 88, in __init__ self.importers = self._importers() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET(path) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed Sep 15 10:42:05 repo goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None Sep 15 10:42:05 repo goferd: [INFO][worker-0] gofer.agent.rmi:149 - sn=7b3315f2-ec18-4ced-be8b-2d714ae0b73b processed in: 114 (ms)
Does this error still occur after restarting httpd?
Yes, restarting httpd on server and/or child node have no effect. Restarting goferd on child node have no effect either. Firewalls turned of and SELinux on permissive - no change. I have now made a completely new test with new installs of parent and child with the same result - node sync does not work.
Here is some results of node sync After doing all the config changes and restarting services, I did get the same error. [root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1 This command may be exited via ctrl+c without affecting the request. Error occurred during synchronization, check the child node logs for details +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Pending Content Sources: Downloads: Id: zoo Units: Added: 0 Removed: 0 Updated: 0 +----------------------------------------------------------------------+ The following [1] errors were reported +----------------------------------------------------------------------+ - 01: An unexpected error occurred. repo_id=None [root@qe-blade-05 ~]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] Restarting httpd on server did not seem to fix it. [root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1 This command may be exited via ctrl+c without affecting the request. Error occurred during synchronization, check the child node logs for details +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Pending Content Sources: Downloads: Id: zoo Units: Added: 0 Removed: 0 Updated: 0 +----------------------------------------------------------------------+ The following [1] errors were reported +----------------------------------------------------------------------+ - 01: An unexpected error occurred. repo_id=None Now I ran the following on the node child [root@mgmt12 ~]# service goferd restart Stopping goferd[ OK ] Starting goferd[ OK ] That did not fix the issue Then I restarted httpd on the child and that fixed the issue and got the sync going [root@mgmt12 ~]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [root@qe-blade-05 ~]# pulp-admin node sync run --node-id node1 This command may be exited via ctrl+c without affecting the request. (1/1) Repository: zoo [==================================================] 100% Synchronization succeeded +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Added Content Sources: Downloads: Source Id: Parent Node Total Failed: 0 Total Succeeded: 32 Total Passes: 1 Total Sources: 0 Id: zoo Units: Added: 39 Removed: 0 Updated: 0 [root@qe-blade-05 ~]#
Unfortunately that solution did not work for me, i still get the same failed result. tried multiple times to restart services in different order, reboots... no change. I have now tested this on two new VM'w with CentOS 7 and get the same results. On the CentOS VM's pulp-server-2.4.2-1.el7 Logs from the new CentOS VM'S: Server logs: Sep 24 21:09:14 pulp pulp: pulp.server.agent.direct.services:INFO: {'reboot': {'scheduled': False, 'details': {}}, 'details': {'node': {'details': {'errors': [{'error_id': 'exception', 'details': {'message': 'RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed', 'repo_id': None}}], 'repositories': [{'action': 'pending', 'sources': {}, 'repo_id': 'rhel-7-pulp', 'units': {'removed': 0, 'added': 0, 'updated': 0}}]}, 'succeeded': False}}, 'succeeded': False, 'num_changes': 0} Child Node logs: Sep 24 21:09:14 pulp2 pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT) Sep 24 21:09:14 pulp2 pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 88, in __init__ self.importers = self._importers() File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET(path) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed Sep 24 21:09:14 pulp2 goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None
I'm having the same issue, fresh child node install: Verified oauth is working Verify ssl is off in nodes.conf ]# cat /etc/pulp/nodes.conf |grep veri verify_ssl: false restarting httpd and goferd... unregistered, registered, activated client, and then bound to single repo: pulp-admin node list +----------------------------------------------------------------------+ Child Nodes +----------------------------------------------------------------------+ Id: node-1 Display Name: node-1 Description: None Bindings: Additive: pulp Update Strategy: additive Notes: pulp-admin node sync run --node-id node-1 This command may be exited via ctrl+c without affecting the request. Error occurred during synchronization, check the child node logs for details +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Pending Content Sources: Downloads: Id: pulp Units: Added: 0 Removed: 0 Updated: 0 +----------------------------------------------------------------------+ The following [1] errors were reported +----------------------------------------------------------------------+ - 01: An unexpected error occurred. repo_id=None Logs: Parent => /var/log/messages <== 2014-09-25T14:50:31.600352+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: Task RMI (succeeded) 2014-09-25T14:50:31.600462+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: Succeeded 2014-09-25T14:50:31.600528+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: sn : 4a6a36e0-4c83-402f-a36f-0b7c16a2c42d 2014-09-25T14:50:31.600680+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: origin : dd65c874-a4b0-42ca-8741-8d703323bae8 2014-09-25T14:50:31.600908+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: user data : {'task_id': 'd9063314-51e6-4b1a-8b60-9c16b74655b0', 'consumer_id': 'vs-prod-plp01.mgmt.prod.uc'} 2014-09-25T14:50:31.600983+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: retval: 2014-09-25T14:50:31.601055+01:00 pulp pulp: pulp.server.agent.direct.services:INFO: {'reboot': {'scheduled': False, 'details': {}}, 'details': {'node': {'details': {'errors': [{'error_id': 'exception', 'details': {'message': 'RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed', 'repo_id': None}}], 'repositories': [{'action': 'pending', 'sources': {}, 'repo_id': 'pulp', 'units': {'removed': 0, 'added': 0, 'updated': 0}}]}, 'succeeded': False}}, 'succeeded': False, 'num_changes': 0} Child ==> /var/log/messages <== 2014-09-25T14:53:08.609463+01:00 fake_servername goferd: [INFO][worker-0] gofer.rmi.dispatcher:612 - call: Content.update() sn=af9652fa-b4f3-4c2d-a464-d2891b25ad56 info={'task_id': 'fcf032d2-b213-4169-916c-80ac63dad44a', 'consumer_id': 'fake_servername.mgmt.prod.uc'} 2014-09-25T14:53:08.732310+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred: AuthenticationFailed 2014-09-25T14:53:08.732536+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last): 2014-09-25T14:53:08.732899+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__ 2014-09-25T14:53:08.733194+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response) 2014-09-25T14:53:08.733484+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 42, in __call__ 2014-09-25T14:53:08.733774+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.app(environ, start_response) 2014-09-25T14:53:08.734120+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi 2014-09-25T14:53:08.734409+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = self.handle_with_processors() 2014-09-25T14:53:08.734693+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors 2014-09-25T14:53:08.735004+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors) 2014-09-25T14:53:08.735243+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process 2014-09-25T14:53:08.735543+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.735784+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor 2014-09-25T14:53:08.736058+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handler() 2014-09-25T14:53:08.736297+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> 2014-09-25T14:53:08.736532+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.736766+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process 2014-09-25T14:53:08.737040+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.737278+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor 2014-09-25T14:53:08.737514+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler() 2014-09-25T14:53:08.737747+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> 2014-09-25T14:53:08.738022+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.738259+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process 2014-09-25T14:53:08.738494+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle() 2014-09-25T14:53:08.738726+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle 2014-09-25T14:53:08.739005+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args) 2014-09-25T14:53:08.739242+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate 2014-09-25T14:53:08.739476+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return f() 2014-09-25T14:53:08.739711+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda> 2014-09-25T14:53:08.739990+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: f = lambda: self._delegate_sub_application(pat, what) 2014-09-25T14:53:08.740229+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application 2014-09-25T14:53:08.740464+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return app.handle_with_processors() 2014-09-25T14:53:08.740700+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 29, in _handle_with_processors 2014-09-25T14:53:08.740978+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return process(self.processors) 2014-09-25T14:53:08.741216+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process 2014-09-25T14:53:08.741451+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.741685+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor 2014-09-25T14:53:08.741960+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handler() 2014-09-25T14:53:08.742197+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> 2014-09-25T14:53:08.742430+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.742664+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in process 2014-09-25T14:53:08.742941+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.743177+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor 2014-09-25T14:53:08.743412+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: result = handler() 2014-09-25T14:53:08.743647+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in <lambda> 2014-09-25T14:53:08.743920+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return p(lambda: process(processors)) 2014-09-25T14:53:08.744155+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 28, in process 2014-09-25T14:53:08.744390+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self.handle() 2014-09-25T14:53:08.744624+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle 2014-09-25T14:53:08.744895+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return self._delegate(fn, self.fvars, args) 2014-09-25T14:53:08.745132+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate 2014-09-25T14:53:08.745368+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return handle_class(cls) 2014-09-25T14:53:08.745603+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class 2014-09-25T14:53:08.745871+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: return tocall(*args) 2014-09-25T14:53:08.746097+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 203, in _auth_decorator 2014-09-25T14:53:08.746335+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: raise AuthenticationFailed(auth_utils.CODE_FAILED) 2014-09-25T14:53:08.746570+01:00 fake_servername pulp: pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: Pulp exception occurred: AuthenticationFailed 2014-09-25T14:53:08.750747+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - synchronization failed 2014-09-25T14:53:08.750816+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.strategies:129 - Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/handlers/strategies.py", line 116, in synchronize validator.validate(request.bindings) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 70, in _validate_plugins child = ChildServer() File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 88, in __init__ self.importers = self._importers() File "/usr/lib/python2.6/site-packages/pulp_node/handlers/validation.py", line 99, in _importers http = bindings.server_info.get_importers() File "/usr/lib/python2.6/site-packages/pulp/bindings/server_info.py", line 43, in get_importers return self.server.GET(path) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 93, in GET return self._request('GET', path, queries) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 192, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: AuthenticationFailed 2014-09-25T14:53:08.763471+01:00 fake_servername goferd: [ERROR][worker-0] pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None 2014-09-25T14:53:08.764569+01:00 fake_servername goferd: [INFO][worker-0] gofer.agent.rmi:149 - sn=af9652fa-b4f3-4c2d-a464-d2891b25ad56 processed in: 176 (ms)
(In reply to Johan from comment #0) > Description of problem: > Fresh default installation of pulp-server acting as a node to another > default pulp-server both RHEL 7. > Followed instructions: > http://pulp-user-guide.readthedocs.org/en/latest/installation.html > http://pulp-user-guide.readthedocs.org/en/latest/nodes.html > > Default certs and verify_ssl=False > > Consumer Client works on the node server and can use yum. > pulp-admin and pulp-consumer works on both servers. > > pulp-admin node sync run --node-id repo fails every time, tried every thing > including fresh installations with same result. > > Also tried pulp-consumer update --keys and restarting goferd - same results. > > Version-Release number of selected component (if applicable): > > > How reproducible: > > Always > > Steps to Reproduce: > 1. Install two RHEL 7 Servers with default installation of pulp-server and > create a repo on one. > 2. Configure one as a node to the other using current documentation and try > to sync the repo. > 3. > > Actual results: > > Node sync failure > > Expected results: > > Node sync success > > Additional info: > > Log from messages: > > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: Pulp exception occurred: > AuthenticationFailed > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent > call last): > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/ > exception.py", line 44, in __call__ > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > self.app(environ, start_response) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/ > postponed.py", line 42, in __call__ > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > self.app(environ, start_response) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 279, in wsgi > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: result = > self.handle_with_processors() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 29, in _handle_with_processors > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > process(self.processors) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return handler() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in <lambda> > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: result = handler() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in <lambda> > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 28, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return self.handle() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > self._delegate(fn, self.fvars, args) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 422, in _delegate > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return f() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 430, in <lambda> > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: f = lambda: > self._delegate_sub_application(pat, what) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 455, in > _delegate_sub_application > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > app.handle_with_processors() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 29, in _handle_with_processors > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > process(self.processors) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return handler() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in <lambda> > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: result = handler() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 26, in <lambda> > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return p(lambda: > process(processors)) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", > line 28, in process > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return self.handle() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > self._delegate(fn, self.fvars, args) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 420, in _delegate > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return > handle_class(cls) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/web/application.py", line 396, in > handle_class > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: return tocall(*args) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/ > decorators.py", line 192, in _auth_decorator > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: userid, is_consumer > = authenticate_user() > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: File > "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/ > decorators.py", line 111, in oauth_authentication > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: raise > AuthenticationFailed(auth_utils.CODE_INVALID_SSL_CERT) > Sep 15 10:42:05 repo pulp: > pulp.server.webservices.middleware.exception:ERROR: AuthenticationFailed: > Pulp exception occurred: AuthenticationFailed > Sep 15 10:42:05 repo goferd: [ERROR][worker-0] > pulp_node.handlers.strategies:129 - synchronization failed > Sep 15 10:42:05 repo goferd: [ERROR][worker-0] > pulp_node.handlers.strategies:129 - Traceback (most recent call last): File > "/usr/lib/python2.7/site-packages/pulp_node/handlers/strategies.py", line > 116, in synchronize validator.validate(request.bindings) File > "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line > 54, in validate self.report.errors.extend(self._validate_plugins(bindings)) > File "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", > line 70, in _validate_plugins child = ChildServer() File > "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line > 88, in __init__ self.importers = self._importers() File > "/usr/lib/python2.7/site-packages/pulp_node/handlers/validation.py", line > 99, in _importers http = bindings.server_info.get_importers() File > "/usr/lib/python2.7/site-packages/pulp/bindings/server_info.py", line 43, in > get_importers return self.server.GET(path) File > "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 93, in GET > return self._request('GET', path, queries) File > "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 151, in > _request self._handle_exceptions(response_code, response_body) File > "/usr/lib/python2.7/site-packages/pulp/bindings/server.py", line 192, in > _handle_exceptions raise code_class_mappings[response_code](response_body) > PermissionsException: RequestException: GET request on > /pulp/api/v2/plugins/importers/ failed with 401 - Pulp exception occurred: > AuthenticationFailed > Sep 15 10:42:05 repo goferd: [ERROR][worker-0] > pulp_node.handlers.handler:111 - An unexpected error occurred. repo_id=None > Sep 15 10:42:05 repo goferd: [INFO][worker-0] gofer.agent.rmi:149 - > sn=7b3315f2-ec18-4ced-be8b-2d714ae0b73b processed in: 114 (ms) Hello, I have exactly the same problem with a child on RHEL7 and a parent on RHEL6. I'm using pulp 2.5, the latest version. I tried a lot of settings, as you. Have you find a solution or a new direction to search ? Thank you.
Has there been any progress on what is causing this? I've hit the same issue with parent and child on CentOS6, pulp 2.5.
Moved to https://pulp.plan.io/issues/528