Hide Forgot
Description of problem: After upgrading pulp 2.2 -> 2.3, node sync fails. Version-Release number of selected component (if applicable): 2.3.0 How reproducible: Only if upgraded child node has done a previous sync of the same repository. Steps to Reproduce: 1. install a pulp nodes 2.2 parent and child. 2. activate the child and bind it to a repository. 3. perform a node sync. 4. upgrade both parent and child nodes. 5. run the node sync. Actual results: pulp.log on the child node: KeyError: 'errors' 2013-10-23 11:04:27,887 - ERROR - Client-side exception occurred Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/client/extensions/core.py", line 478, in run exit_code = Cli.run(self, args) File "/usr/lib/python2.6/site-packages/okaara/cli.py", line 974, in run exit_code = command_or_section.execute(self.prompt, remaining_args) File "/usr/lib/python2.6/site-packages/pulp/client/extensions/extensions.py", line 224, in execute return self.method(*arg_list, **clean_kwargs) File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/commands.py", line 515, in run self.poll([task], kwargs) File "/usr/lib/python2.6/site-packages/pulp/client/commands/polling.py", line 143, in poll self.succeeded(task) File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/commands.py", line 530, in succeeded r = UpdateRenderer(self.context.prompt, report) File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/rendering.py", line 168, in __init__ self.errors = self.details['errors'] KeyError: 'errors' Expected results: manifest is migrated and all is well. Additional info:
The trace in the description is incorrect. Here is the correct one: Child pulp.log: 2013-10-23 10:33:15,701 pulp.server.dispatch.task:INFO: SUCCESS: Task dd7663f4-bfda-44cb-9267-82eeb7878b97: CallRequest: RepoImporterManager.update_importer_config(u'follett-el5-x86_64', importer_config=****) 2013-10-23 10:33:16,463 pulp_node.importers.strategies:ERROR: follett-el5-x86_64 Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory manifest.read(migrate) File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read migration(self.path) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 55, in migrate version_in = manifest[VERSION] KeyError: 'version' 2013-10-23 10:33:16,475 pulp.server.dispatch.task:INFO: SUCCESS: Task 9e6c0f04-f7da-4231-b49d-58528189cd9f: CallRequest: RepoSyncManager.sync(u'follett-el5-x86_64', sync_config_override={}) 2013-10-23 10:33:18,372 pulp.server.dispatch.task:INFO: SUCCESS: Task b229ffdc-647b-4a4a-814b-52ed1e90143d: CallRequest: RepoImporterManager.update_importer_config(u'follett-el6-x86_64', importer_config=****) +----------------------------------------------------------------------+ The following [10] errors were reported +----------------------------------------------------------------------+ - 01: An error occurred while downloading units from the parent for repository [follett-el5-x86_64. The cause may be that the repository has not been published
Example 2.2 manifest.json: # pwd /var/lib/pulp/working/repos/NetEnt-rhel-6-4-server-x86_64-os/importers/nodes_http_importer [root@ind-pulp-01 nodes_http_importer]# cat manifest.json { "total_units": 12565, "publishing_details": { "base_url": "https://cyd-pulp-01.nix.cydmodule.com/pulp/nodes/https/repos/NetEnt-rhel-6-4-server-x86_64-os" }, "id": "c85a8d67-5edb-46f7-8d2a-458fc29858e4", "units_path": "/var/www/pulp/nodes/https/repos/tmp0HZaF4/units.json.gz"
https://github.com/pulp/pulp/pull/696
build: 2.3.0-0.27.beta
looks like this still fails fail-qa 2013-11-05 15:42:48,877 pulp_node.importers.strategies:ERROR: iso-repo Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory manifest.read(migrate) File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read migration(self.path) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate manifest = migration(manifest) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1 manifest.pop(UNITS_SIZE) KeyError: 'units_size' 2013-11-05 15:42:49,413 pulp.plugins.file.distributor:INFO: Beginning publish for repository <iso-repo> 2013-11-05 15:42:50,483 pulp_node.importers.strategies:ERROR: puppet-repo Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory manifest.read(migrate) File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read migration(self.path) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate manifest = migration(manifest) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1 manifest.pop(UNITS_SIZE) KeyError: 'units_size' 2013-11-05 15:42:50,978 pulp_puppet.plugins.distributors.publish:INFO: Beginning publish for repository <puppet-repo> 2013-11-05 15:42:50,979 pulp_puppet.plugins.distributors.publish:INFO: Initializing build directory for repository <puppet-repo> 2013-11-05 15:42:51,002 pulp_puppet.plugins.distributors.publish:INFO: Creating symlinks for modules in repository <puppet-repo> 2013-11-05 15:42:51,014 pulp_puppet.plugins.distributors.publish:INFO: Generating metadata for repository <puppet-repo> 2013-11-05 15:42:51,051 pulp_puppet.plugins.distributors.publish:INFO: Making newly built repository live for repository <puppet-repo> 2013-11-05 15:42:51,080 pulp_puppet.plugins.distributors.publish:INFO: Cleaning up build directory for repository <puppet-repo> 2013-11-05 15:42:52,193 pulp_node.importers.strategies:ERROR: zoo Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory manifest.read(migrate) File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read migration(self.path) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate manifest = migration(manifest) File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1 manifest.pop(UNITS_SIZE) KeyError: 'units_size'
https://github.com/pulp/pulp/pull/703
build: 2.3.0-0.29.beta
fails-qa [root@pulp-v2-server ~]# rpm -qa |grep pulp-server pulp-server-2.3.0-0.29.beta.el6.noarch [root@pulp-v2-server ~]# node sync seems to be failing after upgrade 2013-11-07 15:45:26,755 [ERROR][worker-0] update() @ dispatcher.py:112 - handler failed Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/agent/lib/dispatcher.py", line 108, in update report = handler.update(conduit, units, dict(options)) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/handler.py", line 88, in update bindings = RepositoryBinding.fetch_all(pulp_bindings, conduit.consumer_id) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/model.py", line 511, in fetch_all http = bindings.bind.find_by_id(node_id) File "/usr/lib/python2.6/site-packages/pulp/bindings/consumer.py", line 154, in find_by_id return self.server.GET(path) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 96, in GET return self._request('GET', path, queries) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 154, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 195, in _handle_exceptions raise code_class_mappings[response_code](response_body) PermissionsException: RequestException: GET request on /pulp/api/v2/consumers/node1/bindings/ failed with 401 - Pulp exception occurred: AuthenticationFailed 2013-11-07 15:45:26,759 [INFO][worker-0] sendreply() @ rmi.py:172 - 8cd5dc15-e35e-4bfd-90c5-1b895bb2d5e0 processed in: 106 (ms) :^C [root@pulp-node2 ~]# tail -f /var/log/pulp/pulp.log 2013-11-07 15:43:54,815 pulp.server.webservices.application:INFO: ************************************************************* 2013-11-07 15:43:54,815 pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized 2013-11-07 15:43:54,817 pulp.server.webservices.application:INFO: ************************************************************* 2013-11-07 15:43:54,830 gofer.messaging.broker:INFO: connecting: {pulp-node2.usersys.redhat.com:5672}: transport=TCP host=pulp-node2.usersys.redhat.com port=5672 cacert=/etc/pki/qpid/ca/ca.crt clientcert=/etc/pki/qpid/client/client.pem 2013-11-07 17:00:16,038 pulp.server.webservices.middleware.exception:ERROR: Missing resource(s): repo=SAM Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__ return self.app(environ, start_response) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__ return self.app(environ, start_response) File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi result = self.handle_with_processors() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors return process(self.processors) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor return handler() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor result = handler() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process return self.handle() File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle return self._delegate(fn, self.fvars, args) File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate return f() File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda> f = lambda: self._delegate_sub_application(pat, what) File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application return app.handle_with_processors() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors return process(self.processors) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor return handler() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor result = handler() File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda> return p(lambda : process(processors)) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process return self.handle() File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle return self._delegate(fn, self.fvars, args) File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate return handle_class(cls) File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class return tocall(*args) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator value = method(self, *args, **kwargs) File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 245, in GET raise exceptions.MissingResource(repo=id) MissingResource: Missing resource(s): repo=SAM 2013-11-07 17:00:17,738 pulp_node.importers.strategies:ERROR: SAM Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 127, in synchronize self._synchronize(request) File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 392, in _synchronize unit_inventory = self._unit_inventory(request) File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 212, in _unit_inventory parent_units = manifest.get_units() File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 145, in get_units path = self.units_path() TypeError: 'NoneType' object is not callable
https://github.com/pulp/pulp/pull/710
build: 2.3.0-0.31.beta
verified [root@pulp-v2-server ~]# rpm -qa pulp-server pulp-server-2.3.0-0.33.beta.el6.noarch [root@pulp-v2-server ~]# [root@pulp-v2-server ~]# pulp-admin node sync run --node-id node1 This command may be exited via ctrl+c without affecting the request. [-] Waiting to begin... (1/4) Repository: iso-repo [==================================================] 100% (2/4) Repository: live-rhel-6-server-x86_64-pulp [==================================================] 100% (3/4) Repository: puppet-repo [==================================================] 100% (4/4) Repository: test-repo1 [==================================================] 100% Synchronization succeeded +----------------------------------------------------------------------+ Child Node Synchronization +----------------------------------------------------------------------+ Repository: Action: Merged Id: iso-repo Units: Added: 0 Removed: 0 Updated: 0 Repository: Action: Merged Id: live-rhel-6-server-x86_64-pulp Units: Added: 0 Removed: 0 Updated: 0 Repository: Action: Merged Id: puppet-repo Units: Added: 0 Removed: 0 Updated: 0 Repository: Action: Merged Id: test-repo1 Units: Added: 0 Removed: 0 Updated: 0 [root@pulp-v2-server ~]#
Pulp 2.3 released.