Bug 1022646

Summary:	nodes manifest migration fails on 2.2 to 2.3 upgrades
Product:	[Retired] Pulp	Reporter:	Jeff Ortel <jortel>
Component:	nodes	Assignee:	Jeff Ortel <jortel>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Preethi Thomas <pthomas>
Severity:	high	Docs Contact:
Priority:	high
Version:	Master	CC:	pthomas
Target Milestone:	---	Keywords:	Triaged
Target Release:	2.3.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2013-12-09 14:30:48 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Jeff Ortel 2013-10-23 17:19:48 UTC

Description of problem:

After upgrading pulp 2.2 -> 2.3, node sync fails.

Version-Release number of selected component (if applicable):

2.3.0

How reproducible:

Only if upgraded child node has done a previous sync of the same repository.

Steps to Reproduce:
1.  install a pulp nodes 2.2 parent and child.
2.  activate the child and bind it to a repository.
3.  perform a node sync.
4.  upgrade both parent and child nodes.
5.  run the node sync.

Actual results:

pulp.log on the child node:

KeyError: 'errors'
2013-10-23 11:04:27,887 - ERROR - Client-side exception occurred
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/core.py", line 478, in run
    exit_code = Cli.run(self, args)
  File "/usr/lib/python2.6/site-packages/okaara/cli.py", line 974, in run
    exit_code = command_or_section.execute(self.prompt, remaining_args)
  File "/usr/lib/python2.6/site-packages/pulp/client/extensions/extensions.py", line 224, in execute
    return self.method(*arg_list, **clean_kwargs)
  File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/commands.py", line 515, in run
    self.poll([task], kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/client/commands/polling.py", line 143, in poll
    self.succeeded(task)
  File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/commands.py", line 530, in succeeded
    r = UpdateRenderer(self.context.prompt, report)
  File "/usr/lib/python2.6/site-packages/pulp_node/extensions/admin/rendering.py", line 168, in __init__
    self.errors = self.details['errors']
KeyError: 'errors'

Expected results:

manifest is migrated and all is well.


Additional info:

Comment 1 Jeff Ortel 2013-10-23 17:22:27 UTC

The trace in the description is incorrect.  Here is the correct one:

Child pulp.log:

2013-10-23 10:33:15,701 pulp.server.dispatch.task:INFO: SUCCESS: Task dd7663f4-bfda-44cb-9267-82eeb7878b97: CallRequest: RepoImporterManager.update_importer_config(u'follett-el5-x86_64', importer_config=****)
2013-10-23 10:33:16,463 pulp_node.importers.strategies:ERROR: follett-el5-x86_64
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory
    manifest.read(migrate)
  File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read
    migration(self.path)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 55, in migrate
    version_in = manifest[VERSION]
KeyError: 'version'
2013-10-23 10:33:16,475 pulp.server.dispatch.task:INFO: SUCCESS: Task 9e6c0f04-f7da-4231-b49d-58528189cd9f: CallRequest: RepoSyncManager.sync(u'follett-el5-x86_64', sync_config_override={})
2013-10-23 10:33:18,372 pulp.server.dispatch.task:INFO: SUCCESS: Task b229ffdc-647b-4a4a-814b-52ed1e90143d: CallRequest: RepoImporterManager.update_importer_config(u'follett-el6-x86_64', importer_config=****)

+----------------------------------------------------------------------+
                The following [10] errors were reported
+----------------------------------------------------------------------+

- 01: An error occurred while downloading units from the parent for repository
[follett-el5-x86_64. The cause may be that the repository has not been
published

Comment 2 Jeff Ortel 2013-10-29 14:26:50 UTC

Example 2.2 manifest.json:
# pwd
/var/lib/pulp/working/repos/NetEnt-rhel-6-4-server-x86_64-os/importers/nodes_http_importer
[root@ind-pulp-01 nodes_http_importer]# cat manifest.json 
{
  "total_units": 12565, 
  "publishing_details": {
    "base_url": "https://cyd-pulp-01.nix.cydmodule.com/pulp/nodes/https/repos/NetEnt-rhel-6-4-server-x86_64-os"
  }, 
  "id": "c85a8d67-5edb-46f7-8d2a-458fc29858e4", 
  "units_path": "/var/www/pulp/nodes/https/repos/tmp0HZaF4/units.json.gz"

Comment 3 Jeff Ortel 2013-11-01 13:43:52 UTC

https://github.com/pulp/pulp/pull/696

Comment 4 Jeff Ortel 2013-11-01 22:29:08 UTC

build: 2.3.0-0.27.beta

Comment 5 Preethi Thomas 2013-11-05 20:48:39 UTC

looks like this still fails

fail-qa

2013-11-05 15:42:48,877 pulp_node.importers.strategies:ERROR: iso-repo
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory
    manifest.read(migrate)
  File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read
    migration(self.path)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate
    manifest = migration(manifest)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1
    manifest.pop(UNITS_SIZE)
KeyError: 'units_size'
2013-11-05 15:42:49,413 pulp.plugins.file.distributor:INFO: Beginning publish for repository <iso-repo>
2013-11-05 15:42:50,483 pulp_node.importers.strategies:ERROR: puppet-repo
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory
    manifest.read(migrate)
  File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read
    migration(self.path)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate
    manifest = migration(manifest)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1
    manifest.pop(UNITS_SIZE)
KeyError: 'units_size'
2013-11-05 15:42:50,978 pulp_puppet.plugins.distributors.publish:INFO: Beginning publish for repository <puppet-repo>
2013-11-05 15:42:50,979 pulp_puppet.plugins.distributors.publish:INFO: Initializing build directory for repository <puppet-repo>
2013-11-05 15:42:51,002 pulp_puppet.plugins.distributors.publish:INFO: Creating symlinks for modules in repository <puppet-repo>
2013-11-05 15:42:51,014 pulp_puppet.plugins.distributors.publish:INFO: Generating metadata for repository <puppet-repo>
2013-11-05 15:42:51,051 pulp_puppet.plugins.distributors.publish:INFO: Making newly built repository live for repository <puppet-repo>
2013-11-05 15:42:51,080 pulp_puppet.plugins.distributors.publish:INFO: Cleaning up build directory for repository <puppet-repo>
2013-11-05 15:42:52,193 pulp_node.importers.strategies:ERROR: zoo
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 190, in _unit_inventory
    manifest.read(migrate)
  File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 130, in read
    migration(self.path)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 87, in migrate
    manifest = migration(manifest)
  File "/usr/lib/python2.6/site-packages/pulp_node/migration.py", line 64, in migration_1
    manifest.pop(UNITS_SIZE)
KeyError: 'units_size'

Comment 6 Jeff Ortel 2013-11-05 20:58:01 UTC

https://github.com/pulp/pulp/pull/703

Comment 7 Jeff Ortel 2013-11-07 15:06:42 UTC

build: 2.3.0-0.29.beta

Comment 8 Preethi Thomas 2013-11-07 22:10:09 UTC

fails-qa

[root@pulp-v2-server ~]# rpm -qa |grep pulp-server
pulp-server-2.3.0-0.29.beta.el6.noarch
[root@pulp-v2-server ~]# 

node sync seems to be failing after upgrade

2013-11-07 15:45:26,755 [ERROR][worker-0] update() @ dispatcher.py:112 - handler failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/agent/lib/dispatcher.py", line 108, in update
    report = handler.update(conduit, units, dict(options))
  File "/usr/lib/python2.6/site-packages/pulp_node/handlers/handler.py", line 88, in update
    bindings = RepositoryBinding.fetch_all(pulp_bindings, conduit.consumer_id)
  File "/usr/lib/python2.6/site-packages/pulp_node/handlers/model.py", line 511, in fetch_all
    http = bindings.bind.find_by_id(node_id)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/consumer.py", line 154, in find_by_id
    return self.server.GET(path)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 96, in GET
    return self._request('GET', path, queries)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 154, in _request
    self._handle_exceptions(response_code, response_body)
  File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 195, in _handle_exceptions
    raise code_class_mappings[response_code](response_body)
PermissionsException: RequestException: GET request on /pulp/api/v2/consumers/node1/bindings/ failed with 401 - Pulp exception occurred: AuthenticationFailed
2013-11-07 15:45:26,759 [INFO][worker-0] sendreply() @ rmi.py:172 - 8cd5dc15-e35e-4bfd-90c5-1b895bb2d5e0 processed in: 106 (ms)
:^C
[root@pulp-node2 ~]# tail -f /var/log/pulp/pulp.log 
2013-11-07 15:43:54,815 pulp.server.webservices.application:INFO: *************************************************************
2013-11-07 15:43:54,815 pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
2013-11-07 15:43:54,817 pulp.server.webservices.application:INFO: *************************************************************
2013-11-07 15:43:54,830 gofer.messaging.broker:INFO: connecting:
{pulp-node2.usersys.redhat.com:5672}:
transport=TCP
host=pulp-node2.usersys.redhat.com
port=5672
cacert=/etc/pki/qpid/ca/ca.crt
clientcert=/etc/pki/qpid/client/client.pem


2013-11-07 17:00:16,038 pulp.server.webservices.middleware.exception:ERROR: Missing resource(s): repo=SAM
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 279, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 422, in _delegate
    return f()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 430, in <lambda>
    f = lambda: self._delegate_sub_application(pat, what)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 455, in _delegate_sub_application
    return app.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 566, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 581, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 420, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
    value = method(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/repositories.py", line 245, in GET
    raise exceptions.MissingResource(repo=id)
MissingResource: Missing resource(s): repo=SAM
2013-11-07 17:00:17,738 pulp_node.importers.strategies:ERROR: SAM
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 127, in synchronize
    self._synchronize(request)
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 392, in _synchronize
    unit_inventory = self._unit_inventory(request)
  File "/usr/lib/python2.6/site-packages/pulp_node/importers/strategies.py", line 212, in _unit_inventory
    parent_units = manifest.get_units()
  File "/usr/lib/python2.6/site-packages/pulp_node/manifest.py", line 145, in get_units
    path = self.units_path()
TypeError: 'NoneType' object is not callable

Comment 9 Jeff Ortel 2013-11-08 02:11:13 UTC

https://github.com/pulp/pulp/pull/710

Comment 10 Jeff Ortel 2013-11-08 15:17:32 UTC

build: 2.3.0-0.31.beta

Comment 11 Preethi Thomas 2013-11-13 21:34:22 UTC

verified

[root@pulp-v2-server ~]# rpm -qa pulp-server
pulp-server-2.3.0-0.33.beta.el6.noarch
[root@pulp-v2-server ~]# 

[root@pulp-v2-server ~]# pulp-admin node sync run  --node-id node1
This command may be exited via ctrl+c without affecting the request.

[-]
Waiting to begin...



(1/4) Repository: iso-repo
[==================================================] 100%


(2/4) Repository: live-rhel-6-server-x86_64-pulp
[==================================================] 100%


(3/4) Repository: puppet-repo
[==================================================] 100%


(4/4) Repository: test-repo1
[==================================================] 100%




Synchronization succeeded

+----------------------------------------------------------------------+
                       Child Node Synchronization
+----------------------------------------------------------------------+

Repository: 
  Action: Merged
  Id:     iso-repo
  Units:  
    Added:   0
    Removed: 0
    Updated: 0

Repository: 
  Action: Merged
  Id:     live-rhel-6-server-x86_64-pulp
  Units:  
    Added:   0
    Removed: 0
    Updated: 0

Repository: 
  Action: Merged
  Id:     puppet-repo
  Units:  
    Added:   0
    Removed: 0
    Updated: 0

Repository: 
  Action: Merged
  Id:     test-repo1
  Units:  
    Added:   0
    Removed: 0
    Updated: 0



[root@pulp-v2-server ~]#

Comment 12 Preethi Thomas 2013-12-09 14:30:48 UTC

Pulp 2.3 released.