Bug 1008020

Summary: Content View publish fails with pulp 500 error
Product: Red Hat Satellite Reporter: Mike McCune <mmccune>
Component: Content ManagementAssignee: David Davis <daviddavis>
Status: CLOSED CURRENTRELEASE QA Contact: Garik Khachikyan <gkhachik>
Severity: high Docs Contact:
Priority: high    
Version: 6.0.2CC: cwelton, gkhachik, hhovsepy, jmontleo, jsherril, mkoci
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-24 17:08:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1009127, 1010292    
Bug Blocks:    
Attachments:
Description Flags
katello-debug none

Description Mike McCune 2013-09-13 19:48:17 UTC
Created attachment 797519 [details]
katello-debug

1) Import RH Manifest

2) Enable and Sync "Red Hat Subscription Asset Manager for RHEL 6 Server RPMs x86_64 6Server"

3) Create a Content View

4) Add that repo you synced to the CV

5) Try to publish it

See:

/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/abstract_response.rb:48:in `return!'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.0.5/lib/runcible/base.rb:97:in `block in get_response'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:218:in `process_result'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:169:in `block in transmit'
/opt/rh/ruby193/root/usr/share/ruby/net/http.rb:746:in `start'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:166:in `transmit'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:60:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/request.rb:31:in `execute'
/opt/rh/ruby193/root/usr/share/gems/gems/rest-client-1.6.1/lib/restclient/resource.rb:59:in `post'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.0.5/lib/runcible/base.rb:96:in `get_response'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.0.5/lib/runcible/base.rb:87:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.0.5/lib/runcible/resources/unit.rb:45:in `search'
/opt/rh/ruby193/root/usr/share/gems/gems/runcible-1.0.5/lib/runcible/extensions/unit.rb:81:in `find_all_by_unit_ids'


...

Comment 1 Mike McCune 2013-09-13 19:56:18 UTC
from 

/v/l/pulp/pulp.log

2013-09-13 15:05:10,853 pulp.server.dispatch.task:INFO: SUCCESS: Task 3595f1e1-01ec-4c1a-86a0-aeaea9ebb647: CallRequest: RepoUnitAssociationManager.associate_from_repo('ACME_Corporation-Red_Hat_Enterprise_Linux_Server-Red_Hat_Subscription_Asset_Manager_for_RHEL_6_Server_RPMs_x86_64_6Server', u'ACME_Corporation-Library-APPS-SAM1-Red_Hat_Enterprise_Linux_Server-Red_Hat_Subscription_Asset_Manager_for_RHEL_6_Server_RPMs_x86_64_6Server', import_config_override=****, criteria=****)
2013-09-13 15:05:27,373 pulp.server.webservices.middleware.exception:ERROR: '_last_updated'
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 278, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 557, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 572, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 232, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 413, in _delegate
    return f()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 421, in <lambda>
    f = lambda: self._delegate_sub_application(pat, what)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 446, in _delegate_sub_application
    return app.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 557, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 572, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 232, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 395, in _delegate
    return handle_class(f)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 386, in handle_class
    return tocall(*args)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
    value = method(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/contents.py", line 184, in POST
    units = [ContentUnitsCollection.process_unit(unit) for unit in raw_units]
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/contents.py", line 73, in process_unit
    unit = serialization.content.content_unit_obj(unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/serialization/content.py", line 43, in content_unit_obj
    last_updated = content_unit['_last_updated']
KeyError: '_last_updated'
2013-09-13 15:05:27,374 pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 278, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 557, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 572, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 232, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 413, in _delegate
    return f()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 421, in <lambda>
    f = lambda: self._delegate_sub_application(pat, what)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 446, in _delegate_sub_application
    return app.handle_with_processors()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 557, in processor
    return handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/web/application.py", line 572, in processor
    result = handler()
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.6/site-packages/web/application.py", line 232, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 395, in _delegate
    return handle_class(f)
  File "/usr/lib/python2.6/site-packages/web/application.py", line 386, in handle_class
    return tocall(*args)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
    value = method(self, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/contents.py", line 184, in POST
    units = [ContentUnitsCollection.process_unit(unit) for unit in raw_units]
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/controllers/contents.py", line 73, in process_unit
    unit = serialization.content.content_unit_obj(unit)
  File "/usr/lib/python2.6/site-packages/pulp/server/webservices/serialization/content.py", line 43, in content_unit_obj
    last_updated = content_unit['_last_updated']
KeyError: '_last_updated'

Comment 3 David Davis 2013-09-20 12:45:02 UTC
This seems to happen also when calling unit search:

[ERROR 2013-09-20 08:42:20 pulp_rest] RestClient.post "https://localhost/pulp/api/v2/content/units/rpm/search/", 632 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Authorization"=>"OAuth oauth_body_hash=\"2jmj7l5rSw0yVb%2FvlWAYkK%2FYBwk%3D\", oauth_consumer_key=\"katello\", oauth_nonce=\"IospyJPoxnUr1CtazULH9lqGwYvonhsMbJmyj2iOU\", oauth_signature=\"iqFplE%2FIIwAvH9g56O46%2Fs3qB9I%3D\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"1379680940\", oauth_version=\"1.0\"", "Content-Length"=>"632", "accept"=>"application/json", "content_type"=>"application/json", "pulp-user"=>"admin"
 | \n# => 500 InternalServerError | application/json 4595 bytes


Pulp error:

2013-09-20 08:42:20,152 pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/exception.py", line 44, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/middleware/postponed.py", line 39, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 279, in wsgi
    result = self.handle_with_processors()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor
    return handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor
    result = handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 422, in _delegate
    return f()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 430, in <lambda>
    f = lambda: self._delegate_sub_application(pat, what)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 455, in _delegate_sub_application
    return app.handle_with_processors()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 26, in _handle_with_processors
    return process(self.processors)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 566, in processor
    return handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in process
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/web/application.py", line 581, in processor
    result = handler()
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 23, in <lambda>
    return p(lambda : process(processors))
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 25, in process
    return self.handle()
  File "/usr/lib/python2.7/site-packages/web/application.py", line 230, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 405, in _delegate
    return handle_class(f)
  File "/usr/lib/python2.7/site-packages/web/application.py", line 396, in handle_class
    return tocall(*args)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/decorators.py", line 227, in _auth_decorator
    value = method(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/contents.py", line 184, in POST
    units = [ContentUnitsCollection.process_unit(unit) for unit in raw_units]
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/controllers/contents.py", line 73, in process_unit
    unit = serialization.content.content_unit_obj(unit)
  File "/usr/lib/python2.7/site-packages/pulp/server/webservices/serialization/content.py", line 43, in content_unit_obj
    last_updated = content_unit['_last_updated']
KeyError: '_last_updated'

Comment 4 Justin Sherrill 2013-09-20 12:59:51 UTC
I'm hitting this as well, prohibits package indexing from working.

Comment 5 David Davis 2013-09-20 21:55:22 UTC
This provides a workaround for this bug and adds pulp 2.3 support to katello:

https://github.com/Katello/katello/pull/3015

Comment 8 Garik Khachikyan 2013-10-22 11:45:41 UTC
# VERIFIED

both the normal and composite content publishing went just ok (RHEL 6.4 inside).

checked on:
===
* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.8.25-1.el6sam.noarch
* candlepin-scl-1-5.el6_4.noarch
* candlepin-scl-quartz-2.1.5-5.el6_4.noarch
* candlepin-scl-rhino-1.7R3-1.el6_4.noarch
* candlepin-scl-runtime-1-5.el6_4.noarch
* candlepin-selinux-0.8.25-1.el6sam.noarch
* candlepin-tomcat6-0.8.25-1.el6sam.noarch
* elasticsearch-0.19.9-8.el6sat.noarch
* foreman-1.3.0-18.el6sat.noarch
* foreman-compute-1.3.0-18.el6sat.noarch
* foreman-libvirt-1.3.0-18.el6sat.noarch
* foreman-postgresql-1.3.0-18.el6sat.noarch
* foreman-proxy-1.3.0-3.el6sat.noarch
* katello-1.4.6-39.el6sat.noarch
* katello-all-1.4.6-39.el6sat.noarch
* katello-candlepin-cert-key-pair-1.0-1.noarch
* katello-certs-tools-1.4.4-1.el6sat.noarch
* katello-cli-1.4.3-24.el6sat.noarch
* katello-cli-common-1.4.3-24.el6sat.noarch
* katello-common-1.4.6-39.el6sat.noarch
* katello-configure-1.4.7-5.el6sat.noarch
* katello-configure-foreman-1.4.7-5.el6sat.noarch
* katello-configure-foreman-proxy-1.4.7-5.el6sat.noarch
* katello-foreman-all-1.4.6-39.el6sat.noarch
* katello-glue-candlepin-1.4.6-39.el6sat.noarch
* katello-glue-elasticsearch-1.4.6-39.el6sat.noarch
* katello-glue-pulp-1.4.6-39.el6sat.noarch
* katello-qpid-broker-key-pair-1.0-1.noarch
* katello-qpid-client-key-pair-1.0-1.noarch
* katello-selinux-1.4.4-4.el6sat.noarch
* openldap-2.4.23-31.el6.x86_64
* pulp-katello-plugins-0.2-1.el6sat.noarch
* pulp-nodes-common-2.3.0-0.22.beta.el6sat.noarch
* pulp-nodes-parent-2.3.0-0.22.beta.el6sat.noarch
* pulp-puppet-plugins-2.3.0-0.22.beta.el6sat.noarch
* pulp-rpm-plugins-2.3.0-0.22.beta.el6sat.noarch
* pulp-selinux-2.3.0-0.22.beta.el6sat.noarch
* pulp-server-2.3.0-0.22.beta.el6sat.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.2.2-2.el6sat.noarch
* ruby193-rubygem-net-ldap-0.3.1-3.el6sat.noarch
* ruby193-rubygem-runcible-1.0.7-1.el6sat.noarch
* signo-0.0.22-2.el6sat.noarch
* signo-katello-0.0.22-2.el6sat.noarch

Comment 9 Bryan Kearney 2014-04-24 17:08:41 UTC
This was verified and delivered with MDP2. Closing it out.