Bug 727666

Summary: pulp rpm reinstall causes traceback
Product: [Retired] Pulp Reporter: dgao
Component: z_otherAssignee: Jason Connor <jconnor>
Status: CLOSED CURRENTRELEASE QA Contact: dgao
Severity: high Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: mmccune, pthomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-16 12:11:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 647488    

Description dgao 2011-08-02 19:41:17 UTC
Upon reinstalling pulp rpm, the following message is discovered in ssl_error.log

[Tue Aug 02 10:33:59 2011] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] mod_wsgi (pid=12428): Target WSGI script '/srv/pulp/webservices.wsgi' cannot be loaded as Python module.
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] mod_wsgi (pid=12428): Exception occurred processing WSGI script '/srv/pulp/webservices.wsgi'.
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] Traceback (most recent call last):
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/srv/pulp/webservices.wsgi", line 20, in <module>
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     application = wsgi_application()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 103, in wsgi_application
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     _initialize_pulp()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 81, in _initialize_pulp
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     async.initialize()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/async.py", line 155, in initialize
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     _load_persisted_tasks()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/async.py", line 131, in _load_persisted_tasks
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     task = TaskSnapshot(snapshot).to_task()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/db/model/persistence.py", line 63, in to_task
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     return cls.from_snapshot(self)
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 254, in from_snapshot
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     setattr(task, field, pickle.loads(snapshot[field]))
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib64/python2.7/pickle.py", line 1382, in loads
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     return Unpickler(file).load()
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib64/python2.7/pickle.py", line 858, in load
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     dispatch[key](self)
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib64/python2.7/pickle.py", line 1133, in load_reduce
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     value = func(*args)
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]   File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/storage.py", line 186, in _unpickle_method
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1]     return func.__get__(obj, cls)
[Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] UnboundLocalError: local variable 'func' referenced before assignment


This is found in pulp 0.0.218. Only known workaround is to go into mongo db and run db.task_snapshots.remove() and restart httpd.

Comment 1 dgao 2011-08-02 19:49:14 UTC
Sample snapshot from db:


[root@pulp-qe pulp]# mongo pulp_database
MongoDB shell version: 1.6.4
connecting to: pulp_database
> db.task_snapshots.findOne()
{
	"_id" : "6d50a56c-3101-4790-8904-4491b56969ab",
	"args" : "(lp0\nVtest_consumer\np1\na(lp2\nS'patb'\np3\naa.",
	"class_name" : "ConsumerGroupApi",
	"start_time" : "N.",
	"failure_threshold" : 5,
	"callable" : "cpulp.server.tasking.taskqueue.storage\n_unpickle_method\np0\n(S'__installpackages'\np1\nccopy_reg\n_reconstructor\np2\n(cpulp.server.api.consumer_group\nConsumerGroupApi\np3\nc__builtin__\nobject\np4\nNtp5\nRp6\n(dp7\nS'repoApi'\np8\ng2\n(cpulp.server.api.repo\nRepoApi\np9\ng4\nNtp10\nRp11\n(dp12\nS'errataapi'\np13\ng2\n(cpulp.server.api.errata\nErrataApi\np14\ng4\nNtp15\nRp16\nsS'published_path'\np17\nS'/var/lib/pulp/published/repos'\np18\nsS'distroapi'\np19\ng2\n(cpulp.server.api.distribution\nDistributionApi\np20\ng4\nNtp21\nRp22\nsS'filterapi'\np23\ng2\n(cpulp.server.api.filter\nFilterApi\np24\ng4\nNtp25\nRp26\nsS'fileapi'\np27\ng2\n(cpulp.server.api.file\nFileApi\np28\ng4\nNtp29\nRp30\nsS'distro_path'\np31\nS'/var/lib/pulp/published/ks'\np32\nsS'localStoragePath'\np33\nS'/var/lib/pulp/'\np34\nsS'packageapi'\np35\ng2\n(cpulp.server.api.package\nPackageApi\np36\ng4\nNtp37\nRp38\nsS'cdsapi'\np39\ng2\n(cpulp.server.api.cds\nCdsApi\np40\ng4\nNtp41\nRp42\n(dp43\nS'cds_history_api'\np44\ng2\n(cpulp.server.api.cds_history\nCdsHistoryApi\np45\ng4\nNtp46\nRp47\nsS'dispatcher'\np48\ng2\n(cpulp.server.cds.dispatcher\nGoferDispatcher\np49\ng4\nNtp50\nRp51\nsbsbsS'consumerApi'\np52\ng2\n(cpulp.server.api.consumer\nConsumerApi\np53\ng4\nNtp54\nRp55\n(dp56\ng13\ng2\n(g14\ng4\nNtp57\nRp58\nsS'consumer_history_api'\np59\ng2\n(cpulp.server.api.consumer_history\nConsumerHistoryApi\np60\ng4\nNtp61\nRp62\nsg35\ng2\n(g36\ng4\nNtp63\nRp64\nsS'repoapi'\np65\ng2\n(g9\ng4\nNtp66\nRp67\n(dp68\ng13\ng2\n(g14\ng4\nNtp69\nRp70\nsg17\nS'/var/lib/pulp/published/repos'\np71\nsg19\ng2\n(g20\ng4\nNtp72\nRp73\nsg23\ng2\n(g24\ng4\nNtp74\nRp75\nsg27\ng2\n(g28\ng4\nNtp76\nRp77\nsg31\nS'/var/lib/pulp/published/ks'\np78\nsg33\ng34\nsg35\ng2\n(g36\ng4\nNtp79\nRp80\nsg39\ng2\n(g40\ng4\nNtp81\nRp82\n(dp83\ng44\ng2\n(g45\ng4\nNtp84\nRp85\nsg48\ng2\n(g49\ng4\nNtp86\nRp87\nsbsbsbsbg3\ntp88\nRp89\n.",
	"result" : "N.",
	"scheduler" : "ccopy_reg\n_reconstructor\np0\n(cpulp.server.tasking.scheduler\nImmediateScheduler\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.",
	"timeout_delta" : "cdatetime\ntimedelta\np0\n(I0\nI600\nI0\ntp1\nRp2\n.",
	"id" : "2623b53d-bd14-11e0-8f86-54520040237d",
	"_ns" : "task_snapshots",
	"method_name" : "__installpackages",
	"exception" : "N.",
	"job_id" : "S'776d4218-4f4a-440c-8178-4bb87d429843'\np0\n.",
	"cancel_attempts" : 0,
	"schedule_threshold" : "cdatetime\ntimedelta\np0\n(I0\nI300\nI0\ntp1\nRp2\n.",
	"traceback" : "N.",
	"task_class" : "cpulp.server.tasking.task\nAsyncTask\np0\n.",
	"finish_time" : "N.",
	"state" : "waiting",
	"_progress_callback" : "N.",
	"kwargs" : "(dp0\n.",
	"progress" : null,
	"consecutive_failures" : 0
}

Comment 2 Jason Connor 2011-08-02 21:16:02 UTC
Turns out the new callable for install packages is a private method which the custom unpickler didn't account for.

Added logic for unpickling private methods as well as some instrumentation for better error reporting when the unpickle fails.

Comment 3 Jason Connor 2011-08-02 21:16:26 UTC
Fixed in hash: 6e1602d215da8f188dd648cb08f712d2529ccf9c

Comment 4 Jeff Ortel 2011-08-03 20:11:30 UTC
build: 0.219

Comment 5 dgao 2011-08-08 21:00:09 UTC
Closing as issue no longer exists during rpm reinstalls.

Comment 6 Preethi Thomas 2011-08-16 12:11:24 UTC
Closing with Community Release 15

pulp-0.0.223-4.

Comment 7 Preethi Thomas 2011-08-16 12:22:53 UTC
Closing with Community Release 15

pulp-0.0.223-4.