Upon reinstalling pulp rpm, the following message is discovered in ssl_error.log [Tue Aug 02 10:33:59 2011] [warn] RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] mod_wsgi (pid=12428): Target WSGI script '/srv/pulp/webservices.wsgi' cannot be loaded as Python module. [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] mod_wsgi (pid=12428): Exception occurred processing WSGI script '/srv/pulp/webservices.wsgi'. [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] Traceback (most recent call last): [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/srv/pulp/webservices.wsgi", line 20, in <module> [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] application = wsgi_application() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 103, in wsgi_application [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] _initialize_pulp() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 81, in _initialize_pulp [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] async.initialize() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/async.py", line 155, in initialize [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] _load_persisted_tasks() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/async.py", line 131, in _load_persisted_tasks [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] task = TaskSnapshot(snapshot).to_task() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/db/model/persistence.py", line 63, in to_task [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] return cls.from_snapshot(self) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 254, in from_snapshot [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] setattr(task, field, pickle.loads(snapshot[field])) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib64/python2.7/pickle.py", line 1382, in loads [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] return Unpickler(file).load() [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib64/python2.7/pickle.py", line 858, in load [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] dispatch[key](self) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib64/python2.7/pickle.py", line 1133, in load_reduce [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] value = func(*args) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] File "/usr/lib/python2.7/site-packages/pulp/server/tasking/taskqueue/storage.py", line 186, in _unpickle_method [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] return func.__get__(obj, cls) [Tue Aug 02 10:35:25 2011] [error] [client 127.0.0.1] UnboundLocalError: local variable 'func' referenced before assignment This is found in pulp 0.0.218. Only known workaround is to go into mongo db and run db.task_snapshots.remove() and restart httpd.
Sample snapshot from db: [root@pulp-qe pulp]# mongo pulp_database MongoDB shell version: 1.6.4 connecting to: pulp_database > db.task_snapshots.findOne() { "_id" : "6d50a56c-3101-4790-8904-4491b56969ab", "args" : "(lp0\nVtest_consumer\np1\na(lp2\nS'patb'\np3\naa.", "class_name" : "ConsumerGroupApi", "start_time" : "N.", "failure_threshold" : 5, "callable" : "cpulp.server.tasking.taskqueue.storage\n_unpickle_method\np0\n(S'__installpackages'\np1\nccopy_reg\n_reconstructor\np2\n(cpulp.server.api.consumer_group\nConsumerGroupApi\np3\nc__builtin__\nobject\np4\nNtp5\nRp6\n(dp7\nS'repoApi'\np8\ng2\n(cpulp.server.api.repo\nRepoApi\np9\ng4\nNtp10\nRp11\n(dp12\nS'errataapi'\np13\ng2\n(cpulp.server.api.errata\nErrataApi\np14\ng4\nNtp15\nRp16\nsS'published_path'\np17\nS'/var/lib/pulp/published/repos'\np18\nsS'distroapi'\np19\ng2\n(cpulp.server.api.distribution\nDistributionApi\np20\ng4\nNtp21\nRp22\nsS'filterapi'\np23\ng2\n(cpulp.server.api.filter\nFilterApi\np24\ng4\nNtp25\nRp26\nsS'fileapi'\np27\ng2\n(cpulp.server.api.file\nFileApi\np28\ng4\nNtp29\nRp30\nsS'distro_path'\np31\nS'/var/lib/pulp/published/ks'\np32\nsS'localStoragePath'\np33\nS'/var/lib/pulp/'\np34\nsS'packageapi'\np35\ng2\n(cpulp.server.api.package\nPackageApi\np36\ng4\nNtp37\nRp38\nsS'cdsapi'\np39\ng2\n(cpulp.server.api.cds\nCdsApi\np40\ng4\nNtp41\nRp42\n(dp43\nS'cds_history_api'\np44\ng2\n(cpulp.server.api.cds_history\nCdsHistoryApi\np45\ng4\nNtp46\nRp47\nsS'dispatcher'\np48\ng2\n(cpulp.server.cds.dispatcher\nGoferDispatcher\np49\ng4\nNtp50\nRp51\nsbsbsS'consumerApi'\np52\ng2\n(cpulp.server.api.consumer\nConsumerApi\np53\ng4\nNtp54\nRp55\n(dp56\ng13\ng2\n(g14\ng4\nNtp57\nRp58\nsS'consumer_history_api'\np59\ng2\n(cpulp.server.api.consumer_history\nConsumerHistoryApi\np60\ng4\nNtp61\nRp62\nsg35\ng2\n(g36\ng4\nNtp63\nRp64\nsS'repoapi'\np65\ng2\n(g9\ng4\nNtp66\nRp67\n(dp68\ng13\ng2\n(g14\ng4\nNtp69\nRp70\nsg17\nS'/var/lib/pulp/published/repos'\np71\nsg19\ng2\n(g20\ng4\nNtp72\nRp73\nsg23\ng2\n(g24\ng4\nNtp74\nRp75\nsg27\ng2\n(g28\ng4\nNtp76\nRp77\nsg31\nS'/var/lib/pulp/published/ks'\np78\nsg33\ng34\nsg35\ng2\n(g36\ng4\nNtp79\nRp80\nsg39\ng2\n(g40\ng4\nNtp81\nRp82\n(dp83\ng44\ng2\n(g45\ng4\nNtp84\nRp85\nsg48\ng2\n(g49\ng4\nNtp86\nRp87\nsbsbsbsbg3\ntp88\nRp89\n.", "result" : "N.", "scheduler" : "ccopy_reg\n_reconstructor\np0\n(cpulp.server.tasking.scheduler\nImmediateScheduler\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.", "timeout_delta" : "cdatetime\ntimedelta\np0\n(I0\nI600\nI0\ntp1\nRp2\n.", "id" : "2623b53d-bd14-11e0-8f86-54520040237d", "_ns" : "task_snapshots", "method_name" : "__installpackages", "exception" : "N.", "job_id" : "S'776d4218-4f4a-440c-8178-4bb87d429843'\np0\n.", "cancel_attempts" : 0, "schedule_threshold" : "cdatetime\ntimedelta\np0\n(I0\nI300\nI0\ntp1\nRp2\n.", "traceback" : "N.", "task_class" : "cpulp.server.tasking.task\nAsyncTask\np0\n.", "finish_time" : "N.", "state" : "waiting", "_progress_callback" : "N.", "kwargs" : "(dp0\n.", "progress" : null, "consecutive_failures" : 0 }
Turns out the new callable for install packages is a private method which the custom unpickler didn't account for. Added logic for unpickling private methods as well as some instrumentation for better error reporting when the unpickle fails.
Fixed in hash: 6e1602d215da8f188dd648cb08f712d2529ccf9c
build: 0.219
Closing as issue no longer exists during rpm reinstalls.
Closing with Community Release 15 pulp-0.0.223-4.