Bug 744800

Summary: ISE on pulp-admin commands when I had the pulp-sever running over the weekend.
Product: [Retired] Pulp Reporter: Preethi Thomas <pthomas>
Component: z_otherAssignee: Jeff Ortel <jortel>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.0.0Keywords: Triaged
Target Milestone: ---   
Target Release: Sprint 29   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Preethi Thomas 2011-10-10 14:09:57 UTC
Description of problem:

I had my pulp-server running over the weekend with some scheduled syncs.

on monday morning saw this on pulp-admin repo list

[root@preethi ~]# pulp-admin -u admin -p admin repo list
error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.21 (Fedora) Server at preethi.usersys.redhat.com Port 443</address>
</body></html>


Also from pulp.log

2011-10-09 03:56:08,723 2490:139983452559392: pulp.server.async:INFO: async:361 Task reply handler, started.
2011-10-09 04:03:34,596 5242:139983164552960: gofer.rmi.async:ERROR: threading:131 watchdog
None
2011-10-09 05:10:53,342 30141:139983164552960: gofer.rmi.async:ERROR: async:441 watchdog
None
2011-10-09 05:21:34,575 1824:139983164552960: gofer.messaging.endpoint:ERROR: consumer:137 07129a10-6b6b-4cf1-be06-9eefacccc6a5
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 236, in __del__
    self.close()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 212, in close
    self._lock()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 222, in _lock
    self.__mutex.acquire()
AttributeError: Producer instance has no attribute '_Endpoint__mutex'
2011-10-09 05:21:34,576 1824:139983164552960: gofer.rmi.async:ERROR: threading:269 watchdog
None
2011-10-09 05:25:03,973 3120:139983164552960: gofer.messaging.endpoint:ERROR: endpoint:238 cca020f0-4108-4892-a386-c123bf5865a4
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 236, in __del__
    self.close()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 212, in close
    self._lock()
  File "/usr/lib/python2.7/site-packages/gofer/messaging/endpoint.py", line 222, in _lock
    self.__mutex.acquire()
AttributeError: Producer instance has no attribute '_Endpoint__mutex'
2011-10-09 05:52:53,154 13391:139983164552960: gofer.rmi.async:ERROR: threading:397 watchdog
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/gofer/rmi/async.py", line 438, in run
    watchdog.process()
  File "/usr/lib/python2.7/site-packages/gofer/rmi/async.py", line 380, in process
    for sn,je in self.__jnl.load().items():
  File "/usr/lib/python2.7/site-packages/gofer/rmi/async.py", line 477, in load
    for fn in os.listdir(self.root):
TypeError: must be unicode, not str

Comment 1 Jeff Ortel 2011-10-27 21:02:43 UTC
This looks like a weird restart issue.  The gofer Endpoint.__del__() is being called before __init__() has completed.  This really can't happen.  I believe that jconnor reworking of the mod_wsgi start up stuff "removing the bootstrap" fixes this.

Comment 2 Jeff Ortel 2011-10-28 23:12:50 UTC
build: 0.244

Comment 3 Preethi Thomas 2011-12-21 18:10:13 UTC
verified
I haven't seen this issue in a while

Comment 4 Preethi Thomas 2012-02-24 20:16:59 UTC
Pulp v1.0 is released
Closed Current Release.

Comment 5 Preethi Thomas 2012-02-24 20:18:05 UTC
Pulp v1.0 is released.