Bug 698577

Summary: Getting Internal server Error after restarting "pulp-server" service
Product: [Retired] Pulp Reporter: Sachin Ghai <sghai>
Component: z_otherAssignee: Jason Connor <jconnor>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: high Docs Contact:
Priority: high    
Version: unspecifiedCC: kbidarka, mmccune
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-16 12:09:58 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 647488    

Description Sachin Ghai 2011-04-21 09:41:12 UTC 
Description of problem:
After restarting the pulp-server service, I got "500 Internal server" with pulp-admin tool.


Version-Release number of selected component (if applicable):
pulp 0.168
rhui-tool 2.0.15

How reproducible:
yesterday I installed 168 build and since then I got this issue 2 times.

Steps to Reproduce:
1. Restarted the pulp-server service.. All services restarted without any failure.
2. then tried "pulp-admin -u admin -p admin repo list". This cmd is throwing internal server error

  
Actual results:
[root@dhcp193-131 ~]# pulp-admin -u admin -p admin repo list
error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.15 (Red Hat) Server at dhcp193-131.pnq.redhat.com Port 443</address>
</body></html>



Expected results:
pulp-admin tool should work without "internal server error"

Additional info:


Before restarting pulp-server I was trying CDS sync with pulp-admin tool and it was not working.

And to get rid of this issue, I removed the mongod.lock file and restarted the pulp-server.. but no luck.


From ssl_error_logs seems that might be this issue is related to sync scheduling. but I'm sure no sync was running when I got this Internal server error.

[root@dhcp193-131 ~]# tail -f /var/log/httpd/error_log 
[Thu Apr 21 14:54:15 2011] [notice] Digest: done
[Thu Apr 21 14:54:15 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Thu Apr 21 14:54:15 2011] [notice] mod_python: using mutex_directory /tmp 
[Thu Apr 21 14:54:15 2011] [warn] mod_wsgi: Compiled for Python/2.6.2.
[Thu Apr 21 14:54:15 2011] [warn] mod_wsgi: Runtime using Python/2.6.5.
[Thu Apr 21 14:54:15 2011] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations
[Thu Apr 21 14:54:22 2011] [error] /usr/lib/python2.6/site-packages/oauth2/__init__.py:29: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
[Thu Apr 21 14:54:22 2011] [error]   import sha
[Thu Apr 21 14:54:22 2011] [error] /usr/lib/python2.6/site-packages/httplib2/__init__.py:29: DeprecationWarning: the md5 module is deprecated; use hashlib instead
[Thu Apr 21 14:54:22 2011] [error]   import md5


[root@dhcp193-131 ~]# tail -f /var/log/httpd/ssl_error_log 
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]     _init_repo_scheduled_syncs()
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]   File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 296, in _init_repo_scheduled_syncs
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]     _add_repo_scheduled_sync_task(repo)
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]   File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 171, in _add_repo_scheduled_sync_task
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]     task.scheduler = schedule_to_scheduler(repo['sync_schedule'])
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]   File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 141, in schedule_to_scheduler
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]     return IntervalScheduler(interval, start_time, runs)
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]   File "/usr/lib/python2.6/site-packages/pulp/server/tasking/scheduler.py", line 123, in __init__
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131]     str(start_time), str(interval))
[Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] TypeError: not enough arguments for format string


tail -f /var/log/pulp/pulp.log
2011-04-21 15:02:59,065 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhel61'), (u'id', u'rhel61'), (u'sync_in_progress', False)])
2011-04-21 15:02:59,066 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhel-server-6-updates-srpms-6Server-x86_64'), (u'id', u'rhel-server-6-updates-srpms-6Server-x86_64'), (u'sync_in_progress', False)])
2011-04-21 15:02:59,067 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhui-1.2-5Server-x86_64'), (u'id', u'rhui-1.2-5Server-x86_64'), (u'sync_in_progress', False)])



root@dhcp193-131 ~]# tail -f /var/log/pulp/db.log 
migration to data model version 5 started
migration to data model version 5 complete
migration to data model version 7 started
migration to data model version 7 complete
migration to data model version 8 started
migration to data model version 8 complete
migration to data model version 9 started
migration to data model version 9 complete
migration to data model version 10 started
migration to data model version 10 complete
Comment 1 Kedar Bidarkar 2011-04-21 10:15:52 UTC 
Facing same issue, on a totally different setup.( pulp-168, pulp-2.0.15)
Faced it earlier, even with build pulp-166 and pulp-2.0.15
Comment 2 Jason Connor 2011-04-21 20:11:48 UTC 
missing parens around the strings for substitution
fix pushed in master and rhui branches: de67c69
Comment 3 Jeff Ortel 2011-04-21 21:30:06 UTC 
build: 0.170
Comment 4 Sachin Ghai 2011-04-25 13:02:24 UTC 
I installed the build 0.170 and didn't face this issue as of now, however I'll keep an eye on this for couple of more days.
Comment 5 Preethi Thomas 2011-04-25 14:54:57 UTC 
moving to verified as per Sachin's comment above. Will reopen if we see the issue again.
Comment 6 Preethi Thomas 2011-08-16 12:09:58 UTC 
Closing with Community Release 15

pulp-0.0.223-4.
Comment 7 Preethi Thomas 2011-08-16 12:22:12 UTC 
Closing with Community Release 15

pulp-0.0.223-4.