Description of problem: After restarting the pulp-server service, I got "500 Internal server" with pulp-admin tool. Version-Release number of selected component (if applicable): pulp 0.168 rhui-tool 2.0.15 How reproducible: yesterday I installed 168 build and since then I got this issue 2 times. Steps to Reproduce: 1. Restarted the pulp-server service.. All services restarted without any failure. 2. then tried "pulp-admin -u admin -p admin repo list". This cmd is throwing internal server error Actual results: [root@dhcp193-131 ~]# pulp-admin -u admin -p admin repo list error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p> <p>More information about this error may be available in the server error log.</p> <hr> <address>Apache/2.2.15 (Red Hat) Server at dhcp193-131.pnq.redhat.com Port 443</address> </body></html> Expected results: pulp-admin tool should work without "internal server error" Additional info: Before restarting pulp-server I was trying CDS sync with pulp-admin tool and it was not working. And to get rid of this issue, I removed the mongod.lock file and restarted the pulp-server.. but no luck. From ssl_error_logs seems that might be this issue is related to sync scheduling. but I'm sure no sync was running when I got this Internal server error. [root@dhcp193-131 ~]# tail -f /var/log/httpd/error_log [Thu Apr 21 14:54:15 2011] [notice] Digest: done [Thu Apr 21 14:54:15 2011] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads. [Thu Apr 21 14:54:15 2011] [notice] mod_python: using mutex_directory /tmp [Thu Apr 21 14:54:15 2011] [warn] mod_wsgi: Compiled for Python/2.6.2. [Thu Apr 21 14:54:15 2011] [warn] mod_wsgi: Runtime using Python/2.6.5. [Thu Apr 21 14:54:15 2011] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.2 mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.15 OpenSSL/1.0.0-fips mod_wsgi/3.2 configured -- resuming normal operations [Thu Apr 21 14:54:22 2011] [error] /usr/lib/python2.6/site-packages/oauth2/__init__.py:29: DeprecationWarning: the sha module is deprecated; use the hashlib module instead [Thu Apr 21 14:54:22 2011] [error] import sha [Thu Apr 21 14:54:22 2011] [error] /usr/lib/python2.6/site-packages/httplib2/__init__.py:29: DeprecationWarning: the md5 module is deprecated; use hashlib instead [Thu Apr 21 14:54:22 2011] [error] import md5 [root@dhcp193-131 ~]# tail -f /var/log/httpd/ssl_error_log [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] _init_repo_scheduled_syncs() [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 296, in _init_repo_scheduled_syncs [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] _add_repo_scheduled_sync_task(repo) [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 171, in _add_repo_scheduled_sync_task [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] task.scheduler = schedule_to_scheduler(repo['sync_schedule']) [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] File "/usr/lib/python2.6/site-packages/pulp/server/api/scheduled_sync.py", line 141, in schedule_to_scheduler [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] return IntervalScheduler(interval, start_time, runs) [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] File "/usr/lib/python2.6/site-packages/pulp/server/tasking/scheduler.py", line 123, in __init__ [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] str(start_time), str(interval)) [Thu Apr 21 14:54:22 2011] [error] [client 10.65.193.131] TypeError: not enough arguments for format string tail -f /var/log/pulp/pulp.log 2011-04-21 15:02:59,065 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhel61'), (u'id', u'rhel61'), (u'sync_in_progress', False)]) 2011-04-21 15:02:59,066 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhel-server-6-updates-srpms-6Server-x86_64'), (u'id', u'rhel-server-6-updates-srpms-6Server-x86_64'), (u'sync_in_progress', False)]) 2011-04-21 15:02:59,067 [ERROR][Dummy-6] clear_all_sync_in_progress() @ repo.py:73 - r = SON([(u'_id', u'rhui-1.2-5Server-x86_64'), (u'id', u'rhui-1.2-5Server-x86_64'), (u'sync_in_progress', False)]) root@dhcp193-131 ~]# tail -f /var/log/pulp/db.log migration to data model version 5 started migration to data model version 5 complete migration to data model version 7 started migration to data model version 7 complete migration to data model version 8 started migration to data model version 8 complete migration to data model version 9 started migration to data model version 9 complete migration to data model version 10 started migration to data model version 10 complete
Facing same issue, on a totally different setup.( pulp-168, pulp-2.0.15) Faced it earlier, even with build pulp-166 and pulp-2.0.15
missing parens around the strings for substitution fix pushed in master and rhui branches: de67c69
build: 0.170
I installed the build 0.170 and didn't face this issue as of now, however I'll keep an eye on this for couple of more days.
moving to verified as per Sachin's comment above. Will reopen if we see the issue again.
Closing with Community Release 15 pulp-0.0.223-4.