Red Hat Bugzilla – Bug 807720
Waiting for post_sync_callback causes API unresponsive
Last modified: 2014-03-30 21:39:46 EDT
Created attachment 573363 [details] slowsever.rb - veeeery slow webserver for bug reproduction How reproducible: On slow machines Steps to Reproduce: 1. run attached script "slowserver.rb", it's webserver running on http://localhost:3003/ and waits for 5000 seconds after each requests 2. set in /etc/pulp/pulp.conf [server] section: post_sync_url = http://localhost:3003/ 3. repo=zoo pulp-admin repo create --id $repo\ --feed http://inecas.fedorapeople.org/fakerepo --relativepath $repo pulp-admin repo sync --id $repo 4. wait until the "slowserver.rb" says "responding slowly..." 5. call pulp-admin repo status --id $repo Actual results: Waiting for the response from callback Expected results: Returning the status immediately Additional info: This issue causes a dead-lock situation in Katello: Let's say only 1 process handles the whole workflow: 1. user synchronizes repository through Katello 2. user checks periodically for the status of the synchronization (and get's updated progress-bar) 3. handling the user request, Katello asks pulp for tasks status 4. in the mean-time, Pulp finishes the synchronization, calling post_sync_url 5. Pulp waits until the response from callback and blocks the status calls 6. katello process can not process the call-back unless it get's the response from the last pulp status request - katello waits for pulp 7. pulp can not process the status requests until the call-back finishes 8. we have a dead-lock of this workflow until request timeout from one of the calls
See also https://bugzilla.redhat.com/show_bug.cgi?id=807720
I removed the wait for a response from katello so that pulp simply fires off the request and closes the connection. Changes pushed in: 68389d55e97053049dfd9f5ab20301e07823c6fe
I tested this changed and it seemed to work great on my workstation. Ivan could you test it as well, and if it works for you we can go ahead with the new version of pulp. No changes should be needed from the katello side of things, with this modification.
build: 0.282
verified fix in pulp 1.0 [root@pulp-v1-server ~]# rpm -q pulp pulp-1.0.4-1.el6.noarch [root@pulp-v1-server ~]# [root@pulp-v1-server ~]# ./slowserver.rb [2012-05-02 11:29:15] INFO WEBrick 1.3.1 [2012-05-02 11:29:15] INFO ruby 1.8.7 (2011-06-30) [x86_64-linux] [2012-05-02 11:29:15] WARN TCPServer Error: Address already in use - bind(2) [2012-05-02 11:29:15] INFO WEBrick::HTTPServer#start: pid=3296 port=3003 responding slowly... [root@pulp-v1-server ~]# repo=zoo [root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo create --id $repo --feed http://repos.fedorapeople.org/repos/katello/katello/6Server/x86_64/ --relativepath $repo Successfully created repository [ zoo ] [root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo sync --id $repo Sync for repository zoo started Use "repo status" to check on the progress [root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo status --id $repo +------------------------------------------+ Status for zoo +------------------------------------------+ Repository: zoo Number of Packages: 159 Last Sync: 2012-05-02 11:30:54-04:00 [root@pulp-v1-server ~]#
Pulp v1.1 Release