Bug 807720 - Waiting for post_sync_callback causes API unresponsive
Waiting for post_sync_callback causes API unresponsive
Product: Pulp
Classification: Community
Component: user-experience (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Jason Connor
Preethi Thomas
Depends On:
  Show dependency treegraph
Reported: 2012-03-28 10:49 EDT by Ivan Necas
Modified: 2014-03-30 21:39 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2012-05-25 10:14:24 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
slowsever.rb - veeeery slow webserver for bug reproduction (329 bytes, application/x-ruby)
2012-03-28 10:49 EDT, Ivan Necas
no flags Details

  None (edit)
Description Ivan Necas 2012-03-28 10:49:08 EDT
Created attachment 573363 [details]
slowsever.rb - veeeery slow webserver for bug reproduction

How reproducible:
On slow machines

Steps to Reproduce:
1. run attached script "slowserver.rb", it's webserver running on http://localhost:3003/ and waits for 5000 seconds after each requests
2. set in /etc/pulp/pulp.conf [server] section:
   post_sync_url = http://localhost:3003/
   pulp-admin repo create --id $repo\
                          --feed http://inecas.fedorapeople.org/fakerepo
                          --relativepath $repo
   pulp-admin repo sync --id $repo

4. wait until the "slowserver.rb" says "responding slowly..."
5. call
  pulp-admin repo status --id $repo

Actual results:
Waiting for the response from callback

Expected results:
Returning the status immediately

Additional info:
This issue causes a dead-lock situation in Katello:

Let's say only 1 process handles the whole workflow:

1. user synchronizes repository through Katello
2. user checks periodically for the status of the synchronization (and get's updated progress-bar)
3. handling the user request, Katello asks pulp for tasks status
4. in the mean-time, Pulp finishes the synchronization, calling post_sync_url
5. Pulp waits until the response from callback and blocks the status calls
6. katello process can not process the call-back unless it get's the response from the last pulp status request - katello waits for pulp
7. pulp can not process the status requests until the call-back finishes
8. we have a dead-lock of this workflow until request timeout from one of the calls
Comment 1 Ivan Necas 2012-03-28 10:49:51 EDT
See also https://bugzilla.redhat.com/show_bug.cgi?id=807720
Comment 2 Jason Connor 2012-03-28 15:51:00 EDT
I removed the wait for a response from katello so that pulp simply fires off the request and closes the connection.

Changes pushed in: 68389d55e97053049dfd9f5ab20301e07823c6fe
Comment 3 Justin Sherrill 2012-03-30 17:33:58 EDT
I tested this changed and it seemed to work great on my workstation.  Ivan could you test it as well, and if it works for you we can go ahead with the new version of pulp.  No changes should be needed from the katello side of things, with this modification.
Comment 4 Jeff Ortel 2012-04-12 18:59:08 EDT
build: 0.282
Comment 5 Preethi Thomas 2012-05-02 11:39:01 EDT
verified fix in pulp 1.0 

[root@pulp-v1-server ~]# rpm -q pulp
[root@pulp-v1-server ~]# 

[root@pulp-v1-server ~]# ./slowserver.rb 
[2012-05-02 11:29:15] INFO  WEBrick 1.3.1
[2012-05-02 11:29:15] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
[2012-05-02 11:29:15] WARN  TCPServer Error: Address already in use - bind(2)
[2012-05-02 11:29:15] INFO  WEBrick::HTTPServer#start: pid=3296 port=3003
responding slowly...
[root@pulp-v1-server ~]# repo=zoo
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo create --id $repo --feed http://repos.fedorapeople.org/repos/katello/katello/6Server/x86_64/ --relativepath $repo
Successfully created repository [ zoo ]

[root@pulp-v1-server ~]#  pulp-admin -u admin -p admin repo sync --id $repo
Sync for repository zoo started
Use "repo status" to check on the progress
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo status --id $repo
               Status for zoo
Repository: zoo
Number of Packages: 159
Last Sync: 2012-05-02 11:30:54-04:00

[root@pulp-v1-server ~]#
Comment 6 Preethi Thomas 2012-05-25 10:14:24 EDT
Pulp v1.1 Release

Note You need to log in before you can comment on or make changes to this bug.