Bug 807720

Summary: Waiting for post_sync_callback causes API unresponsive
Product: [Retired] Pulp Reporter: Ivan Necas <inecas>
Component: user-experienceAssignee: Jason Connor <jconnor>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: high Docs Contact:
Priority: unspecified    
Version: 1.0.0CC: jconnor, jsherril, mmccune
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-05-25 14:14:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
slowsever.rb - veeeery slow webserver for bug reproduction none

Description Ivan Necas 2012-03-28 14:49:08 UTC 
Created attachment 573363 [details]
slowsever.rb - veeeery slow webserver for bug reproduction

How reproducible:
On slow machines

Steps to Reproduce:
1. run attached script "slowserver.rb", it's webserver running on http://localhost:3003/ and waits for 5000 seconds after each requests
2. set in /etc/pulp/pulp.conf [server] section:
   post_sync_url = http://localhost:3003/
3. 
   repo=zoo
   pulp-admin repo create --id $repo\
                          --feed http://inecas.fedorapeople.org/fakerepo
                          --relativepath $repo
   pulp-admin repo sync --id $repo

4. wait until the "slowserver.rb" says "responding slowly..."
5. call
  pulp-admin repo status --id $repo

Actual results:
Waiting for the response from callback

Expected results:
Returning the status immediately

Additional info:
This issue causes a dead-lock situation in Katello:

Let's say only 1 process handles the whole workflow:

1. user synchronizes repository through Katello
2. user checks periodically for the status of the synchronization (and get's updated progress-bar)
3. handling the user request, Katello asks pulp for tasks status
4. in the mean-time, Pulp finishes the synchronization, calling post_sync_url
5. Pulp waits until the response from callback and blocks the status calls
6. katello process can not process the call-back unless it get's the response from the last pulp status request - katello waits for pulp
7. pulp can not process the status requests until the call-back finishes
8. we have a dead-lock of this workflow until request timeout from one of the calls
Comment 1 Ivan Necas 2012-03-28 14:49:51 UTC 
See also https://bugzilla.redhat.com/show_bug.cgi?id=807720
Comment 2 Jason Connor 2012-03-28 19:51:00 UTC 
I removed the wait for a response from katello so that pulp simply fires off the request and closes the connection.

Changes pushed in: 68389d55e97053049dfd9f5ab20301e07823c6fe
Comment 3 Justin Sherrill 2012-03-30 21:33:58 UTC 
I tested this changed and it seemed to work great on my workstation.  Ivan could you test it as well, and if it works for you we can go ahead with the new version of pulp.  No changes should be needed from the katello side of things, with this modification.
Comment 4 Jeff Ortel 2012-04-12 22:59:08 UTC 
build: 0.282
Comment 5 Preethi Thomas 2012-05-02 15:39:01 UTC 
verified fix in pulp 1.0 

[root@pulp-v1-server ~]# rpm -q pulp
pulp-1.0.4-1.el6.noarch
[root@pulp-v1-server ~]# 


[root@pulp-v1-server ~]# ./slowserver.rb 
[2012-05-02 11:29:15] INFO  WEBrick 1.3.1
[2012-05-02 11:29:15] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
[2012-05-02 11:29:15] WARN  TCPServer Error: Address already in use - bind(2)
[2012-05-02 11:29:15] INFO  WEBrick::HTTPServer#start: pid=3296 port=3003
responding slowly...
[root@pulp-v1-server ~]# repo=zoo
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo create --id $repo --feed http://repos.fedorapeople.org/repos/katello/katello/6Server/x86_64/ --relativepath $repo
Successfully created repository [ zoo ]

[root@pulp-v1-server ~]#  pulp-admin -u admin -p admin repo sync --id $repo
Sync for repository zoo started
Use "repo status" to check on the progress
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo status --id $repo
+------------------------------------------+
               Status for zoo
+------------------------------------------+
Repository: zoo
Number of Packages: 159
Last Sync: 2012-05-02 11:30:54-04:00

[root@pulp-v1-server ~]#
Comment 6 Preethi Thomas 2012-05-25 14:14:24 UTC 
Pulp v1.1 Release