Bug 807720 - Waiting for post_sync_callback causes API unresponsive
Summary: Waiting for post_sync_callback causes API unresponsive
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: user-experience
Version: 1.0.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Jason Connor
QA Contact: Preethi Thomas
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-03-28 14:49 UTC by Ivan Necas
Modified: 2014-03-31 01:39 UTC (History)
3 users (show)

Fixed In Version:
Clone Of:
Environment:
Last Closed: 2012-05-25 14:14:24 UTC
Embargoed:


Attachments (Terms of Use)
slowsever.rb - veeeery slow webserver for bug reproduction (329 bytes, application/x-ruby)
2012-03-28 14:49 UTC, Ivan Necas
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 807720 0 unspecified CLOSED Waiting for post_sync_callback causes API unresponsive 2021-02-22 00:41:40 UTC

Internal Links: 807720 816188

Description Ivan Necas 2012-03-28 14:49:08 UTC
Created attachment 573363 [details]
slowsever.rb - veeeery slow webserver for bug reproduction

How reproducible:
On slow machines

Steps to Reproduce:
1. run attached script "slowserver.rb", it's webserver running on http://localhost:3003/ and waits for 5000 seconds after each requests
2. set in /etc/pulp/pulp.conf [server] section:
   post_sync_url = http://localhost:3003/
3. 
   repo=zoo
   pulp-admin repo create --id $repo\
                          --feed http://inecas.fedorapeople.org/fakerepo
                          --relativepath $repo
   pulp-admin repo sync --id $repo

4. wait until the "slowserver.rb" says "responding slowly..."
5. call
  pulp-admin repo status --id $repo

Actual results:
Waiting for the response from callback

Expected results:
Returning the status immediately

Additional info:
This issue causes a dead-lock situation in Katello:

Let's say only 1 process handles the whole workflow:

1. user synchronizes repository through Katello
2. user checks periodically for the status of the synchronization (and get's updated progress-bar)
3. handling the user request, Katello asks pulp for tasks status
4. in the mean-time, Pulp finishes the synchronization, calling post_sync_url
5. Pulp waits until the response from callback and blocks the status calls
6. katello process can not process the call-back unless it get's the response from the last pulp status request - katello waits for pulp
7. pulp can not process the status requests until the call-back finishes
8. we have a dead-lock of this workflow until request timeout from one of the calls

Comment 1 Ivan Necas 2012-03-28 14:49:51 UTC
See also https://bugzilla.redhat.com/show_bug.cgi?id=807720

Comment 2 Jason Connor 2012-03-28 19:51:00 UTC
I removed the wait for a response from katello so that pulp simply fires off the request and closes the connection.

Changes pushed in: 68389d55e97053049dfd9f5ab20301e07823c6fe

Comment 3 Justin Sherrill 2012-03-30 21:33:58 UTC
I tested this changed and it seemed to work great on my workstation.  Ivan could you test it as well, and if it works for you we can go ahead with the new version of pulp.  No changes should be needed from the katello side of things, with this modification.

Comment 4 Jeff Ortel 2012-04-12 22:59:08 UTC
build: 0.282

Comment 5 Preethi Thomas 2012-05-02 15:39:01 UTC
verified fix in pulp 1.0 

[root@pulp-v1-server ~]# rpm -q pulp
pulp-1.0.4-1.el6.noarch
[root@pulp-v1-server ~]# 


[root@pulp-v1-server ~]# ./slowserver.rb 
[2012-05-02 11:29:15] INFO  WEBrick 1.3.1
[2012-05-02 11:29:15] INFO  ruby 1.8.7 (2011-06-30) [x86_64-linux]
[2012-05-02 11:29:15] WARN  TCPServer Error: Address already in use - bind(2)
[2012-05-02 11:29:15] INFO  WEBrick::HTTPServer#start: pid=3296 port=3003
responding slowly...
[root@pulp-v1-server ~]# repo=zoo
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo create --id $repo --feed http://repos.fedorapeople.org/repos/katello/katello/6Server/x86_64/ --relativepath $repo
Successfully created repository [ zoo ]

[root@pulp-v1-server ~]#  pulp-admin -u admin -p admin repo sync --id $repo
Sync for repository zoo started
Use "repo status" to check on the progress
[root@pulp-v1-server ~]# pulp-admin -u admin -p admin repo status --id $repo
+------------------------------------------+
               Status for zoo
+------------------------------------------+
Repository: zoo
Number of Packages: 159
Last Sync: 2012-05-02 11:30:54-04:00

[root@pulp-v1-server ~]#

Comment 6 Preethi Thomas 2012-05-25 14:14:24 UTC
Pulp v1.1 Release


Note You need to log in before you can comment on or make changes to this bug.