Red Hat Bugzilla – Bug 807332
provider sync of two repos in diff. products fails with huge dump
Last modified: 2015-01-04 16:59:16 EST
Description of problem:
Recent Katello changes brought the system to be failing on sync of provider that has two different products where a repo is defined for each.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
client remember --option org --value ACME_Corporation
provider create --name Provider1
product create --name zoo --provider Provider1
repo create --name zoo --product zoo --url http://inecas.fedorapeople.org/fakerepos/zoo3/
product create --name pulp --provider Provider1
repo create --name pulp --product pulp --url http://repos.fedorapeople.org/repos/pulp/pulp/v1/stable/6Server/x86_64/
provider synchronize --name Provider1
waiting for a while with 0% progress there is dumping really a huge error log to console.
no errors, no issues on syncing
Individual repo sync however works!
Could you provide the stack trace you're seeing. I could not reproduce using the steps.
I was able to reproduce with this conditions:
1. setting post_sync_url in /etc/pulp/pulp.conf
2. running katello in devel mode (or running it on a slower machine)
This didn't occurred on my not-virtualized laptop. So it seems very likely a dead-lock, that ends up in request time-out when requesting the sync status.
Better error handling commited in 26ee22ccaf2da68b2039d6d9015d542b1dcb7e37 . The timeout problem is still there.
The root cause of the problem is described in Pulp bug I've filed 
Spoke with jconner and he tried a modification to pulp:
and basically now pulp will not wait for a response before closing the connection.
I tested with ivan's simple server script and now the pulp-admin command returns as expected. However, when doing this with a katello instance, apache does not seem to forward the request through to the thin processes (it simply returns a 403).
So we either need to find another solution, find a way to force apache to forward the request, or instead have pulp go directly to a think process (port 5000 for example), instead of apache.
I'm not a big fan of pointing it directly to thin process. I don't have much experience with setting up apache this way. If Pulp could postpone the callback somewhere, where isn't the lock that causes the status calls not being served. It's strange another calls, like `repo list` work ok, even when waiting for the callback.
Ivan, I'm not a huge fan of that either but this is a temporary solution (pulp is adding an item to their backlog to come up with a more permanent solution). Speaking with jconner it seems to that it would take a good deal more work to move something like this out of the main task queue. So while I don't like it either, I'm ok with pointing to port 5000 for v1.
We would also need to change the authorization logic, as there would be no HTTP_X_FORWARDED_FOR header.
Created attachment 573926 [details]
patch against repo_sync.py in pulp-1.0.0-8 moving callback to separate thread
What if we moved the callback request into separate thread and wait for the response from there, so that it wouldn't block the original process? I've tried a patch against repo_sync.py in pulp-1.0.0-8 (see attachment) and it seemed to work. @jason - any thoughts on that?
Ok, I've pushed a modified version of the supplied patch that fires the actual post request off in a separate thread.
The fix is in two commits:
[root@se-blade ~]# katello --username admin --password admin provider synchronize --name Provider1
Provider [ Provider1 ] synchronized
QA Verified, following steps above and subsequently confirming in UI.
Tested on brew build of 0.1.309-1.el6 which has package pulp-1.0.4-1.el6.noarch
getting rid of 6.0.0 version since that doesn't exist