Bug 807332
Summary: | provider sync of two repos in diff. products fails with huge dump | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Satellite | Reporter: | Garik Khachikyan <gkhachik> | ||||
Component: | API | Assignee: | Bryan Kearney <bkearney> | ||||
Status: | CLOSED CURRENTRELEASE | QA Contact: | Corey Welton <cwelton> | ||||
Severity: | urgent | Docs Contact: | |||||
Priority: | urgent | ||||||
Version: | 6.0.1 | CC: | cpelland, cwelton, inecas, jconnor, jortel, jsherril, mkoci | ||||
Target Milestone: | Unspecified | Keywords: | Regression | ||||
Target Release: | Unused | ||||||
Hardware: | Unspecified | ||||||
OS: | Unspecified | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2012-08-22 18:31:55 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
Garik Khachikyan
2012-03-27 14:32:22 UTC
Could you provide the stack trace you're seeing. I could not reproduce using the steps. I was able to reproduce with this conditions: 1. setting post_sync_url in /etc/pulp/pulp.conf 2. running katello in devel mode (or running it on a slower machine) This didn't occurred on my not-virtualized laptop. So it seems very likely a dead-lock, that ends up in request time-out when requesting the sync status. Better error handling commited in 26ee22ccaf2da68b2039d6d9015d542b1dcb7e37 . The timeout problem is still there. The root cause of the problem is described in Pulp bug I've filed [1] [1] https://bugzilla.redhat.com/show_bug.cgi?id=807720 Spoke with jconner and he tried a modification to pulp: http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=68389d55e97053049dfd9f5ab20301e07823c6fe and basically now pulp will not wait for a response before closing the connection. I tested with ivan's simple server script and now the pulp-admin command returns as expected. However, when doing this with a katello instance, apache does not seem to forward the request through to the thin processes (it simply returns a 403). So we either need to find another solution, find a way to force apache to forward the request, or instead have pulp go directly to a think process (port 5000 for example), instead of apache. Thoughts? I'm not a big fan of pointing it directly to thin process. I don't have much experience with setting up apache this way. If Pulp could postpone the callback somewhere, where isn't the lock that causes the status calls not being served. It's strange another calls, like `repo list` work ok, even when waiting for the callback. Ivan, I'm not a huge fan of that either but this is a temporary solution (pulp is adding an item to their backlog to come up with a more permanent solution). Speaking with jconner it seems to that it would take a good deal more work to move something like this out of the main task queue. So while I don't like it either, I'm ok with pointing to port 5000 for v1. We would also need to change the authorization logic, as there would be no HTTP_X_FORWARDED_FOR header. Created attachment 573926 [details]
patch against repo_sync.py in pulp-1.0.0-8 moving callback to separate thread
What if we moved the callback request into separate thread and wait for the response from there, so that it wouldn't block the original process? I've tried a patch against repo_sync.py in pulp-1.0.0-8 (see attachment) and it seemed to work. @jason - any thoughts on that?
Ok, I've pushed a modified version of the supplied patch that fires the actual post request off in a separate thread. The fix is in two commits: 2c76534652cd59082f9cc1589e11abdef3f4a6a2 1180e17fa46b96018ec4e58af3972c5e597a14df build: 1.0.3 [root@se-blade ~]# katello --username admin --password admin provider synchronize --name Provider1 Provider [ Provider1 ] synchronized QA Verified, following steps above and subsequently confirming in UI. Tested on brew build of 0.1.309-1.el6 which has package pulp-1.0.4-1.el6.noarch getting rid of 6.0.0 version since that doesn't exist |