Bug 714782

Summary: creating a repo with same relativepath as existing repo while syncing causes a crash
Product: [Retired] Pulp Reporter: Pradeep Kilambi <pkilambi>
Component: user-experienceAssignee: Pradeep Kilambi <pkilambi>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: medium    
Version: 1.0.0CC: scollier, skarmark
Target Milestone: ---Keywords: Triaged
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-02-24 20:13:19 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Pradeep Kilambi 2011-06-20 18:10:42 UTC 
Description of problem:
Creating a repo with mataching relative paths as existing repo, while the existing repo sync is in progress causes a crash. This is on f15(not sure if this matters). Either way we probably shouldnt allow users to create repos with overlapping relative paths..

1. Create a repo A
$ sudo pulp-admin repo create --id=test_el6_updates --feed=https://cdn.redhat.com/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/ --feed_ca=/home/pkilambi/rcm-debug/cdn.redhat.com-chain.crt --feed_cert=/home/pkilambi/rcm-debug.pem
Successfully created repository [ test_el6_updates ]

2. start a sync
$ sudo pulp-admin repo sync -F --id=test_el6_updates
Sync for repository test_el6_updates started
Step: Downloading Items or Verifying
[======================   ] 89%
Rpms: 4512/5043
Total: 4512/5043 items
error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.17 (Fedora) Server at localhost.localdomain Port 443</address>
</body></html>


3. create repo B with same url as repo A (this process hangs for a while before both fail with 500)
$ sudo pulp-admin repo create --id=test_el6_updates_5 --feed=https://cdn.redhat.com/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/ --feed_ca=/home/pkilambi/rcm-debug/cdn.redhat.com-chain.crt --feed_cert=/home/pkilambi/rcm-debug.pem
error: operation failed: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator,
 root@localhost and inform them of the time the error occurred,
and anything you might have done that may have
caused the error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.2.17 (Fedora) Server at localhost.localdomain Port 443</address>
</body></html>


IN /var/log/httpd/error_log:
[Mon Jun 20 13:43:17 2011] [error] 2011-06-20 13:43:17,773 pulp.server.util:INFO: started repo metadata update
[Mon Jun 20 13:52:30 2011] [error] 2011-06-20 13:52:30,769 pulp.server.tasking.task:INFO: Task canceled: Task 21c261d9-9b62-11e0-984c-
0023ae793dbc: _sync(test_el6_release, synchronizer=<pulp.server.api.synchronizers.YumSynchronizer object at 0x7f8c00615a10>, skip_dict
={}, max_speed=None, threads=None)
*** glibc detected *** (wsgi:pulp)    : double free or corruption (!prev): 0x00007f8b98005730 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x32eea76fea)[0x7f8c35001fea]
/usr/lib64/libnsspem.so(+0x19253)[0x7f8c049c4253]
/usr/lib64/libnsspem.so(+0xa94c)[0x7f8c049b594c]
/usr/lib64/libnsspem.so(+0x912c)[0x7f8c049b412c]
/usr/lib64/libnsspem.so(+0xa325)[0x7f8c049b5325]
/usr/lib64/libnsspem.so(+0x184ae)[0x7f8c049c34ae]
...
7f8c3895a000-7f8c39db9000 rw-p 00000000 00:00 0                          [heap]
7fffbe789000-7fffbe7c6000 rw-p 00000000 00:00 0                          [stack]
7fffbe7ff000-7fffbe800000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
[Mon Jun 20 13:52:52 2011] [notice] child pid 8533 exit signal Aborted (6)
[Mon Jun 20 13:53:19 2011] [error] 2011-06-20 13:53:19,498 pulp.server.async:INFO: Task reply handler, started.
[Mon Jun 20 13:53:21 2011] [error] Exception in thread asynctaskreplyqueue:
[Mon Jun 20 13:53:21 2011] [error] Traceback (most recent call last):
[Mon Jun 20 13:53:21 2011] [error]   File "/usr/lib64/python2.7/threading.py", line 530, in __bootstrap_inner
[Mon Jun 20 13:53:21 2011] [error]     self.run()
[Mon Jun 20 13:53:21 2011] [error]   File "/home/pkilambi/code/gofer/src/gofer/messaging/consumer.py", line 64, in run
[Mon Jun 20 13:53:21 2011] [error]     receiver = self.__open()
[Mon Jun 20 13:53:21 2011] [error]   File "/home/pkilambi/code/gofer/src/gofer/messaging/consumer.py", line 94, in __open
[Mon Jun 20 13:53:21 2011] [error]     return ssn.receiver(addr)
[Mon Jun 20 13:53:21 2011] [error]   File "<string>", line 6, in receiver
[Mon Jun 20 13:53:21 2011] [error]   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 605, in receiver
[Mon Jun 20 13:53:21 2011] [error]     receiver.close()
[Mon Jun 20 13:53:21 2011] [error]   File "<string>", line 6, in close
[Mon Jun 20 13:53:21 2011] [error]   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 1037, in close
[Mon Jun 20 13:53:21 2011] [error]     if not self.session._ewait(lambda: self.closed, timeout=timeout):
[Mon Jun 20 13:53:21 2011] [error]   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 554, in _ewait
[Mon Jun 20 13:53:21 2011] [error]     self.check_error()
[Mon Jun 20 13:53:21 2011] [error]   File "/usr/lib/python2.7/site-packages/qpid/messaging/endpoints.py", line 543, in check_error
[Mon Jun 20 13:53:21 2011] [error]     raise self.error
[Mon Jun 20 13:53:21 2011] [error] ReceiverError: resource-locked: Queue asynctaskreplyqueue has an exclusive consumer. No more consumers allowed. (qpid/broker/Queue.cpp:385)(405)
[Mon Jun 20 13:53:21 2011] [error]
Comment 1 scollier 2011-11-30 23:19:04 UTC 
Similar error using:

Red Hat Enterprise Linux Server release 6.1 (Santiago)
Linux cf-se4.cloud.lab.eng.bos.redhat.com 2.6.32-131.17.1.el6.x86_64 #1 SMP Thu Sep 29 10:24:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
PyYAML-3.09-14.el6_1.x86_64
facter-1.5.9-1.el6.noarch
js-1.70-12.el6_0.x86_64
mongodb-1.6.4-3.el6_0.x86_64
mongodb-server-1.6.4-3.el6_0.x86_64
puppet-2.6.11-1.el6_1.noarch
pymongo-1.9-8.el6_1.x86_64
katello-0.1.111-2.el6.x86_64
katello-all-0.1.111-2.el6.x86_64
katello-cli-0.1.13-1.el6.noarch
katello-configure-0.1.16-2.el6.noarch
m2crypto-0.21.1.pulp-5.el6.x86_64
katello-glue-pulp-0.1.111-2.el6.x86_64
pulp-common-0.0.248-3.el6.noarch
mod_wsgi-3.2-6.pulp.el6.x86_64
pulp-0.0.248-3.el6.noarch

My issue was pulp timeouts in a cloudforms environment.  When i would promote a changeset with 3 "products", i would experience the timeout.  I had the default organization and a custom organization (that i was working in).

saw this in /var/log/httpd/error_log:

*** glibc detected *** (wsgi:pulp)    : double free or corruption (fasttop): 0x00007f38ac300230 ***

and katello ping showed:

# katello --username=xxxxx --password=xxxxxx ping
--------------------------------------------------------------------------------
                                                             Katello Status

 Status   Service  Result   Duration  Message                                                  
--------------------------------------------------------------------------------
 fail                                                                                                   
          candlepin        ok       26ms                                                                
          candlepin_auth   ok       12ms                                                                
          pulp             fail                Request Timeout                                          
          pulp_auth        fail                Pulp::PulpPing: Request Timeout  (GET /pulp/api/users/)
Comment 2 Pradeep Kilambi 2011-12-08 14:50:08 UTC 
This has been resolved with relative path restrictions in pulp.
Comment 3 Jeff Ortel 2011-12-15 20:18:00 UTC 
build: 0.255
Comment 4 Preethi Thomas 2011-12-22 13:20:23 UTC 
verified

[root@preethi ~]# rpm -q pulp
pulp-0.0.255-1.fc15.noarch
[root@preethi ~]# pulp-admin repo create --id=test_el6_updates --feed=https://cdn.redhat.com/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/
Successfully created repository [ test_el6_updates ]

[root@preethi ~]# 
[root@preethi ~]# 
[root@preethi ~]# 
[root@preethi ~]# pulp-admin repo create --id=test_el6_updates --feed=https://cdn.redhat.com/content/dist/rhel/rhui/server-6/updates/6Server/x86_64/os/
error:  error: operation failed: A repository with the id, test_el6_updates, already exists
[root@preethi ~]#
Comment 5 Preethi Thomas 2012-02-24 20:13:19 UTC 
Pulp v1.0 is released
Closed Current Release.