Bug 693737 - API sync spam produces a yum related traceback in pulp.log
Summary: API sync spam produces a yum related traceback in pulp.log
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Pulp
Classification: Retired
Component: z_other
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: Sprint 22
Assignee: John Matthews
QA Contact: dgao
URL:
Whiteboard:
Depends On:
Blocks: verified-to-close
TreeView+ depends on / blocked
 
Reported: 2011-04-05 13:27 UTC by dgao
Modified: 2011-08-16 12:21 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-08-16 12:08:32 UTC
Embargoed:


Attachments (Terms of Use)

Description dgao 2011-04-05 13:27:32 UTC
The error shows up in the log but the API call stills produces a successful return. 

After n number of sync calls:

==> pulp.log <==
2011-04-05 09:19:24,702 [INFO][Thread-1606] sync() @ repo_sync.py:487 - YumSynchronizer reported 7 successes, 0 downloads, 0 errors
2011-04-05 09:19:24,710 [ERROR][Thread-1606] failed() @ task.py:207 - Task id:52ece4e8-5f87-11e0-8a6f-545200104e49, method_name:_sync:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 131, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo.py", line 1491, in _sync
    threads)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 121, in sync
    return synchronizer.add_packages_from_dir(repo_dir, repo, skip_dict)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 231, in add_packages_from_dir
    package_list = pulp.server.util.get_repo_packages(dir)
  File "/usr/lib/python2.7/site-packages/pulp/server/util.py", line 214, in get_repo_packages
    r.sack.populate(r, 'metadata', None, 0)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 163, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 221, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1224, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1413, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1405, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1395, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1370, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1198, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 985, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 808, in _getFile
    size=size
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 982, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 968, in retryfunc
    fo = PyCurlFileObject(url, filename, opts)
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1063, in __init__
    self._do_open()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1350, in _do_open
    self._set_opts()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1161, in _set_opts
    self.curl_obj.setopt(pycurl.NOPROGRESS, False)
error: cannot invoke setopt() - perform() is currently running

==> grinder.log <==
2011-04-05 09:19:24,712 [ERROR][Thread-1638] getRepoData() @ RepoFetch.py:153 - Unable to Fetch Repo data file filelists
2011-04-05 09:19:24,714 [ERROR][Thread-1638] getRepoData() @ RepoFetch.py:153 - Unable to Fetch Repo data file updateinfo
2011-04-05 09:19:24,717 [ERROR][Thread-1638] getRepoData() @ RepoFetch.py:153 - Unable to Fetch Repo data file other
2011-04-05 09:19:24,717 [ERROR][Thread-1638] getRepoData() @ RepoFetch.py:153 - Unable to Fetch Repo data file primary

==> pulp.log <==
2011-04-05 09:19:24,718 [ERROR][Thread-1638] failed() @ task.py:207 - Task id:5379de2e-5f87-11e0-8e0b-545200104e49, method_name:_sync:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/tasking/task.py", line 131, in run
    result = self.callable(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo.py", line 1491, in _sync
    threads)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 117, in sync
    progress_callback, max_speed, threads)
  File "/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py", line 484, in sync
    report = self.yum_repo_grinder.fetchYumRepo(store_path, callback=progress_callback)
  File "/usr/lib/python2.7/site-packages/grinder/RepoFetch.py", line 355, in fetchYumRepo
    self.yumFetch.getRepoData()
  File "/usr/lib/python2.7/site-packages/grinder/RepoFetch.py", line 157, in getRepoData
    shutil.copyfile(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 82, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: u'/var/lib/pulp//repos/pub/updates//repodata.new/repomd.xml'

Comment 1 dgao 2011-04-05 14:15:59 UTC
As a note: the sync call is executing on the same repo 100x w/ a yum feed to an internal repo that have 7pkgs.

Comment 2 John Matthews 2011-04-11 13:07:44 UTC
Issue described here was related to multiple syncs operating at the same time on the same exact repo.  We do not want to allow that behavior.

Below commit will limit only 1 sync per repo at the same time.
http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=33ef2654804e6f9a0b2bbd575e493f9a9e206f1a

Comment 3 Jeff Ortel 2011-04-11 14:08:09 UTC
build: 0.162

Comment 4 Jay Dobies 2011-04-13 19:23:02 UTC
Fixed in build 0.163.

Comment 5 dgao 2011-05-10 18:11:09 UTC
Changes to restrict one sync per repo at any given time eliminate this issue.

Comment 6 Preethi Thomas 2011-08-16 12:08:32 UTC
Closing with Community Release 15

pulp-0.0.223-4.

Comment 7 Preethi Thomas 2011-08-16 12:21:31 UTC
Closing with Community Release 15

pulp-0.0.223-4.


Note You need to log in before you can comment on or make changes to this bug.