Bug 350391

Uh... the error seems pretty clear to me, 

yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el4-ecetra-x86_64. Please verify its path and try again
mash failed in /data/mash/el4-ecetra

Is there actually any repodata in "/data/mash/el4-ecetra" ?  Perhaps createrepo
is failing because it's trying to use -d and that doesn't exist in RHEL5?

If it would be soooo easy Jesse - I would not bug you :-)

I have createrepo 0.4.10 here... That has -d. And yes, there actually is repodata:
root@treasure ~# find /data/mash/el4-ecetra -type f|grep -v rpm$
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/primary.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/filelists.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/other.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/repomd.xml
/data/mash/el4-ecetra/i386/os/RedHat/repodata/primary.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/filelists.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/other.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/repomd.xml
/data/mash/el4-ecetra/source/SRPMS/repodata/primary.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/filelists.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/other.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/repomd.xml

Maybe also good info:
If I add a "print self.conf.cachedir" right after "# Set attributes not from the
config file" in def readRepoConfig from yum's __init__.py

The following I see when running mash:
/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache

Ah. And if I do it only for i386 (setting arches = i386) it works. I try this
with x86_64 now as well.

And if I do arches = x86_64 the problem is there again. :-(

ping.

What happens if you run this in a chroot of rawhide packages?  When we compose
rawhide we don't actually run mash directly on the host, we create (or update) a
chroot of the tree about to be mashed and then use mash from inside there.

Inside the chroot. The same.

mock-chroot> mash -o /data el5-ecetra     
Getting package lists for el5-ecetra...
Sorting packages...
Checking signatures...

< ... >

Writing out files for /data/el5-ecetra/i386/os/RedHat...
Writing out files for /data/el5-ecetra/x86_64/os/RedHat...
Writing out files for /data/el5-ecetra/source/SRPMS...
Waiting for createrepo to finish...
Previous repo file missing: /data/el5-ecetra/source/SRPMS/repodata/primary.xml.gz
Previous repo file missing:
/data/el5-ecetra/x86_64/os/RedHat/repodata/primary.xml.gz
Previous repo file missing: /data/el5-ecetra/i386/os/RedHat/repodata/primary.xml.gz
Resolving multilib for arch x86_64 using method all
Waiting for depsolve and createrepo to finish...
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in <module>
    main()
  File "/usr/bin/mash", line 70, in main
    rc = themash.doMultilib()
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 409, in doMultilib
    pid = self.doDepSolveAndMultilib(arch, cachedir)
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 353, in
doDepSolveAndMultilib
    yumbase.doTsSetup()
  File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 63, in doTsSetup
    return self._getTs()
  File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 77, in _getTs
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 526, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 381, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.5/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 142, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 199, in
_check_db_version
    if repo.repoXML.repoData.has_key(mdtype):
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 769, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 765, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el5-ecetra-x86_64. Please verify its path and try again
mash failed in /data/el5-ecetra


mock-chroot> rpm -q createrepo python mash
createrepo-0.4.10-1.fc8
python-2.5.1-14.fc8
mash-0.2.8-1.fc8

What happens if you manually run createrepo with the same options mash would use
in those paths (from the chroot)

Createrepo doesn't spot any error and produces the repodata just fine.

mock-chroot> find | grep -i repoda                
./x86_64/os/RedHat/repodata
./x86_64/os/RedHat/repodata/other.sqlite.bz2
./x86_64/os/RedHat/repodata/filelists.sqlite.bz2
./x86_64/os/RedHat/repodata/other.xml.gz
./x86_64/os/RedHat/repodata/filelists.xml.gz
./x86_64/os/RedHat/repodata/repomd.xml
./x86_64/os/RedHat/repodata/primary.sqlite.bz2
./x86_64/os/RedHat/repodata/primary.xml.gz
./source/SRPMS/repodata
./source/SRPMS/repodata/other.sqlite.bz2
./source/SRPMS/repodata/filelists.sqlite.bz2
./source/SRPMS/repodata/other.xml.gz
./source/SRPMS/repodata/filelists.xml.gz
./source/SRPMS/repodata/repomd.xml
./source/SRPMS/repodata/primary.sqlite.bz2
./source/SRPMS/repodata/primary.xml.gz
./i386/os/RedHat/repodata
./i386/os/RedHat/repodata/other.sqlite.bz2
./i386/os/RedHat/repodata/filelists.sqlite.bz2
./i386/os/RedHat/repodata/other.xml.gz
./i386/os/RedHat/repodata/filelists.xml.gz
./i386/os/RedHat/repodata/repomd.xml
./i386/os/RedHat/repodata/primary.sqlite.bz2
./i386/os/RedHat/repodata/primary.xml.gz



Well. If I run mash then, it removes the full tree /data/el5-ecetra and also the
old repodata of course... :-/

Right, because mash creates repodata on its own to make use of.

That's very strange. I created a chroot with mock from latest and greatest
fedora devel and the problem is still the same. I'll now try f7 chroot.

ok. f7 chroot does the same error.

Bill, your turn.. (:

What versions of:
mash
createrepo
yum
?

(In reply to comment #3)
> Maybe also good info:
> If I add a "print self.conf.cachedir" right after "# Set attributes not from the
> config file" in def readRepoConfig from yum's __init__.py
> 
> The following I see when running mash:
>
/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache

OK, so the code in question is:

        tmpdir = "/tmp/mash-%s/" % (self.config.name,)
        tmproot = os.path.join(tmpdir, "%s-%s.tmp" % (self.config.name, arch))
        yumcachedir = os.path.join(tmproot, "yumcache")

I'm not seeing how you'd get a duplicate path in there.

Created attachment 261351 [details]
patch

Oh wait, cachedir is relative to installroot. Does this help at all?

Created attachment 261371 [details]
patch, this time without syntax errors

Hm. And why does that work on f7 then!?

However:

/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
That's better now. But I still get this:

Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in ?
    main()
  File "/usr/bin/mash", line 70, in main
    rc = themash.doMultilib()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 408, in doMultilib
    pid = self.doDepSolveAndMultilib(arch, cachedir)
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 352, in
doDepSolveAndMultilib
    yumbase.doTsSetup()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 63, in doTsSetup
    return self._getTs()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 77, in _getTs
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 527, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 382, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 142, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 199, in
_check_db_version
    if repo.repoXML.repoData.has_key(mdtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 769, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 765, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el4-ecetra-x86_64. Please verify its path and try again
mash failed in /data/mash/el4-ecetra

(In reply to comment #15)
> What versions of:
> mash
> createrepo
> yum
> ?

of@treasure ~$ rpm -q mash createrepo yum
mash-0.2.8-1
createrepo-0.4.10-1.5EL
yum-3.2.7-1

All compiled myself on that system (took rawhide pkgs).

Created attachment 261401 [details]
untested patch

Try this, may take some massaging. You'll need to adjust your config to set the
repodata_path.

Created attachment 261641 [details]
updated patch

This *should* solve your problem. (You'll need to set repodata_path in your
mash config).

Let me know how well it works for you.

AFAICS, this WorksForMe (tm). :-)

Bill, will this fix go upstrem as well?

Thx,
 Oliver

Fixed in 0.2.10-1.

There's another bug, that I forgot to tell you:
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in <module>
    main()
  File "/usr/bin/mash", line 63, in main
    rc = themash.doCompose()
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 232, in doCompose
    key = pkg['sigkey'].lower()
AttributeError: 'NoneType' object has no attribute 'lower'
mash failed in /data/mash/development


This from another machine, mash-ing the AlphaCore tree; However. That also
happens on EL5 when mashing rpms without signatures...

A simple check with pkg['signkey'] should do the trick...

If you prefer a new bz#, please let me know.

Actually, sigkey should return '' instead of None there. Sounds like a bug
somewhere in the rpm/python/yum stack.

Possible, but since that bug is easily fixable in mash, we should do so, don't
you think?

fyi.

[oliver@tuborg ~]$ rpm -q rpm python yum
rpm-4.4.2.2-2.fc7
python-2.5-14.fc7
yum-3.2.7-1.fc7

The same mash code works on F7/FC6, though. I'm not sure what's different about
your repo.

The only thing that is different, is that I do ask *my* koji and not the
*upstream* koji - I guess...

How are you handling signatures on your local packages?

I don't sign them at the moment, but I'd like to do so; But I don't see how
upstream koji does that actually...

And one more problem:
oot@treasure /etc/mash#  mash -o /data/mash el4-ecetra
Getting package lists for el4-ecetra...
Sorting packages...
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in ?
    main()
  File "/usr/bin/mash", line 63, in main
    rc = themash.doCompose()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 207, in doCompose
    if (excludearch[pkg['build_id']] and has_any(masharch.compat[target_arch],
excludearch[pkg['build_id']])) or \
KeyError: 2909
mash failed in /data/mash/el4-ecetra

well. latest mash does the job fine. closing bz.

mash-0.2.10-2.fc8 has been submitted as an update for Fedora 8

mash-0.2.10-3.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.