Bug 350391 - mash from devel doesn't work on rhel5
mash from devel doesn't work on rhel5
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: mash (Show other bugs)
rawhide
All Linux
low Severity low
: ---
: ---
Assigned To: Bill Nottingham
Fedora Extras Quality Assurance
: Reopened
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-10-24 08:28 EDT by Oliver Falk
Modified: 2014-03-16 23:11 EDT (History)
2 users (show)

See Also:
Fixed In Version: 0.2.10-3.fc8
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-04-29 16:49:26 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
patch (1.10 KB, patch)
2007-11-16 10:22 EST, Bill Nottingham
no flags Details | Diff
patch, this time without syntax errors (1.19 KB, patch)
2007-11-16 10:32 EST, Bill Nottingham
no flags Details | Diff
untested patch (4.53 KB, patch)
2007-11-16 11:07 EST, Bill Nottingham
no flags Details | Diff
updated patch (5.26 KB, patch)
2007-11-16 14:04 EST, Bill Nottingham
no flags Details | Diff

  None (edit)
Description Oliver Falk 2007-10-24 08:28:35 EDT
Jesse. I'm encountering the following problem with mash on an RHEL 5 server (so
it's python 2.4):

Previous repo file missing:
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/primary.xml.gz
Resolving multilib for arch x86_64 using method devel
Waiting for depsolve and createrepo to finish...
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in ?
    main()
  File "/usr/bin/mash", line 70, in main
    rc = themash.doMultilib()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 409, in doMultilib
    pid = self.doDepSolveAndMultilib(arch, cachedir)
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 353, in
doDepSolveAndMultilib
    yumbase.doTsSetup()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 63, in doTsSetup
    return self._getTs()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 77, in _getTs
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 526, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 381, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 142, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 199, in
_check_db_version
    if repo.repoXML.repoData.has_key(mdtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 769, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 765, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el4-ecetra-x86_64. Please verify its path and try again
mash failed in /data/mash/el4-ecetra




I've already tried different versions of createrepo, yum, ... nothing helped.

Help's appreciated! :-)



Jesse, since mash is in notting's home at people.rh, you may forward this to him
of course, I only assigned you, since you're pkg maintainer regarding to pkgdb.
Comment 1 Jesse Keating 2007-10-24 09:33:44 EDT
Uh... the error seems pretty clear to me, 

yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el4-ecetra-x86_64. Please verify its path and try again
mash failed in /data/mash/el4-ecetra

Is there actually any repodata in "/data/mash/el4-ecetra" ?  Perhaps createrepo
is failing because it's trying to use -d and that doesn't exist in RHEL5?
Comment 2 Oliver Falk 2007-10-24 09:56:43 EDT
If it would be soooo easy Jesse - I would not bug you :-)

I have createrepo 0.4.10 here... That has -d. And yes, there actually is repodata:
root@treasure ~# find /data/mash/el4-ecetra -type f|grep -v rpm$
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/primary.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/filelists.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/other.xml.gz
/data/mash/el4-ecetra/x86_64/os/RedHat/repodata/repomd.xml
/data/mash/el4-ecetra/i386/os/RedHat/repodata/primary.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/filelists.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/other.xml.gz
/data/mash/el4-ecetra/i386/os/RedHat/repodata/repomd.xml
/data/mash/el4-ecetra/source/SRPMS/repodata/primary.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/filelists.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/other.xml.gz
/data/mash/el4-ecetra/source/SRPMS/repodata/repomd.xml
Comment 3 Oliver Falk 2007-10-24 09:59:05 EDT
Maybe also good info:
If I add a "print self.conf.cachedir" right after "# Set attributes not from the
config file" in def readRepoConfig from yum's __init__.py

The following I see when running mash:
/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache
Comment 4 Oliver Falk 2007-10-24 10:11:19 EDT
Ah. And if I do it only for i386 (setting arches = i386) it works. I try this
with x86_64 now as well.
Comment 5 Oliver Falk 2007-10-24 10:47:40 EDT
And if I do arches = x86_64 the problem is there again. :-(
Comment 6 Oliver Falk 2007-10-29 11:18:26 EDT
ping.
Comment 7 Jesse Keating 2007-10-29 11:40:51 EDT
What happens if you run this in a chroot of rawhide packages?  When we compose
rawhide we don't actually run mash directly on the host, we create (or update) a
chroot of the tree about to be mashed and then use mash from inside there.
Comment 8 Oliver Falk 2007-10-30 10:03:50 EDT
Inside the chroot. The same.

mock-chroot> mash -o /data el5-ecetra     
Getting package lists for el5-ecetra...
Sorting packages...
Checking signatures...

< ... >

Writing out files for /data/el5-ecetra/i386/os/RedHat...
Writing out files for /data/el5-ecetra/x86_64/os/RedHat...
Writing out files for /data/el5-ecetra/source/SRPMS...
Waiting for createrepo to finish...
Previous repo file missing: /data/el5-ecetra/source/SRPMS/repodata/primary.xml.gz
Previous repo file missing:
/data/el5-ecetra/x86_64/os/RedHat/repodata/primary.xml.gz
Previous repo file missing: /data/el5-ecetra/i386/os/RedHat/repodata/primary.xml.gz
Resolving multilib for arch x86_64 using method all
Waiting for depsolve and createrepo to finish...
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in <module>
    main()
  File "/usr/bin/mash", line 70, in main
    rc = themash.doMultilib()
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 409, in doMultilib
    pid = self.doDepSolveAndMultilib(arch, cachedir)
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 353, in
doDepSolveAndMultilib
    yumbase.doTsSetup()
  File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 63, in doTsSetup
    return self._getTs()
  File "/usr/lib/python2.5/site-packages/yum/depsolve.py", line 77, in _getTs
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 526, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.5/site-packages/yum/__init__.py", line 381, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.5/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 142, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 199, in
_check_db_version
    if repo.repoXML.repoData.has_key(mdtype):
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 769, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.5/site-packages/yum/yumRepo.py", line 765, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el5-ecetra-x86_64. Please verify its path and try again
mash failed in /data/el5-ecetra


mock-chroot> rpm -q createrepo python mash
createrepo-0.4.10-1.fc8
python-2.5.1-14.fc8
mash-0.2.8-1.fc8
Comment 9 Jesse Keating 2007-10-30 10:19:27 EDT
What happens if you manually run createrepo with the same options mash would use
in those paths (from the chroot)
Comment 10 Oliver Falk 2007-10-30 10:43:18 EDT
Createrepo doesn't spot any error and produces the repodata just fine.

mock-chroot> find | grep -i repoda                
./x86_64/os/RedHat/repodata
./x86_64/os/RedHat/repodata/other.sqlite.bz2
./x86_64/os/RedHat/repodata/filelists.sqlite.bz2
./x86_64/os/RedHat/repodata/other.xml.gz
./x86_64/os/RedHat/repodata/filelists.xml.gz
./x86_64/os/RedHat/repodata/repomd.xml
./x86_64/os/RedHat/repodata/primary.sqlite.bz2
./x86_64/os/RedHat/repodata/primary.xml.gz
./source/SRPMS/repodata
./source/SRPMS/repodata/other.sqlite.bz2
./source/SRPMS/repodata/filelists.sqlite.bz2
./source/SRPMS/repodata/other.xml.gz
./source/SRPMS/repodata/filelists.xml.gz
./source/SRPMS/repodata/repomd.xml
./source/SRPMS/repodata/primary.sqlite.bz2
./source/SRPMS/repodata/primary.xml.gz
./i386/os/RedHat/repodata
./i386/os/RedHat/repodata/other.sqlite.bz2
./i386/os/RedHat/repodata/filelists.sqlite.bz2
./i386/os/RedHat/repodata/other.xml.gz
./i386/os/RedHat/repodata/filelists.xml.gz
./i386/os/RedHat/repodata/repomd.xml
./i386/os/RedHat/repodata/primary.sqlite.bz2
./i386/os/RedHat/repodata/primary.xml.gz



Well. If I run mash then, it removes the full tree /data/el5-ecetra and also the
old repodata of course... :-/
Comment 11 Jesse Keating 2007-10-30 11:07:26 EDT
Right, because mash creates repodata on its own to make use of.
Comment 12 Oliver Falk 2007-10-31 08:45:01 EDT
That's very strange. I created a chroot with mock from latest and greatest
fedora devel and the problem is still the same. I'll now try f7 chroot.
Comment 13 Oliver Falk 2007-10-31 09:11:58 EDT
ok. f7 chroot does the same error.
Comment 14 Jesse Keating 2007-11-15 14:41:46 EST
Bill, your turn.. (:
Comment 15 Bill Nottingham 2007-11-15 14:54:06 EST
What versions of:
mash
createrepo
yum
?
Comment 16 Bill Nottingham 2007-11-16 10:18:45 EST
(In reply to comment #3)
> Maybe also good info:
> If I add a "print self.conf.cachedir" right after "# Set attributes not from the
> config file" in def readRepoConfig from yum's __init__.py
> 
> The following I see when running mash:
>
/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache

OK, so the code in question is:

        tmpdir = "/tmp/mash-%s/" % (self.config.name,)
        tmproot = os.path.join(tmpdir, "%s-%s.tmp" % (self.config.name, arch))
        yumcachedir = os.path.join(tmproot, "yumcache")

I'm not seeing how you'd get a duplicate path in there.
Comment 17 Bill Nottingham 2007-11-16 10:22:34 EST
Created attachment 261351 [details]
patch

Oh wait, cachedir is relative to installroot. Does this help at all?
Comment 18 Bill Nottingham 2007-11-16 10:32:12 EST
Created attachment 261371 [details]
patch, this time without syntax errors
Comment 19 Oliver Falk 2007-11-16 10:50:31 EST
Hm. And why does that work on f7 then!?

However:

/tmp/mash-el4-ecetra/el4-ecetra-x86_64.tmp/yumcache
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
That's better now. But I still get this:

Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in ?
    main()
  File "/usr/bin/mash", line 70, in main
    rc = themash.doMultilib()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 408, in doMultilib
    pid = self.doDepSolveAndMultilib(arch, cachedir)
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 352, in
doDepSolveAndMultilib
    yumbase.doTsSetup()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 63, in doTsSetup
    return self._getTs()
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 77, in _getTs
    self._tsInfo.setDatabases(self.rpmdb, self.pkgSack)
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 527, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 382, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 242, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 142, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 199, in
_check_db_version
    if repo.repoXML.repoData.has_key(mdtype):
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 769, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.4/site-packages/yum/yumRepo.py", line 765, in _getRepoXML
    raise Errors.RepoError, msg
yum.Errors.RepoError: Cannot retrieve repository metadata (repomd.xml) for
repository: el4-ecetra-x86_64. Please verify its path and try again
mash failed in /data/mash/el4-ecetra
Comment 20 Oliver Falk 2007-11-16 10:51:45 EST
(In reply to comment #15)
> What versions of:
> mash
> createrepo
> yum
> ?

of@treasure ~$ rpm -q mash createrepo yum
mash-0.2.8-1
createrepo-0.4.10-1.5EL
yum-3.2.7-1

All compiled myself on that system (took rawhide pkgs).
Comment 21 Bill Nottingham 2007-11-16 11:07:03 EST
Created attachment 261401 [details]
untested patch

Try this, may take some massaging. You'll need to adjust your config to set the
repodata_path.
Comment 22 Bill Nottingham 2007-11-16 14:04:33 EST
Created attachment 261641 [details]
updated patch

This *should* solve your problem. (You'll need to set repodata_path in your
mash config).

Let me know how well it works for you.
Comment 23 Oliver Falk 2007-11-18 16:14:44 EST
AFAICS, this WorksForMe (tm). :-)

Bill, will this fix go upstrem as well?

Thx,
 Oliver
Comment 24 Bill Nottingham 2007-11-19 16:34:54 EST
Fixed in 0.2.10-1.
Comment 25 Oliver Falk 2007-11-21 11:55:26 EST
There's another bug, that I forgot to tell you:
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in <module>
    main()
  File "/usr/bin/mash", line 63, in main
    rc = themash.doCompose()
  File "/usr/lib/python2.5/site-packages/mash/__init__.py", line 232, in doCompose
    key = pkg['sigkey'].lower()
AttributeError: 'NoneType' object has no attribute 'lower'
mash failed in /data/mash/development


This from another machine, mash-ing the AlphaCore tree; However. That also
happens on EL5 when mashing rpms without signatures...

A simple check with pkg['signkey'] should do the trick...

If you prefer a new bz#, please let me know.
Comment 26 Bill Nottingham 2007-11-21 12:08:00 EST
Actually, sigkey should return '' instead of None there. Sounds like a bug
somewhere in the rpm/python/yum stack.
Comment 27 Oliver Falk 2007-11-21 12:12:48 EST
Possible, but since that bug is easily fixable in mash, we should do so, don't
you think?

fyi.

[oliver@tuborg ~]$ rpm -q rpm python yum
rpm-4.4.2.2-2.fc7
python-2.5-14.fc7
yum-3.2.7-1.fc7
Comment 28 Bill Nottingham 2007-11-21 12:19:19 EST
The same mash code works on F7/FC6, though. I'm not sure what's different about
your repo.
Comment 29 Oliver Falk 2007-11-21 12:25:53 EST
The only thing that is different, is that I do ask *my* koji and not the
*upstream* koji - I guess...
Comment 30 Bill Nottingham 2007-11-21 12:36:08 EST
How are you handling signatures on your local packages?
Comment 31 Oliver Falk 2007-11-22 02:48:28 EST
I don't sign them at the moment, but I'd like to do so; But I don't see how
upstream koji does that actually...
Comment 32 Oliver Falk 2007-11-29 09:29:47 EST
And one more problem:
oot@treasure /etc/mash#  mash -o /data/mash el4-ecetra
Getting package lists for el4-ecetra...
Sorting packages...
Traceback (most recent call last):
  File "/usr/bin/mash", line 82, in ?
    main()
  File "/usr/bin/mash", line 63, in main
    rc = themash.doCompose()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 207, in doCompose
    if (excludearch[pkg['build_id']] and has_any(masharch.compat[target_arch],
excludearch[pkg['build_id']])) or \
KeyError: 2909
mash failed in /data/mash/el4-ecetra
Comment 33 Oliver Falk 2007-12-03 06:32:00 EST
well. latest mash does the job fine. closing bz.
Comment 34 Fedora Update System 2008-04-11 10:18:31 EDT
mash-0.2.10-2.fc8 has been submitted as an update for Fedora 8
Comment 35 Fedora Update System 2008-04-29 16:49:20 EDT
mash-0.2.10-3.fc8 has been pushed to the Fedora 8 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.