Bug 636697

Summary: [MASH/KOJI] what about imported RPMs without a build ?
Product: [Fedora] Fedora Reporter: tguthmann
Component: mashAssignee: Bill Nottingham <notting>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 12CC: dcantrell, notting, rvokal
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: mash-0.5.20-1.fc13 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-13 05:57:55 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Possible workaround
none
slightly reworked change
none
Final patch none

Description tguthmann 2010-09-22 23:42:43 UTC
Description of problem:
I've just discovered that mash does not include packages without a koji 
build despite the fact that a RPM has been imported (koji import 
--create-build [package]). Mash seems to complain because there is no SRPM associated to the package/RPM. Saying sources=false in the repository config doesn't help. See code below.


Version-Release number of selected component (if applicable):
- mash 0.5.17
- koji 1.4.0


How reproducible:
Always reproducible. Import an RPM without any SRPM (and obviously no build) into koji. Then try to mash the repository containing this RPM (already packaged by a 3rd party).


Steps to Reproduce:
1. koji add-tag --owner=you yourrepo packagename
2. koji import --create-build packagename-1.0.0-1.noarch.rpm
3. koji tag-pkg yourepo packagename-1.0.0
4. mash yourrepo


Actual results:
mash throws an exception and exit.
# koji tag-pkg repoX package-17.00-00
# mash repoX
2010-09-22 11:14:40 mash: Getting package lists for repoX...
2010-09-22 11:14:40 mash: Sorting packages...
Traceback (most recent call last):
   File "/usr/bin/mash", line 97, in ?
     main()
   File "/usr/bin/mash", line 77, in main
     rc = themash.doCompose()
   File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 339, 
in doCompose
     srcpkg = src_hash[pkg['build_id']]
KeyError: 389
mash failed in /var/tmp/mash/repoX


Expected results:
mash should work properly and the RPM without build _and_ without SRPM is listed in the generated mash repository.


Additional info:
The issue of bug can be found in mash/__init__.py around line 335. Indeed src_hash does not contain a hash (the package key) for the pkg without SRPM (srcpkg = src_hash[pkg['build_id']]). Later in the code, you do a test to see if there is a SRPM but we never reach this state (fn = _get_reference(srcpkg, builds_hash)). This snippet below should be slighty redone but sadly I've reached my python skills limits so I can't really help anymore. Sorry.

        src_hash = dict([(x['build_id'],x) for x in source.packages()])
        excludearch = dict([(x['build_id'],'') for x in source.packages()])
        exclusivearch = excludearch.copy()
        # Get excludearch/exclusivearch lists for noarch packages
        for pkg in noarch.packages():
            # TOM DEBUG
            self.logger.info("buildid=%s, srcpkg=%s" % (pkg['build_id'],src_hash[pkg['build_id']]) )
            srcpkg = src_hash[pkg['build_id']]
            self.logger.debug("Checking %s for Exclude/ExclusiveArch" % (nevra(pkg),))
            fn = _get_reference(srcpkg, builds_hash)
            # if build has no source rpm, check the binary
            if fn == None:
                fn = _get_reference(pkg, builds_hash)
            try:
                hdr = koji.get_rpm_header(fn)
            except:
                self.logger.error("Couldn't read header from %s (%s)" % (nevra(pkg), fn))
                if fn:
                    fn.close()
                continue
            excludearch[pkg['build_id']] = hdr['EXCLUDEARCH']
            exclusivearch[pkg['build_id']] = hdr['EXCLUSIVEARCH']
            fn.close()

Thanks
Thomas

Comment 1 tguthmann 2010-09-23 00:42:21 UTC
Created attachment 449074 [details]
Possible workaround

Hi,

You will find in attachment a workaround to fix this issue. However I am 
looking forward for a proper fix.

This patch only defaults the dictonnary with 0 if the key is not 
present. Then I do a test to see if the value is 0 and use the RPM 
instead to get the necessary info. 

Cheers

Comment 2 Bill Nottingham 2010-09-24 18:57:41 UTC
Created attachment 449491 [details]
slightly reworked change

Your patch is in the right direction; here's a reworked version. Does this work for you?

Comment 3 tguthmann 2010-09-26 23:59:52 UTC
Hi Bill,

Sorry but it does not work. 

See output below (python is 2.4.3 on el5.5)

# mash REPO
2010-09-27 09:57:09 mash: Getting package lists for REPO...
2010-09-27 09:57:09 mash: Sorting packages...
Traceback (most recent call last):
  File "/usr/bin/mash", line 97, in ?
    main()
  File "/usr/bin/mash", line 77, in main
    rc = themash.doCompose()
  File "/usr/lib/python2.4/site-packages/mash/__init__.py", line 340, in doCompose
    srcpkg = src_hash.get([pkg['build_id']])
TypeError: list objects are unhashable
mash failed in /var/tmp/mash/REPO

Cheers,
Thomas

Comment 4 Bill Nottingham 2010-09-27 17:04:02 UTC
Whoops, remove the braces around 'pkg['build_id']. Sorry about that (wonder how it worked in a brief test here.)

Comment 5 tguthmann 2010-09-27 20:39:13 UTC
Created attachment 450032 [details]
Final patch

Fixed a typo.

Comment 6 tguthmann 2010-09-27 20:42:56 UTC
Hey Bill,

Yes it works now. It's all good. I let you change the status to closed and I hope to see the next release soon :) Thanks for your time and for mash.

Comment 7 Bill Nottingham 2010-09-27 21:02:06 UTC
OK, will be in 0.5.20-1. (Still working out one issue there before release.)

Comment 8 Fedora Update System 2010-09-28 16:17:34 UTC
mash-0.5.20-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.el5

Comment 9 Fedora Update System 2010-09-28 16:21:55 UTC
mash-0.5.20-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14

Comment 10 Fedora Update System 2010-09-28 16:33:58 UTC
mash-0.5.20-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc13

Comment 11 Fedora Update System 2010-09-28 17:31:46 UTC
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14

Comment 12 Fedora Update System 2010-10-13 05:55:56 UTC
mash-0.5.20-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 13 Fedora Update System 2010-10-13 12:49:03 UTC
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 14 Fedora Update System 2011-04-14 20:58:36 UTC
mash-0.5.20-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.