Bug 860860 - spacewalk-repo-sync fails on importing errata from Oracle Linux Repository
spacewalk-repo-sync fails on importing errata from Oracle Linux Repository
Status: CLOSED CURRENTRELEASE
Product: Spacewalk
Classification: Community
Component: Server (Show other bugs)
1.7
x86_64 Linux
unspecified Severity unspecified
: ---
: ---
Assigned To: Michael Mráka
Red Hat Satellite QA List
:
Depends On:
Blocks: space19
  Show dependency treegraph
 
Reported: 2012-09-26 18:01 EDT by daniel.schindler
Modified: 2013-05-02 07:53 EDT (History)
0 users

See Also:
Fixed In Version: spacewalk-backend-1.9.1-1
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-03-06 13:35:06 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
updateinfo.xml provided by Oracle repodata (3.67 MB, application/xml)
2012-09-26 18:04 EDT, daniel.schindler
no flags Details

  None (edit)
Description daniel.schindler 2012-09-26 18:01:35 EDT
Description of problem:
When running "spacewalk-repo-sync" to sync an Oracle Linux channel it fails on importing errata.

Version-Release number of selected component (if applicable):
spacewalk-backend-tools-1.7.38-1.el6.noarch

How reproducible:
Try to sync an Oracle public yum repository into Spacewalk.

Steps to Reproduce:
1. Create a repository for Oracle Linux Addons (http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/)
2. Create a new channel and map newly created repository
3. Run spacewalk-repo-sync on newly created channel
  
Actual results:
The actual sync fails when it comes to importing errata, provided by updateinfo.xml:

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                49
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 904 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 143, in upload_updates
    notice = self.fix_notice(notice)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 469, in fix_notice
    if "suse" in notice['from'].lower():
AttributeError: 'NoneType' object has no attribute 'lower'

Expected results:


Additional info:
Output of /var/log/messages
------
Sep 26 23:27:10 oelspw abrt: detected unhandled Python exception in '/usr/bin/spacewalk-repo-sync'
Sep 26 23:27:10 oelspw abrtd: New client connected
Sep 26 23:27:10 oelspw abrtd: Directory 'pyhook-2012-09-26-23:27:10-27513' creation detected
Sep 26 23:27:10 oelspw abrt-server[27573]: Saved Python crash dump of pid 27513 to /var/spool/abrt/pyhook-2012-09-26-23:27:10-27513
Sep 26 23:27:10 oelspw abrtd: Package 'spacewalk-backend-tools' isn't signed with proper key
Sep 26 23:27:10 oelspw abrtd: 'post-create' on '/var/spool/abrt/pyhook-2012-09-26-23:27:10-27513' exited with 1
Sep 26 23:27:10 oelspw abrtd: Corrupted or bad directory /var/spool/abrt/pyhook-2012-09-26-23:27:10-27513, deleting
Comment 1 daniel.schindler 2012-09-26 18:04:29 EDT
Created attachment 617799 [details]
updateinfo.xml provided by Oracle repodata

I don't know if the Oracle guys are doing something wrong in their updateinfo, so I attached it.
Comment 2 daniel.schindler 2012-10-02 19:42:42 EDT
I get a different AttributeError when running latest spacewalk 1.8 nightly

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
Packages already synced:         37
Packages to sync:                 2
1/2 : VirtualBox-4.1-4.1.22_80657_el6-1-0.x86_64
2/2 : VirtualBox-4.2-4.2.0_80737_el6-1-0.x86_64
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 96, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 108, in sync
    self.error_msg("ERROR: %s" % e.value)
AttributeError: 'exceptions.AttributeError' object has no attribute 'value'

###################################################
/var/log/messages output:

Oct  3 01:36:10 oelspw abrt: detected unhandled Python exception in '/usr/bin/spacewalk-repo-sync'
Oct  3 01:36:10 oelspw abrtd: New client connected
Oct  3 01:36:10 oelspw abrtd: Directory 'pyhook-2012-10-03-01:36:10-19621' creation detected
Oct  3 01:36:10 oelspw abrt-server[19840]: Saved Python crash dump of pid 19621 to /var/spool/abrt/pyhook-2012-10-03-01:36:10-19621
Oct  3 01:36:10 oelspw abrtd: Package 'spacewalk-backend-tools' isn't signed with proper key
Oct  3 01:36:10 oelspw abrtd: 'post-create' on '/var/spool/abrt/pyhook-2012-10-03-01:36:10-19621' exited with 1
Oct  3 01:36:10 oelspw abrtd: Corrupted or bad directory /var/spool/abrt/pyhook-2012-10-03-01:36:10-19621, deleting

#####################################################

Any help would be appreciated :)
Comment 3 Michael Mráka 2012-10-04 11:05:18 EDT
The issue from comment #0 has been fixed by
commit a24d11d4dd808b5eaedbbe076fef42443befc244
    860860 - don't fail when from attribute is missing

Not sure what's causing comment #2 issue - I haven't reproduced it.
Comment 4 daniel.schindler 2012-10-04 12:06:26 EDT
(In reply to comment #3)
> The issue from comment #0 has been fixed by
> commit a24d11d4dd808b5eaedbbe076fef42443befc244
>     860860 - don't fail when from attribute is missing
> 
> Not sure what's causing comment #2 issue - I haven't reproduced it.

Thank you Michael. Do you plan reproducing the issue? Please let me know if you need any further information.

Regards,
Daniel
Comment 5 daniel.schindler 2012-10-05 05:57:05 EDT
Hi.

I've switched back to Spacewalk 1.7 and I get a different AttributeError after applying Michael's change to reposync.py. It's now moaning about UpdateNotice...

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 143, in upload_updates
    notice = self.fix_notice(notice)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 470, in fix_notice
    if "suse" in notice.get('from','').lower():
AttributeError: 'UpdateNotice' object has no attribute 'get'
Comment 6 Michael Mráka 2012-10-05 08:22:46 EDT
Oops, I'm sorry, correct fix (both for 1.7 and nightly) is
d501a2821b9c4a2a7d3cd626e0d4cf881605567b
    860860 - don't fail when from attribute is missing
Comment 7 Michael Mráka 2012-10-05 10:11:19 EDT
Subsequent syncs still fail with
606 errata skipped because of empty package list.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in ?
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 130, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 294, in upload_updates
    importer.run()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/importLib.py", line 627, in run
    self.submit()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/errataImport.py", line 196, in submit
    dml = self.backend.processErrata(self.batch)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 676, in processErrata
    transactional=1)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1337, in __processObjectCollection
    return self.__processObjectCollection__(objColl, parentTable, childDict, **kwargs)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1507, in __processObjectCollection__
    return self.__doDML(dml)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1615, in __doDML
    self.__doInsert(dml.insert, dml.tables)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1622, in __doInsert
    self.__doInsertTable(tname, dict)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1636, in __doInsertTable
    insertObj.query(hash)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backendLib.py", line 431, in query
    executeStatement(statement, values, chunksize)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backendLib.py", line 456, in executeStatement
    count += statement.executemany(**tempdict)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 172, in executemany
    return apply(self._execute_wrapper, (self._executemany, ) + p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_cx_Oracle.py", line 108, in _execute_wrapper
    retval = apply(function, p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_cx_Oracle.py", line 208, in _executemany
    self._real_cursor.executemany(None, arr)
spacewalk.common.rhnException.rhnFault: (54, 'ORA-01400: cannot insert NULL into ("RHNSAT"."RHNERRATA"."PRODUCT")\n', '\n     Package Upload Failed due to uniqueness constraint violation.\n     Make sure the package does not have any duplicate dependencies or\n     does not already exists on the server\n     ')
Comment 8 daniel.schindler 2012-10-05 11:40:35 EDT
Are you using an Oracle backend? I use PostgreSQL and it seems to finish without error...
#################
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
912 errata skipped because of empty package list.
Sync completed.
Total time: 0:01:10

This might be a noob question but: Is all errata skipped due to the fact that none of the packages in the channel matches to any errata?

I will try to sync the full distribution from http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64. That repository provides all packages so some errata should match here :)
I will get back with the results.

Regards,
Daniel
Comment 9 daniel.schindler 2012-10-06 18:55:10 EDT
Same problem here :(
I have added the OL6 latest channel and tried to sync it. When it comes to importing it fails with similar not-null-constraint (I user postgresql backend).

By the way: Why are less packages synced than there are in the repository???

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel ol6_latest-x86_64 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/
Packages in repo:             17941
Packages already synced:      14146
Packages to sync:                 2
1/2 : kernel-debug-2.6.32-131.21.1.el6-0.x86_64
2/2 : kernel-debug-devel-2.6.32-131.21.1.el6-0.x86_64
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/ has 913 errata.
738 errata skipped because of empty package list.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 286, in upload_updates
    importer.run()
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/importLib.py", line 627, in run
    self.submit()
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/errataImport.py", line 196, in submit
    dml = self.backend.processErrata(self.batch)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 686, in processErrata
    transactional=1)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1347, in __processObjectCollection
    return self.__processObjectCollection__(objColl, parentTable, childDict, **kwargs)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1517, in __processObjectCollection__
    return self.__doDML(dml)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1625, in __doDML
    self.__doInsert(dml.insert, dml.tables)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1632, in __doInsert
    self.__doInsertTable(tname, dict)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1646, in __doInsertTable
    insertObj.query(hash)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backendLib.py", line 431, in query
    executeStatement(statement, values, chunksize)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backendLib.py", line 456, in executeStatement
    count += statement.executemany(**tempdict)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 172, in executemany
    return apply(self._execute_wrapper, (self._executemany, ) + p, kw)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 282, in _execute_wrapper
    retval = apply(function, p, kw)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 318, in _executemany
    self._real_cursor.executemany(self.sql, all_kwargs)
psycopg2.IntegrityError: null value in column "product" violates not-null constraint
Comment 10 daniel.schindler 2012-10-23 10:08:11 EDT
Hi Michael,

Have you found the time to take a closer look at the problem?

Kind regards,
Daniel
Comment 11 Michael Mráka 2012-10-31 09:45:09 EDT
Second issue fixed in spacewalk master by
commit 9d9fffa018e46233baf02e3b35ef7cc63337a7ed
    860860 - release and title are optional in older updateinfo version
    fixing
    Traceback (most recent call last):
      File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
        sys.exit(abs(main() or 0))
      File "/usr/bin/spacewalk-repo-sync", line 93, in main
        sync.sync()
      File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
        self.import_updates(plugin, url)
      File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
        self.upload_updates(notices)
    ...
      File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 318, in _executemany
        self._real_cursor.executemany(self.sql, all_kwargs)
    psycopg2.IntegrityError: null value in column "product" violates not-null constraint
Comment 12 daniel.schindler 2012-10-31 13:39:14 EDT
Hi Michael,

This worked perfectly. Thanks a lot for your help.

Happy Halloween and kind regards,
Daniel
Comment 13 daniel.schindler 2013-01-22 03:22:00 EST
Hi Michael,

Just to give you an update: I am in contact with Avi Miller - Principal Program Manager at Oracle. They are fixing updateinfo.xml format to work seamlessly with spacewalk.

Regards,
Daniel
Comment 14 Michael Mráka 2013-01-22 03:44:11 EST
Hi David,

Great news. Thanks.

Michael
Comment 15 Stephen Herr 2013-03-01 12:07:36 EST
Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.
Comment 16 Stephen Herr 2013-03-06 13:35:06 EST
Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19
Comment 17 daniel.schindler 2013-05-02 07:53:20 EDT
As a follow up on this issue: Oracle has made some changes to there updateinfo metadata to make it work with Spacewalk and other Linux management tools.

https://blogs.oracle.com/linux/entry/updates_to_errata_on_uln

Note You need to log in before you can comment on or make changes to this bug.