Bug 860860

Summary:

spacewalk-repo-sync fails on importing errata from Oracle Linux Repository

Product:

[Community] Spacewalk

Reporter:

daniel.schindler

Component:

Server

Assignee:

Michael Mráka <mmraka>

Status:

CLOSED CURRENTRELEASE

QA Contact:

Red Hat Satellite QA List <satqe-list>

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

1.7

Target Milestone:

---

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

spacewalk-backend-1.9.1-1

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2013-03-06 18:35:06 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

917805

Attachments:

Description	Flags
updateinfo.xml provided by Oracle repodata	none

Description daniel.schindler 2012-09-26 22:01:35 UTC

Description of problem:
When running "spacewalk-repo-sync" to sync an Oracle Linux channel it fails on importing errata.

Version-Release number of selected component (if applicable):
spacewalk-backend-tools-1.7.38-1.el6.noarch

How reproducible:
Try to sync an Oracle public yum repository into Spacewalk.

Steps to Reproduce:
1. Create a repository for Oracle Linux Addons (http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/)
2. Create a new channel and map newly created repository
3. Run spacewalk-repo-sync on newly created channel
  
Actual results:
The actual sync fails when it comes to importing errata, provided by updateinfo.xml:

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                49
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 904 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 143, in upload_updates
    notice = self.fix_notice(notice)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 469, in fix_notice
    if "suse" in notice['from'].lower():
AttributeError: 'NoneType' object has no attribute 'lower'

Expected results:


Additional info:
Output of /var/log/messages
------
Sep 26 23:27:10 oelspw abrt: detected unhandled Python exception in '/usr/bin/spacewalk-repo-sync'
Sep 26 23:27:10 oelspw abrtd: New client connected
Sep 26 23:27:10 oelspw abrtd: Directory 'pyhook-2012-09-26-23:27:10-27513' creation detected
Sep 26 23:27:10 oelspw abrt-server[27573]: Saved Python crash dump of pid 27513 to /var/spool/abrt/pyhook-2012-09-26-23:27:10-27513
Sep 26 23:27:10 oelspw abrtd: Package 'spacewalk-backend-tools' isn't signed with proper key
Sep 26 23:27:10 oelspw abrtd: 'post-create' on '/var/spool/abrt/pyhook-2012-09-26-23:27:10-27513' exited with 1
Sep 26 23:27:10 oelspw abrtd: Corrupted or bad directory /var/spool/abrt/pyhook-2012-09-26-23:27:10-27513, deleting

Comment 1 daniel.schindler 2012-09-26 22:04:29 UTC

Created attachment 617799 [details]
updateinfo.xml provided by Oracle repodata

I don't know if the Oracle guys are doing something wrong in their updateinfo, so I attached it.

Comment 2 daniel.schindler 2012-10-02 23:42:42 UTC

I get a different AttributeError when running latest spacewalk 1.8 nightly

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
Packages already synced:         37
Packages to sync:                 2
1/2 : VirtualBox-4.1-4.1.22_80657_el6-1-0.x86_64
2/2 : VirtualBox-4.2-4.2.0_80737_el6-1-0.x86_64
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 103, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 96, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 108, in sync
    self.error_msg("ERROR: %s" % e.value)
AttributeError: 'exceptions.AttributeError' object has no attribute 'value'

###################################################
/var/log/messages output:

Oct  3 01:36:10 oelspw abrt: detected unhandled Python exception in '/usr/bin/spacewalk-repo-sync'
Oct  3 01:36:10 oelspw abrtd: New client connected
Oct  3 01:36:10 oelspw abrtd: Directory 'pyhook-2012-10-03-01:36:10-19621' creation detected
Oct  3 01:36:10 oelspw abrt-server[19840]: Saved Python crash dump of pid 19621 to /var/spool/abrt/pyhook-2012-10-03-01:36:10-19621
Oct  3 01:36:10 oelspw abrtd: Package 'spacewalk-backend-tools' isn't signed with proper key
Oct  3 01:36:10 oelspw abrtd: 'post-create' on '/var/spool/abrt/pyhook-2012-10-03-01:36:10-19621' exited with 1
Oct  3 01:36:10 oelspw abrtd: Corrupted or bad directory /var/spool/abrt/pyhook-2012-10-03-01:36:10-19621, deleting

#####################################################

Any help would be appreciated :)

Comment 3 Michael Mráka 2012-10-04 15:05:18 UTC

The issue from comment #0 has been fixed by
commit a24d11d4dd808b5eaedbbe076fef42443befc244
    860860 - don't fail when from attribute is missing

Not sure what's causing comment #2 issue - I haven't reproduced it.

Comment 4 daniel.schindler 2012-10-04 16:06:26 UTC

(In reply to comment #3)
> The issue from comment #0 has been fixed by
> commit a24d11d4dd808b5eaedbbe076fef42443befc244
>     860860 - don't fail when from attribute is missing
> 
> Not sure what's causing comment #2 issue - I haven't reproduced it.

Thank you Michael. Do you plan reproducing the issue? Please let me know if you need any further information.

Regards,
Daniel

Comment 5 daniel.schindler 2012-10-05 09:57:05 UTC

Hi.

I've switched back to Spacewalk 1.7 and I get a different AttributeError after applying Michael's change to reposync.py. It's now moaning about UpdateNotice...

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel oraclelinux6 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 143, in upload_updates
    notice = self.fix_notice(notice)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 470, in fix_notice
    if "suse" in notice.get('from','').lower():
AttributeError: 'UpdateNotice' object has no attribute 'get'

Comment 6 Michael Mráka 2012-10-05 12:22:46 UTC

Oops, I'm sorry, correct fix (both for 1.7 and nightly) is
d501a2821b9c4a2a7d3cd626e0d4cf881605567b
    860860 - don't fail when from attribute is missing

Comment 7 Michael Mráka 2012-10-05 14:11:19 UTC

Subsequent syncs still fail with
606 errata skipped because of empty package list.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in ?
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 130, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.4/site-packages/spacewalk/satellite_tools/reposync.py", line 294, in upload_updates
    importer.run()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/importLib.py", line 627, in run
    self.submit()
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/errataImport.py", line 196, in submit
    dml = self.backend.processErrata(self.batch)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 676, in processErrata
    transactional=1)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1337, in __processObjectCollection
    return self.__processObjectCollection__(objColl, parentTable, childDict, **kwargs)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1507, in __processObjectCollection__
    return self.__doDML(dml)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1615, in __doDML
    self.__doInsert(dml.insert, dml.tables)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1622, in __doInsert
    self.__doInsertTable(tname, dict)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backend.py", line 1636, in __doInsertTable
    insertObj.query(hash)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backendLib.py", line 431, in query
    executeStatement(statement, values, chunksize)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/importlib/backendLib.py", line 456, in executeStatement
    count += statement.executemany(**tempdict)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 172, in executemany
    return apply(self._execute_wrapper, (self._executemany, ) + p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_cx_Oracle.py", line 108, in _execute_wrapper
    retval = apply(function, p, kw)
  File "/usr/lib/python2.4/site-packages/spacewalk/server/rhnSQL/driver_cx_Oracle.py", line 208, in _executemany
    self._real_cursor.executemany(None, arr)
spacewalk.common.rhnException.rhnFault: (54, 'ORA-01400: cannot insert NULL into ("RHNSAT"."RHNERRATA"."PRODUCT")\n', '\n     Package Upload Failed due to uniqueness constraint violation.\n     Make sure the package does not have any duplicate dependencies or\n     does not already exists on the server\n     ')

Comment 8 daniel.schindler 2012-10-05 15:40:35 UTC

Are you using an Oracle backend? I use PostgreSQL and it seems to finish without error...
#################
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/
Packages in repo:                51
No new packages to sync.
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64/ has 912 errata.
912 errata skipped because of empty package list.
Sync completed.
Total time: 0:01:10

This might be a noob question but: Is all errata skipped due to the fact that none of the packages in the channel matches to any errata?

I will try to sync the full distribution from http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64. That repository provides all packages so some errata should match here :)
I will get back with the results.

Regards,
Daniel

Comment 9 daniel.schindler 2012-10-06 22:55:10 UTC

Same problem here :(
I have added the OL6 latest channel and tried to sync it. When it comes to importing it fails with similar not-null-constraint (I user postgresql backend).

By the way: Why are less packages synced than there are in the repository???

[root@oelspw ~]# /usr/bin/python -u /usr/bin/spacewalk-repo-sync --channel ol6_latest-x86_64 --type yum
Repo URL: http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/
Packages in repo:             17941
Packages already synced:      14146
Packages to sync:                 2
1/2 : kernel-debug-2.6.32-131.21.1.el6-0.x86_64
2/2 : kernel-debug-devel-2.6.32-131.21.1.el6-0.x86_64
Repo http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/ has 913 errata.
738 errata skipped because of empty package list.
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 93, in main
    sync.sync()
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
    self.import_updates(plugin, url)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
    self.upload_updates(notices)
  File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 286, in upload_updates
    importer.run()
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/importLib.py", line 627, in run
    self.submit()
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/errataImport.py", line 196, in submit
    dml = self.backend.processErrata(self.batch)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 686, in processErrata
    transactional=1)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1347, in __processObjectCollection
    return self.__processObjectCollection__(objColl, parentTable, childDict, **kwargs)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1517, in __processObjectCollection__
    return self.__doDML(dml)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1625, in __doDML
    self.__doInsert(dml.insert, dml.tables)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1632, in __doInsert
    self.__doInsertTable(tname, dict)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backend.py", line 1646, in __doInsertTable
    insertObj.query(hash)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backendLib.py", line 431, in query
    executeStatement(statement, values, chunksize)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/importlib/backendLib.py", line 456, in executeStatement
    count += statement.executemany(**tempdict)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/sql_base.py", line 172, in executemany
    return apply(self._execute_wrapper, (self._executemany, ) + p, kw)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 282, in _execute_wrapper
    retval = apply(function, p, kw)
  File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 318, in _executemany
    self._real_cursor.executemany(self.sql, all_kwargs)
psycopg2.IntegrityError: null value in column "product" violates not-null constraint

Comment 10 daniel.schindler 2012-10-23 14:08:11 UTC

Hi Michael,

Have you found the time to take a closer look at the problem?

Kind regards,
Daniel

Comment 11 Michael Mráka 2012-10-31 13:45:09 UTC

Second issue fixed in spacewalk master by
commit 9d9fffa018e46233baf02e3b35ef7cc63337a7ed
    860860 - release and title are optional in older updateinfo version
    fixing
    Traceback (most recent call last):
      File "/usr/bin/spacewalk-repo-sync", line 100, in <module>
        sys.exit(abs(main() or 0))
      File "/usr/bin/spacewalk-repo-sync", line 93, in main
        sync.sync()
      File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 101, in sync
        self.import_updates(plugin, url)
      File "/usr/lib/python2.6/site-packages/spacewalk/satellite_tools/reposync.py", line 129, in import_updates
        self.upload_updates(notices)
    ...
      File "/usr/lib/python2.6/site-packages/spacewalk/server/rhnSQL/driver_postgresql.py", line 318, in _executemany
        self._real_cursor.executemany(self.sql, all_kwargs)
    psycopg2.IntegrityError: null value in column "product" violates not-null constraint

Comment 12 daniel.schindler 2012-10-31 17:39:14 UTC

Hi Michael,

This worked perfectly. Thanks a lot for your help.

Happy Halloween and kind regards,
Daniel

Comment 13 daniel.schindler 2013-01-22 08:22:00 UTC

Hi Michael,

Just to give you an update: I am in contact with Avi Miller - Principal Program Manager at Oracle. They are fixing updateinfo.xml format to work seamlessly with spacewalk.

Regards,
Daniel

Comment 14 Michael Mráka 2013-01-22 08:44:11 UTC

Hi David,

Great news. Thanks.

Michael

Comment 15 Stephen Herr 2013-03-01 17:07:36 UTC

Marking bug as ON_QA since tonight's build of Spacewalk nightly is a release candidate for Spacewalk 1.9.

Comment 16 Stephen Herr 2013-03-06 18:35:06 UTC

Spacewalk 1.9 has been released.

https://fedorahosted.org/spacewalk/wiki/ReleaseNotes19

Comment 17 daniel.schindler 2013-05-02 11:53:20 UTC

As a follow up on this issue: Oracle has made some changes to there updateinfo metadata to make it work with Spacewalk and other Linux management tools.

https://blogs.oracle.com/linux/entry/updates_to_errata_on_uln