Bug 1112633 - taskomatic: CobblerSyncTask repeatedly failing - KickstartFactory.getNewestTree throws IndexOutOfBoundsException
Summary: taskomatic: CobblerSyncTask repeatedly failing - KickstartFactory.getNewestTr...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Satellite 5
Classification: Red Hat
Component: Server
Version: 560
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Stephen Herr
QA Contact: Ales Dujicek
URL:
Whiteboard:
Depends On:
Blocks: sat570-lowbug
TreeView+ depends on / blocked
 
Reported: 2014-06-24 11:11 UTC by Lukas Pramuk
Modified: 2015-01-13 09:55 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-01-13 09:55:10 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Lukas Pramuk 2014-06-24 11:11:45 UTC
Description of problem:
Imagine a situation when you phase out RHEL5 ks trees, but you have still RHEL5 ks profiles present, one of ks profile might have ticked "Always use the newest Tree for this base channel. "Newest" is determined by the date it was last modified.". If such situation happens then Taskomatic will be doomed, failing every single minute with IndexOutOfBoundsException throwed by KickstartFactory.getNewestTree() call.

Version-Release number of selected component (if applicable):
Satellite-5.6.0-RHEL5-re20130927.0
satellite-schema-5.6.0.19-2.el5sat

How reproducible:
100%

Steps to Reproduce:
0. Sync ks trees for any major rhel release, e.g. rhel5 
1. Create rhel5 ks profile having ticked "Always use the newest Tree for this base channel. "Newest" is determined by the date it was last modified."

2. Phase out rhel5 kickstartable trees
# mv /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-5* /tmp/

3. Taskomatic daemon repeatedly failing every single minute:
# tail -f /var/log/rhn/rhn_taskomatic_daemon.log

INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,143 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - RuntimeExceptioneError trying to sync to cobbler: null
INFO   | jvm 1    | 2014/06/24 06:34:00 | java.lang.IndexOutOfBoundsException
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at java.util.LinkedList.get(LinkedList.java:517)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.domain.kickstart.KickstartFactory.getNewestTree(KickstartFactory.java:123)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:89)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:82)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:169)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,143 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet.
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,144 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - 
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,144 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - 

3b.@webui you can check cobbler-sync-bunch results (/rhn/admin/BunchDetail.do?label=cobbler-sync-bunch): FAILED


Actual results:
throwed Exception, taskomatic blocked by repeatedly throwed Exceptions


Expected results:
no Exceptions, taskomatic running its tasks


Additional info:
KickstartFactory.java:123 
            return trees.get(trees.size() - 1);       <- size can be sometimes 0:-)

Comment 1 Stephen Herr 2014-06-24 20:34:16 UTC
There is also a similar problem where if you delete a custom Distribution CobblerSync fails with this error due to a stale Hibernate cache until Taskomatic is restarted:

ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Stack trace:org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#19]

Comment 2 Stephen Herr 2014-06-24 21:09:19 UTC
Both problems should be fixed by Spacewalk master commit:
aa53830b192d958fbc45a892b280b7ea5bf0d68c

Comment 4 Clifford Perry 2015-01-13 09:55:10 UTC
With the release of Red Hat Satellite 5.7 on January 12th 2015 this bug is
being moved to a Closed Current Release state. 

The Satellite 5.7 GA Errata:
 - https://rhn.redhat.com/errata/RHSA-2015-0033.html 

Satellite 5.7 Release Notes:
 -
https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.7/html-single/Release_Notes/index.html

Satellite Customer Portal Blog announcement for release:
 - https://access.redhat.com/blogs/1169563/posts/1315743 

Cliff


Note You need to log in before you can comment on or make changes to this bug.