Bug 1112633

Summary: taskomatic: CobblerSyncTask repeatedly failing - KickstartFactory.getNewestTree throws IndexOutOfBoundsException
Product: Red Hat Satellite 5 Reporter: Lukas Pramuk <lpramuk>
Component: ServerAssignee: Stephen Herr <sherr>
Status: CLOSED CURRENTRELEASE QA Contact: Ales Dujicek <adujicek>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 560CC: adujicek, cperry
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-01-13 09:55:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1127641    

Description Lukas Pramuk 2014-06-24 11:11:45 UTC
Description of problem:
Imagine a situation when you phase out RHEL5 ks trees, but you have still RHEL5 ks profiles present, one of ks profile might have ticked "Always use the newest Tree for this base channel. "Newest" is determined by the date it was last modified.". If such situation happens then Taskomatic will be doomed, failing every single minute with IndexOutOfBoundsException throwed by KickstartFactory.getNewestTree() call.

Version-Release number of selected component (if applicable):
Satellite-5.6.0-RHEL5-re20130927.0
satellite-schema-5.6.0.19-2.el5sat

How reproducible:
100%

Steps to Reproduce:
0. Sync ks trees for any major rhel release, e.g. rhel5 
1. Create rhel5 ks profile having ticked "Always use the newest Tree for this base channel. "Newest" is determined by the date it was last modified."

2. Phase out rhel5 kickstartable trees
# mv /var/satellite/rhn/kickstart/ks-rhel-x86_64-server-5* /tmp/

3. Taskomatic daemon repeatedly failing every single minute:
# tail -f /var/log/rhn/rhn_taskomatic_daemon.log

INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,143 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - RuntimeExceptioneError trying to sync to cobbler: null
INFO   | jvm 1    | 2014/06/24 06:34:00 | java.lang.IndexOutOfBoundsException
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at java.util.LinkedList.get(LinkedList.java:517)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.domain.kickstart.KickstartFactory.getNewestTree(KickstartFactory.java:123)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:89)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:82)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:169)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,143 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet.
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,144 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - 
INFO   | jvm 1    | 2014/06/24 06:34:00 | 2014-06-24 06:34:00,144 [DefaultQuartzScheduler_Worker-6] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - 

3b.@webui you can check cobbler-sync-bunch results (/rhn/admin/BunchDetail.do?label=cobbler-sync-bunch): FAILED


Actual results:
throwed Exception, taskomatic blocked by repeatedly throwed Exceptions


Expected results:
no Exceptions, taskomatic running its tasks


Additional info:
KickstartFactory.java:123 
            return trees.get(trees.size() - 1);       <- size can be sometimes 0:-)

Comment 1 Stephen Herr 2014-06-24 20:34:16 UTC
There is also a similar problem where if you delete a custom Distribution CobblerSync fails with this error due to a stale Hibernate cache until Taskomatic is restarted:

ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Stack trace:org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#19]

Comment 2 Stephen Herr 2014-06-24 21:09:19 UTC
Both problems should be fixed by Spacewalk master commit:
aa53830b192d958fbc45a892b280b7ea5bf0d68c

Comment 4 Clifford Perry 2015-01-13 09:55:10 UTC
With the release of Red Hat Satellite 5.7 on January 12th 2015 this bug is
being moved to a Closed Current Release state. 

The Satellite 5.7 GA Errata:
 - https://rhn.redhat.com/errata/RHSA-2015-0033.html 

Satellite 5.7 Release Notes:
 -
https://access.redhat.com/documentation/en-US/Red_Hat_Satellite/5.7/html-single/Release_Notes/index.html

Satellite Customer Portal Blog announcement for release:
 - https://access.redhat.com/blogs/1169563/posts/1315743 

Cliff