Bug 525561

Summary:	Bad db data in upgrade leads to unusable kickstarts
Product:	Red Hat Satellite 5	Reporter:	Justin Sherrill <jsherril>
Component:	Provisioning	Assignee:	Justin Sherrill <jsherril>
Status:	CLOSED CURRENTRELEASE	QA Contact:	Brandon Perkins <bperkins>
Severity:	high	Docs Contact:
Priority:	high
Version:	530	CC:	cperry, gkhachik, jhutar, sdodson, tao, xdmoon
Target Milestone:	---
Target Release:	---
Hardware:	All
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2009-11-23 16:44:09 UTC	Type:	---
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	518256

Description Justin Sherrill 2009-09-24 20:29:46 UTC

Description of problem:

After an upgrade to 5.3.0 the taskomatic task that syncs distros and profiles to cobbler will fail partway through if a certain situation exists.  (most likely this situation existed before the upgrade.)

Two symptoms are seen:

1.  When you click on some (but not necessarily all) kickstarts an error will be displayed saying that the distro is invalid.  

2.  Within /var/log/rhn/rhn_Taskomatic.conf, you'll see the following error:

INFO   | jvm 1    | 2009/09/21 15:08:00 | java.lang.NullPointerException
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCommand.getDistroForKickstart(CobblerProfileCommand.java:133)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileCreateCommand.store(CobblerProfileCreateCommand.java:65)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileSyncCommand.createProfile(CobblerProfileSyncCommand.java:102)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.manager.kickstart.cobbler.CobblerProfileSyncCommand.store(CobblerProfileSyncCommand.java:76)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:108)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at com.redhat.rhn.taskomatic.task.SingleThreadedTestableTask.execute(SingleThreadedTestableTask.java:54)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
INFO   | jvm 1    | 2009/09/21 15:08:00 |       at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
INFO   | jvm 1    | 2009/09/21 15:08:00 | 2009-09-21 15:08:00,305 [DefaultQuartzScheduler_Worker-4] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - Not re-throwing any more err
ors.



This error will repeat every 5 or 10 minutes. 


The cause of this error is that there is a rhnKSData entry without a corresponding rhnKickstartDefaults entry.  The only reason I can think that this would exist is by using the sat-rm.py script while having a profile associated with a distro that belongs to that channel.

To confirm that the data is bad, simply run this query:

sql>  select  ks.label, dk.kickstart_id  from rhnKSData ks,
rhnKickstartDefaults dk
  where  ks.id = dk.kickstart_id (+) where dk.kickstart_id is null;


If any rows are returned, then that is the confirmation that this is the problem. 


To fix the issue we could either do a code change to handle the issue, or have the customer run something like:

sql> delete from rhnKSData where id not in (select kickstart_id from
rhnKickstartDefaults);

That should delete any of the bad entries.  Note: due to the way they are displayed, these bad entries will NOT show up in the webUI, so they will be 'hidden' from any view except the database itself.

Comment 2 Justin Sherrill 2009-09-29 17:17:36 UTC

That select query had a small typo in it.  It should be:

select  ks.label, dk.kickstart_id  from rhnKSData ks, rhnKickstartDefaults dk  where  ks.id = dk.kickstart_id (+) and dk.kickstart_id is null;

Comment 4 Justin Sherrill 2009-10-15 12:00:50 UTC

Hi Jan,

You should be able to reproduce doing the following:

1)  Install some pre-5.3 satellite (i.e 5.0, 5.1, 5.2).  
2)  Create one or more kickstarts and make sure to keep track of their id (the ksid=# part of the url after you create them).  

3)  Run the following query:

sql>  delete from rhnKickstartDefaults where kickstart_id = 102;

replacing 102 with the kickstart id from step #2.   Re-run this for each kickstart id that you want to put in a 'bad' state. 

sql> commit;

3) upgrade!

Comment 5 Justin Sherrill 2009-10-26 21:08:01 UTC

Fixed in spacewalk master: 1ae5ebbdc7f1cf7adb5202dbf3151009769f1278

Comment 7 Justin Sherrill 2009-11-23 16:44:09 UTC

This looks to have been already released.