Created attachment 518944 [details]
Script which demonstrates the issue

Description of problem:

When altering a kickstart profile's distribution/tree via the satellite web-UI, or API, the underlying cobbler profile does not get updated, so the cobbler profile remains associated with the original distribution.

This becomes a problem if you delete the (now supposedly unused) distribution via satellite.  At this point, the distribution is deleted from cobbler, which recursively deletes all related profiles and systems, which should now be associated with the new profile.

Partial recovery from this can be achieved by restarting satellite, at which point the deleted kickstart profile(s) will be regenerated from the satellite database.  However the cobbler system entries are permanently deleted and must be recovered by other means or re-entered, potentially a serious problem if large numbers of systems are affected!

Version-Release number of selected component (if applicable):

Problem observed on satellite 5.4.0, retested on 5.4.1 and it is still apparent (both on RHEL5)

How reproducible:
Always

Steps to Reproduce:
1.Create a kickstart distribution, call it tmpdistributionA
2.Create a kickstart profile,call it tmpksprofile, associate it with tmpdistributionA
3.Add some systems associated with tmpksprofile via "cobbler system add"
4.Create a kickstart distribution, call it tmpdistributionB
5.Modify tmpksprofile to be associated with tmpdistributionB
6.Verify that the cobbler view of the profile is still pointing at tmpdistributionA
7.Delete (the now supposedly unused) tmpdistributionA via the satellite web UI
8.Observe that tmpksprofile and any associated systems are deleted from cobbler
9.Attempt to access tmpksprofile via the satellite UI, observe "Internal Server Error"
10.<partial workaround> restart satellite via "rhn-satellite restart", observe that after a while tmpksprofile is restored
11. Observe that the cobbler systems are permanently deleted!!
  
Actual results:

Deleting an unused kickstart distribution via the satellite web UI results in unrelated profiles and systems being deleted.

Expected results:

Deleting an unused kickstart distribution should have no side effects and should definitely not delete unrelated profiles and systems :)

Additional info:

Timeline of the problem, recreated using the attached script:

root@satellite_vm cobbler_bug]# ./recreate.sh
Thu Aug 18 20:01:09 BST 2011 : profile=shtmp_ksprofile
spacecmd -u rhn-admin -p rhn-admin -- kickstart_details shtmp_ksprofile 2>/dev/null | grep "^Tree"
Tree:        shtmp_distributionA

Thu Aug 18 20:01:11 BST 2011Changing distro to shtmp_distributionB for profile shtmp_ksprofile
spacecmd -u rhn-admin -p rhn-admin -- kickstart_setdistribution shtmp_ksprofile shtmp_distributionB 2>/dev/null

Checking the kickstart distro via the satellite API:
spacecmd -u rhn-admin -p rhn-admin -- kickstart_details shtmp_ksprofile 2>/dev/null | grep "^Tree"
Tree:        shtmp_distributionB

cobbler profile list:
   shtmp_ksprofile:1:RedHat
cobbler system list:
   foobar1
   foobar10
   foobar2
   foobar3
   foobar4
   foobar5
   foobar6
   foobar7
   foobar8
   foobar9

Now checking it via the cobbler profile report - it's not the same...
Thu Aug 18 20:01:15 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
Thu Aug 18 20:01:26 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
Thu Aug 18 20:01:37 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
Thu Aug 18 20:01:47 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
Thu Aug 18 20:01:58 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
Thu Aug 18 20:02:09 BST 2011
Distribution                   : shtmp_distributionA:1:RedHat
We've waited a minute, now deleting the old distro shtmp_distributionA as we don't need it anymore...

spacecmd -y -u rhn-admin -p rhn-admin -- distribution_delete shtmp_distributionA 2>/dev/null
cobbler profile list:
cobbler system list:

 - It's clear that deleting the old, now unrelated distribution has caused
   cobbler to delete not only the profile, but also all of the systems
associated with that profile, browsing /var/lib/cobbler/config/ shows the
files are gone.

 - At this point trying to browse the KS profile via the web UI give "Internal
   server error"
"
error Internal Server Error

The server experienced a problem which prevented your request from being
filled out. It may not be possible to execute this action at this time.

Please help us correct this problem by contacting us with details of how you
received this message
"

 - Restarting satellite via rhn-satellite restart provides a means to recover
   the lost profile, which is now associated with the right profile:

rhn-satellite restart && <wait a while>

[root@satellite_vm config]# cobbler profile list
   shtmp_ksprofile:1:RedHat
[root@satellite_vm config]# cobbler profile report | grep "^Dist"
Distribution                   : shtmp_distributionB:1:RedHat

 - However the systems are gone forever :-O

Info from the cobbler log confirms:
START RECREATE
Thu Aug 18 20:01:07 2011 - INFO | REMOTE find_items(profile); criteria({'uid': 'MTMxMzY5MjQ4OS42OTYwMDY0NjguMjY2NTg'}); sort(None); user(?)
Thu Aug 18 20:01:07 2011 - INFO | find_items; ['profile']
Thu Aug 18 20:01:10 2011 - INFO | REMOTE find_items(profile); criteria({'uid': 'MTMxMzY5MjQ4OS42OTYwMDY0NjguMjY2NTg'}); sort(None); user(?)
Thu Aug 18 20:01:10 2011 - INFO | find_items; ['profile']
Thu Aug 18 20:01:13 2011 - INFO | REMOTE find_items(profile); criteria({'uid': 'MTMxMzY5MjQ4OS42OTYwMDY0NjguMjY2NTg'}); sort(None); user(?)
Thu Aug 18 20:01:13 2011 - INFO | find_items; ['profile']
Thu Aug 18 20:01:15 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:01:15 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:01:16 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:01:27 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:01:37 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:01:48 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:01:58 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:02:09 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:02:20 2011 - INFO | REMOTE find_items(distro); criteria({'uid': 'MTMxMzY5MjM1OC4wMTEyMjQyODEuOTg3NDY'}); sort(None); user(?)
Thu Aug 18 20:02:20 2011 - INFO | find_items; ['distro']
Thu Aug 18 20:02:20 2011 - INFO | REMOTE remove_item (distro, recursive=1); user(rhn-admin); name(shtmp_distributionA:1:RedHat)
Thu Aug 18 20:02:20 2011 - DEBUG | authorize; ['rhn-admin', 'remove_item', 'shtmp_distributionA:1:RedHat', None, True]
Thu Aug 18 20:02:20 2011 - DEBUG | REMOTE rhn-admin authorization result: True; user(?)
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['distro', 'shtmp_distributionA:1:RedHat']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(distro); ['shtmp_distributionA:1:RedHat']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['distro']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['profile', 'shtmp_ksprofile:1:RedHat']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(profile); ['shtmp_ksprofile:1:RedHat']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar1']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar1']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar10']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar10']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar2']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar2']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar3']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar3']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar4']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar4']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar5']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar5']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar6']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar6']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar7']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar7']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar8']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar8']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:20 2011 - DEBUG | get_item; ['system', 'foobar9']
Thu Aug 18 20:02:20 2011 - INFO | remove_item(system); ['foobar9']
Thu Aug 18 20:02:20 2011 - DEBUG | get_items; ['system']
Thu Aug 18 20:02:31 2011 - DEBUG | get_items; ['profile']
Thu Aug 18 20:02:31 2011 - DEBUG | get_items; ['system']

taskomatic log has lots of errors like this:
INFO   | jvm 1    | 2011/08/18 20:00:00 |       at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
INFO   | jvm 1    | 2011/08/18 20:00:00 | 2011-08-18 20:00:00,507 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet.
INFO   | jvm 1    | 2011/08/18 20:00:00 | 2011-08-18 20:00:00,508 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#3]
INFO   | jvm 1    | 2011/08/18 20:00:00 | 2011-08-18 20:00:00,509 [DefaultQuartzScheduler_Worker-9] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask -
INFO   | jvm 1    | 2011/08/18 20:00:00 | 2011-08-18 20:00:00,604 [DefaultQuartzScheduler_Worker-2] INFO  com.redhat.rhn.taskomatic.task.SessionCleanup - session_cleanup: starting delete of stale sessions
INFO   | jvm 1    | 2011/08/18 20:01:00 | 2011-08-18 20:01:00,344 [DefaultQuartzScheduler_Worker-8] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - RuntimeExceptioneError trying to sync to cobbler: No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#3]
INFO   | jvm 1    | 2011/08/18 20:01:00 | org.hibernate.ObjectNotFoundException: No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#3]
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.impl.SessionFactoryImpl$1.handleEntityNotFound(SessionFactoryImpl.java:377)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.event.def.DefaultLoadEventListener.load(DefaultLoadEventListener.java:145)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:195)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.impl.SessionImpl.internalLoad(SessionImpl.java:846)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:567)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.type.ManyToOneType.assemble(ManyToOneType.java:196)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:130)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2161)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2126)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.loader.Loader.list(Loader.java:2096)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:378)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:341)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:172)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1121)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.hibernate.impl.QueryImpl.list(QueryImpl.java:79)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.common.hibernate.HibernateFactory.listObjectsByNamedQuery(HibernateFactory.java:265)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.domain.kickstart.KickstartFactory.lookupKickstartTrees(KickstartFactory.java:620)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.manager.kickstart.cobbler.CobblerDistroSyncCommand.store(CobblerDistroSyncCommand.java:134)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.taskomatic.task.CobblerSyncTask.execute(CobblerSyncTask.java:96)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.taskomatic.task.RhnJavaJob.execute(RhnJavaJob.java:80)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at com.redhat.rhn.taskomatic.TaskoJob.execute(TaskoJob.java:168)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
INFO   | jvm 1    | 2011/08/18 20:01:00 |       at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
INFO   | jvm 1    | 2011/08/18 20:01:00 | 2011-08-18 20:01:00,344 [DefaultQuartzScheduler_Worker-8] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - re-throwing exception since we havent yet.
INFO   | jvm 1    | 2011/08/18 20:01:00 | 2011-08-18 20:01:00,347 [DefaultQuartzScheduler_Worker-8] ERROR com.redhat.rhn.taskomatic.task.CobblerSyncTask - No row with the given identifier exists: [com.redhat.rhn.domain.kickstart.KickstartableTree#3]