Bug 692935

Summary: Removing a lab controller results in an ISE
Product: [Retired] Beaker Reporter: Matt Brodeur <mbrodeur>
Component: web UIAssignee: Raymond Mancy <rmancy>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 0.6CC: bpeck, dcallagh, ebaak, mcsontos, rmancy, rousseau, stl
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-05-19 05:16:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Matt Brodeur 2011-04-01 18:19:51 UTC
Description of problem:
Clicking "Remove (-)" on any lab controller in https://beaker/labcontrollers/ results in the familiar "500 Internal error"

Version-Release number of selected component (if applicable):
beaker-server-0.6.7-2.el5

How reproducible:
100%

Steps to Reproduce:
1. https://beaker/labcontrollers/
2. Click "Remove (-)"
  
Actual results:
500 Internal error

The server encountered an unexpected condition which prevented it from fulfilling the request.

Powered by CherryPy 2.3.0 


Expected results:
Lab controller is removed from the list.


Additional info:

01/Apr/2011:14:18:02 HTTP INFO Page handler: <bound method LabControllers.remove of <bkr.server.labcontroller.LabControllers object at 0x2af3630f21d0>>
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 121, in _run
    self.main()
  File "/usr/lib/python2.4/site-packages/cherrypy/_cphttptools.py", line 264, in main
    body = page_handler(*virtual_path, **self.params)
  File "<string>", line 3, in remove
  File "/usr/lib/python2.4/site-packages/turbogears/identity/conditions.py", line 207, in require
    return fn(self, *args, **kwargs)
  File "<string>", line 3, in remove
  File "/usr/lib/python2.4/site-packages/turbogears/controllers.py", line 358, in expose
    output = database.run_with_transaction(
  File "<string>", line 5, in run_with_transaction
  File "/usr/lib/python2.4/site-packages/turbogears/database.py", line 413, in sa_rwt
    session.commit()
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/scoping.py", line 98, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/session.py", line 557, in commit
    self.transaction.commit()
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/session.py", line 262, in commit
    self._prepare_impl()
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/session.py", line 246, in _prepare_impl
    self.session.flush()
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/session.py", line 789, in flush
    self.uow.flush(self, objects)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/unitofwork.py", line 237, in flush
    flush_context.execute()
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/unitofwork.py", line 449, in execute
    UOWExecutor().execute(self, tasks)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/unitofwork.py", line 937, in execute
    self.execute_delete_steps(trans, task)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/unitofwork.py", line 958, in execute_delete_steps
    self.delete_objects(trans, task)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/unitofwork.py", line 943, in delete_objects
    task.mapper._delete_obj(task.polymorphic_todelete_objects, trans)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/orm/mapper.py", line 1287, in _delete_obj
    c = connection.execute(statement, del_objects)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 844, in execute
    return Connection.executors[c](self, object, multiparams, params)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 895, in execute_clauseelement
    return self._execute_compiled(elem.compile(dialect=self.dialect, column_keys=keys, inline=len(params) > 1), distilled_params=params)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 907, in _execute_compiled
    self.__execute_raw(context)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 916, in __execute_raw
    self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 960, in _cursor_execute
    self._handle_dbapi_exception(e, statement, parameters, cursor)
  File "/usr/lib/python2.4/site-packages/sqlalchemy/engine/base.py", line 942, in _handle_dbapi_exception
    raise exceptions.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
OperationalError: (OperationalError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`beaker/recipe_set`, CONSTRAINT `recipe_set_ibfk_5` FOREIGN KEY (`lab_controller_id`) REFERENCES `lab_controller` (`id`))') u'DELETE FROM lab_controller WHERE lab_controller.id = %s' [3L]

Comment 1 Bill Peck 2011-04-01 19:09:54 UTC
Ray,

We probably have some outstanding jobs still running on that lab.  We should show the user a list of active recipes off that lab controller, and confirm that removing the lab will cancel these.

Comment 2 Raymond Mancy 2011-04-15 03:05:07 UTC
It seems to be a problem with constraints and tables that have lab_controller_id FKs.

I'm not sure if it's even a good idea to remove lab controllers. Job history will be incorrect, and it may break various parts of the code. I'd suggest that we just disable it. 

I'll close this as WONTFIX, unless someone comes up with a compelling reason for making this work.

Comment 3 Matt Brodeur 2011-04-15 18:51:05 UTC
(In reply to comment #2)
> 
> I'm not sure if it's even a good idea to remove lab controllers. Job history
> will be incorrect, and it may break various parts of the code. I'd suggest that
> we just disable it. 
> 
> I'll close this as WONTFIX, unless someone comes up with a compelling reason
> for making this work.

First, you just closed/wontfix-ed a bug about a server error generated when clicking an existing UI feature.  I would expect *AT LEAST* that the Remove link would be taken out of the UI if it wasn't being fixed.

Second, how long do you expect Beaker to be around?  If we assume we'll be rewriting or replacing this thing again in a year then I'm fine with this course.  Otherwise stop and think about what the lab controller list might look like in five years after labs have opened, closed, moved, merged, split, etc.  Right now the BNE lab controller still shows up in the selector list for all systems.  Anyone can assign their systems to this "disabled" controller.  What will the scheduler do if it finds a suitable candidate system on a disabled controller?  Does the scheduler reject distro updates from disabled controllers?

Comment 4 Bill Peck 2011-04-15 19:07:12 UTC
Agree 100% with Matt here.

I looked for all the places we reference LabController:

GuestRecipe
 - location points to lab controller, but its done via a query and has logic 
   for zero results

System
 - Can be assigned to a lab controller but can also be Null

Distro
 - Already handles distros going away from lab controllers.
 - Removing a lab controller will require us to remove all distro mappings for
   that distro

RecipeSet
 - Has lab controller reference so that multi-host recipes will be scheduled
   against same lab.
 - Again, can be Null.


Those are all the places where I see we reference lab controller.

Comment 5 Raymond Mancy 2011-04-18 00:48:19 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > 
> > I'm not sure if it's even a good idea to remove lab controllers. Job history
> > will be incorrect, and it may break various parts of the code. I'd suggest that
> > we just disable it. 
> > 
> > I'll close this as WONTFIX, unless someone comes up with a compelling reason
> > for making this work.
> 
> First, you just closed/wontfix-ed a bug about a server error generated when
> clicking an existing UI feature.  I would expect *AT LEAST* that the Remove
> link would be taken out of the UI if it wasn't being fixed.

Why would I create a bug for a removal of a feature that we haven't yet confirmed that there is no compelling reason to remove? 

> 
> Second, how long do you expect Beaker to be around? If we assume we'll be
> rewriting or replacing this thing again in a year then I'm fine with this
> course.  Otherwise stop and think about what the lab controller list might look
> like in five years after labs have opened, closed, moved, merged, split, etc. 
> Right now the BNE lab controller still shows up in the selector list for all
> systems. 

You incorrectly assume those systems will be going away once their lab controller goes away. We have a policy of not deleting those that have job histories. So if we had a million systems (of which had at least one job run on them) on the BNE lab controller, not a single one would disappear once we remove the LC regardless of how the removal of LCs are handled. 


> Anyone can assign their systems to this "disabled" controller. 

Anyone can assign their system to a functioning lab on the other side of the world anyway and equally stop their system from working. Either way, you'd have to have rocks in your head.

> What
> will the scheduler do if it finds a suitable candidate system on a disabled
> controller?  

It doesn't find suitable candidate systems on a disabled controller.

> Does the scheduler reject distro updates from disabled
> controllers?

Not sure.

Comment 6 Raymond Mancy 2011-04-18 01:05:51 UTC
(In reply to comment #4)
> Agree 100% with Matt here.
> 
> I looked for all the places we reference LabController:
> 
> GuestRecipe
>  - location points to lab controller, but its done via a query and has logic 
>    for zero results
> 
> System
>  - Can be assigned to a lab controller but can also be Null
> 
> Distro
>  - Already handles distros going away from lab controllers.
>  - Removing a lab controller will require us to remove all distro mappings for
>    that distro
> 
> RecipeSet
>  - Has lab controller reference so that multi-host recipes will be scheduled
>    against same lab.
>  - Again, can be Null.
> 
> 
> Those are all the places where I see we reference lab controller.


Right, it's not that hard to implement.

I was just thinking about the fact that we will never be able to include historical job data (of those jobs that ran against a deleted LC) in reports where we are doing a cross reference against an LC. I'm sure in the scheme of things this would be inconsequential anyway with the BNE LC (I don't think it had all that many jobs run against it's systems, but as mbrodeur said what happens in 5 years time with other lab controllers).In fact I'm not sure if management care about this kind of data or not. They probably don't, I don't know.

Though if they even think that they perhaps might at one point in the future, then we can't guarantee that they will have this information.

Comment 7 Bill Peck 2011-04-21 17:55:03 UTC
(In reply to comment #6)
> 
> Right, it's not that hard to implement.
> 
> I was just thinking about the fact that we will never be able to include
> historical job data (of those jobs that ran against a deleted LC) in reports
> where we are doing a cross reference against an LC. I'm sure in the scheme of
> things this would be inconsequential anyway with the BNE LC (I don't think it
> had all that many jobs run against it's systems, but as mbrodeur said what
> happens in 5 years time with other lab controllers).In fact I'm not sure if
> management care about this kind of data or not. They probably don't, I don't
> know.

You have a point about reporting against lab controllers.

rousseau, Would we ever need to run reports against a lab controller which has been removed.

> 
> Though if they even think that they perhaps might at one point in the future,
> then we can't guarantee that they will have this information.

Comment 8 Edward Rousseau 2011-04-26 18:09:26 UTC
It depends if we have any historical data live somewhere else or if it needs to be available on the LC for requery. Ideally we would have the data up until the time of the LC removal so the actual removal would not affect reporting. We need a longer term solution to the historical data problem ...

Comment 9 Raymond Mancy 2011-04-28 23:31:03 UTC
As we don't yet have any resolution to this issue at large it seems.
For the time being I'll remove the 'Remove' button so no one is tempted to hit it and give an ISE. 

Ed, 
Unfortunately it doesn't work like that at the moment. Our database is quite normalised and we only store lab controller names in one place. So once it's gone, it's like it never existed. 

Perhaps one option is to introduce some redundancy in tables where we store historical data (i.e storing actual names instead of ids). Of course this approach is littered with a few obvious problems.

Perhaps we could just have a 'deleted' field, like we do for jobs. Where they disappear, but are still in the DB.

Comment 12 Raymond Mancy 2011-05-10 11:27:09 UTC
Still show the lab controllers on the admin menu.
If systems still attached, warn, and then remove their lab controllers.