Bug 962861 - removing compute nodes from cloud prevents VMs from launching
removing compute nodes from cloud prevents VMs from launching
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-nova (Show other bugs)
All Linux
medium Severity low
: ---
: 4.0
Assigned To: Russell Bryant
Ami Jeain
Depends On:
  Show dependency treegraph
Reported: 2013-05-14 11:33 EDT by Dan Yocum
Modified: 2015-06-04 17:51 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2013-11-14 10:37:40 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)
nova.conf from OS1 (65.59 KB, text/plain)
2013-05-14 12:40 EDT, Dan Yocum
no flags Details

  None (edit)
Description Dan Yocum 2013-05-14 11:33:49 EDT
Description of problem:
When compute nodes are removed from a currently running cloud, launching new VMs fails.

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create cloud with X compute nodes
2. Use cloud - launch/terminate VMs, 'nova-manage service list', etc.
3. Disable N compute nodes
4. Try to use the cloud and watch VMs fail to start.
Actual results:

VMs enter error state, fail to start

Expected results:

VMs start!

Additional info:

After I removed 24 compute nodes from our 64 node cloud (turned off and disabled all openstack-* services), I did 'mysql -B -e "delete from nova.services;"' to clean out the services table.  

I could only get the cloud to launch VMs again after running 'mysql -B -e "delete from nova.compute_nodes;"' and letting it re-populate.
Comment 2 Russell Bryant 2013-05-14 11:51:06 EDT
Can you share your configuration?  The choice of scheduler driver, and in the case of the filter scheduler, which filters are enabled will affect what the expected behavior is here.
Comment 3 Dan Yocum 2013-05-14 12:40:23 EDT
Created attachment 747763 [details]
nova.conf from OS1
Comment 4 Dan Yocum 2013-05-14 12:41:34 EDT
nova.conf attached - I haven't twiddled any of the scheduler knobs except for ram overcommit (2x).
Comment 5 Russell Bryant 2013-05-30 15:26:59 EDT
So here how this is *supposed* to work.  The default set of filters used by the filter scheduler is the ComputeFilter.  This filter is supposed to filter out compute nodes that are not currently up.  

Being "up" is determined by making a call to the internal servicegroup API, which has several optional backends.  The default backend is database driven.  If you turn on debug logging, you should be seeing an entry when it's checking to see if a service is up:

from nova/servicegroup/drivers/db.py:

        LOG.debug('DB_Driver.is_up last_heartbeat = %(lhb)s elapsed = %(el)s',
                  {'lhb': str(last_heartbeat), 'el': str(elapsed)})

Check the contents of the 'services' table in the nova database.  You should notice that the 'updated_at' column is getting periodically updated.  This is the value used to determine whether the service is up.

The amount of time to wait before considering a service as down is 60 seconds.  This can be changed by setting the service_down_time configuration option in nova.conf.

So, with all of that said ... how long are you waiting after shutting down the compute node?  It is expected to fail within the first minute, unless you decrease the setting for the option I mentioned.
Comment 7 Jason Dillaman 2013-07-11 10:47:37 EDT
I was only able to recreate this issue when deleting all entries from the service table as indicated in the bug description (mysql -B -e "delete from nova.services;").  The reason this causes an issue is because the non-enforced foreign key from compute_nodes.service_id is now pointing to a missing record.

Dan, I just want to verify that you deleted all services from the table and not just the services for the deleted compute hosts, correct?  May I ask where you read about this procedure?  The only documentation that I could find recommended deleting only the records from the services and compute_nodes tables that match the deleted hosts.
Comment 8 Dan Yocum 2013-07-11 11:09:24 EDT
You are correct, I did a complete delete from the nova.service table.  The schema should probably updated to prevent that sort of action by people like me with a little db knowledge.  :-D (a little knowledge can be a dangerous thing).

Comment 10 Russell Bryant 2013-11-14 10:37:40 EST
Based on the comments here, it sounds like this problem was due to hacking the db directly.

Note You need to log in before you can comment on or make changes to this bug.