Cells: Don't allow active -> build

Adds support for passing 'expected_vm_state' in an instance_update dict
much like the current support for 'expected_task_state'.

For cells, when the API cell receives an instance update containing a
vm_state of BUILDING, raise an exception if the instance is in some
other state.  This addresses out-of-order messaging issues that can
cause an instance to appear to have a huge delay going ACTIVE.  (A
periodic task run can later heal the state, but it can take a long
while)

Fixes bug 1180283

Change-Id: I64252b30e2596812f3b84da64b1fc8681661d7f8

1 files changed, 16 insertions, 0 deletions

diff --git a/nova/cells/messaging.py b/nova/cells/messaging.py
index 2e2fa735a..d13677f74 100644
--- a/nova/cells/messaging.py
+++ b/nova/cells/messaging.py
@@ -31,6 +31,7 @@ from nova.cells import state as cells_state
 from nova.cells import utils as cells_utils
 from nova import compute
 from nova.compute import rpcapi as compute_rpcapi
+from nova.compute import vm_states
 from nova.consoleauth import rpcapi as consoleauth_rpcapi
 from nova import context
 from nova.db import base
@@ -810,6 +811,21 @@ class _BroadcastMessageMethods(_BaseMessageMethods):
         LOG.debug(_("Got update for instance %(instance_uuid)s: "
                 "%(instance)s") % locals())
 
+        # To attempt to address out-of-order messages, do some sanity
+        # checking on the VM state.
+        expected_vm_state_map = {
+                # For updates containing 'vm_state' of 'building',
+                # only allow them to occur if the DB already says
+                # 'building' or if the vm_state is None.  None
+                # really shouldn't be possible as instances always
+                # start out in 'building' anyway.. but just in case.
+                vm_states.BUILDING: [vm_states.BUILDING, None]}
+
+        expected_vm_states = expected_vm_state_map.get(
+                instance.get('vm_state'))
+        if expected_vm_states:
+                instance['expected_vm_state'] = expected_vm_states
+
         # It's possible due to some weird condition that the instance
         # was already set as deleted... so we'll attempt to update
         # it with permissions that allows us to read deleted.