Address race condition from concurrent task state update

task_state acts like a guard to avoid concurrent tasks to be
scheduled. There might be two race conditions:

 1) two tasks are concurrently accepted by api, check task_state to be
 None and allow the tasks to be executed concurrently.

 2) one ordinary task is running, so that task_state is set. The delete
 task is accepted at API, and will "take over" and change task_state to
 DELETING. However the first task may continue to update task_state or
 set it to None as it finishes.

This patch specifies current expected task_state when updating task_state.
If unexpected state is met, abort the task without updating. Various
compute tests are fixed accordingly to set the pre condition of the
task state.

Part of bug 1037372
Change-Id: I5fdf0946c728a47febb56ad468043a828b2736c8

1 files changed, 13 insertions, 0 deletions

diff --git a/nova/db/sqlalchemy/api.py b/nova/db/sqlalchemy/api.py
index 3f6c504ab..d9c157c21 100644
--- a/nova/db/sqlalchemy/api.py
+++ b/nova/db/sqlalchemy/api.py
@@ -1745,6 +1745,10 @@ def instance_update_and_get_original(context, instance_uuid, values):
     :param instance_uuid: = instance uuid
     :param values: = dict containing column values
 
+    If "expected_task_state" exists in values, the update can only happen
+    when the task state before update matches expected_task_state. Otherwise
+    a UnexpectedTaskStateError is thrown.
+
     :returns: a tuple of the form (old_instance_ref, new_instance_ref)
 
     Raises NotFound if instance does not exist.
@@ -1762,6 +1766,15 @@ def _instance_update(context, instance_uuid, values, copy_old_instance=False):
     with session.begin():
         instance_ref = instance_get_by_uuid(context, instance_uuid,
                                             session=session)
+        if "expected_task_state" in values:
+            # it is not a db column so always pop out
+            expected = values.pop("expected_task_state")
+            if not isinstance(expected, (tuple, list, set)):
+                expected = (expected,)
+            actual_state = instance_ref["task_state"]
+            if actual_state not in expected:
+                raise exception.UnexpectedTaskStateError(actual=actual_state,
+                                                         expected=expected)
 
         if copy_old_instance:
             old_instance_ref = copy.copy(instance_ref)