Merge "Added databased version check on startup w/ docs"

4 files changed, 174 insertions, 35 deletions

diff --git a/doc/source/migration.rst b/doc/source/migration.rst
index 219255d4..b2bafdcc 100644
--- a/doc/source/migration.rst
+++ b/doc/source/migration.rst
@@ -2,38 +2,118 @@
 Database Migrations
 ===================
 
-Keystone uses SQLAlchemy Migrate (``sqlalchemy-migrate``) to manage migrations.
+Keystone uses SQLAlchemy Migrate (``sqlalchemy-migrate``) to manage
+migrations.
+
+Migrations are tracked using a metadata table (``migrate_version``), which
+allows keystone to compare the state of your database to the state it
+expects, and to move between versions.
 
 .. WARNING::
 
-    Backup your database before applying migrations. Migrations may attempt to modify both your schema and data, and could result in data loss.
+    Backup your database before applying migrations. Migrations may
+    attempt to modify both your schema and data, and could result in data
+    loss.
 
-    Always review the behavior of migrations in a staging environment before applying them in production.
+    Always review the behavior of migrations in a staging environment
+    before applying them in production.
 
 Getting Started
 ===============
 
-Migrations are tracked using a metadata table. Place a new or existing schema under version control to enable migration support (SQLite in this case)::
+Your initial approach to migrations should depend on whether you have an
+empty database or a schema full of data.
+
+Starting with an empty database
+-------------------------------
+
+If you have an empty database for keystone to work with, you can simply
+run::
+
+    $ ./bin/keystone-manage database sync
+
+This command will initialize your metadata table, and run through all the
+schema & data migrations necessary to bring your database in sync with
+keystone. That's it!
+
+Starting with an existing database
+----------------------------------
 
-    $ python keystone/backends/sqlalchemy/migrate_repo/manage.py version_control --url=sqlite:///bin/keystone.db --repository=keystone/backends/sqlalchemy/migrate_repo/
+Place an existing database under version control to enable migration
+support::
 
-If you are starting with an existing schema, you **must** set your database to the appropriate schema version number using a
-SQL command. For example, if you're starting from a ``diablo`` or ``diablo/stable`` database, set your current database version to ``1``::
+    $ ./bin/keystone-manage database version_control
+
+This command simply creates a ``migrate_version`` table, set at version 0,
+which indicates that no migrations have been applied.
+
+If you are starting with an existing schema, you can set your database to
+the current schema version using a SQL command. For example, if you're
+starting from a diablo-compatible database, set your current
+database version to ``1``::
 
     UPDATE migrate_version SET version=1;
 
+Determine your appropriate database ``version`` by referencing the following table:
+
+    +------------+-------------+
+    | Release    | ``version`` |
+    +============+=============+
+    | pre-diablo | (see below) |
+    +------------+-------------+
+    | diablo     | 1           |
+    +------------+-------------+
+    | essex-m1   | 3           |
+    +------------+-------------+
+    | essex-m2   | 4           |
+    +------------+-------------+
+
+From there, you can upgrade normally (see :ref:`upgrading`).
+
+Starting with a pre-diablo database (cactus)
+--------------------------------------------
+
+You'll need to manually migrate your database to a diablo-compatible
+schema, and continue forward from there (if desired) using migrations.
+
+.. _upgrading:
+
 Upgrading & Downgrading
 =======================
 
-Fresh installs of Keystone will need to run database upgrades, which will build a schema and bootstrap it with any necessary data.
+.. note::
+
+    Attempting to start keystone with an outdated schema will cause
+    keystone to abort, to avoid corrupting your data.
+
+Upgrade to the latest version automatically::
+
+    $ ./bin/keystone-manage database sync
+
+Check your current schema version::
+
+    $ ./bin/keystone-manage database version
+
+Upgrade to a specific version::
+
+    $ ./bin/keystone-manage database upgrade <version_number>
+
+Downgrade to a specific version (will likely result in data loss!)::
+
+    $ ./bin/keystone-manage database downgrade <version_number>
 
-Upgrade::
+Opting Out of Migrations
+========================
 
-    $ python keystone/backends/sqlalchemy/migrate_repo/manage.py upgrade --url=sqlite:///bin/keystone.db --repository=keystone/backends/sqlalchemy/migrate_repo/
+If you don't want to use migrations (e.g. if you want to manage your
+schema manually), keystone will complain in your logs on startup, but
+won't actually stop you from doing so.
 
-Downgrade (will likely result in data loss!)::
+It's recommended that you use migrations to get up and running, but if
+you want to manage migrations manually after that, simply drop the
+``migrate_version`` table::
 
-    $ python keystone/backends/sqlalchemy/migrate_repo/manage.py downgrade 1 --url=sqlite:///bin/keystone.db --repository=keystone/backends/sqlalchemy/migrate_repo/
+    DROP TABLE migrate_version;
 
 Useful Links
 ============
diff --git a/keystone/backends/sqlalchemy/__init__.py b/keystone/backends/sqlalchemy/__init__.py
index db1f7d19..41fd0b3f 100755
--- a/keystone/backends/sqlalchemy/__init__.py
+++ b/keystone/backends/sqlalchemy/__init__.py
@@ -19,16 +19,26 @@
 from sqlalchemy.orm import joinedload, aliased, sessionmaker
 
 import ast
+import logging
 
 from sqlalchemy import create_engine
 from sqlalchemy.pool import StaticPool
 
+try:
+    from migrate.versioning import exceptions as versioning_exceptions
+except ImportError:
+    from migrate import exceptions as versioning_exceptions
+
 from keystone import utils
 from keystone.backends.sqlalchemy import models
+from keystone.backends.sqlalchemy import migration
 import keystone.backends.api as top_api
 import keystone.backends.models as top_models
 
 
+logger = logging.getLogger(__name__)
+
+
 _DRIVER = None
 
 
@@ -36,32 +46,57 @@ class Driver():
     def __init__(self, options):
         self.session = None
         self._engine = None
-        connection_str = options['sql_connection']
+        self.connection_str = options['sql_connection']
         model_list = ast.literal_eval(options["backend_entities"])
 
-        self._init_engine(connection_str)
+        self._init_engine(model_list)
         self._init_models(model_list)
         self._init_session_maker()
 
-    def _init_engine(self, connection_str):
-        if connection_str == "sqlite://":
-            # in-memory sqlite
+    def _init_engine(self, model_list):
+        if self.connection_str == "sqlite://":
+            # initialize in-memory sqlite (i.e. for testing)
             self._engine = create_engine(
-                connection_str,
+                self.connection_str,
                 connect_args={'check_same_thread': False},
                 poolclass=StaticPool)
+
+            # TODO(dolph): we should be using version control, but
+            # we don't have a way to pass our in-memory instance to
+            # the versioning api
+            self._init_tables(model_list)
         else:
+            # initialize a "real" database
             self._engine = create_engine(
-                connection_str,
+                self.connection_str,
                 pool_recycle=3600)
+            self._init_version_control()
+
+    def _init_version_control(self):
+        """Verify the state of the database"""
+        repo_path = migration.get_migrate_repo_path()
+
+        try:
+            repo_version = migration.get_repo_version(repo_path)
+            db_version = migration.get_db_version(self._engine, repo_path)
+
+            if repo_version != db_version:
+                msg = ('Database (%s) is not up to date (current=%s, '
+                    'latest=%s); run `keystone-manage database sync` or '
+                    'override your migrate version manually (see docs)' %
+                    (self.connection_str, db_version, repo_version))
+                logging.warning(msg)
+                raise Exception(msg)
+        except versioning_exceptions.DatabaseNotControlledError:
+            msg = ('Database (%s) is not version controlled;'
+                'run `keystone-manage database sync`' %
+                (self.connection_str))
+            logging.warning(msg)
 
     def _init_models(self, model_list):
-        tables = []
-
         for model in model_list:
             model_path = '.'.join([__package__, 'models', model])
             module = utils.import_module(model_path)
-            tables.append(module.__table__)
 
             top_models.set_value(model, module)
 
@@ -70,6 +105,14 @@ class Driver():
                 api_module = utils.import_module(api_path)
                 top_api.set_value(module.__api__, api_module.get())
 
+    def _init_tables(self, model_list):
+        tables = []
+
+        for model in model_list:
+            model_path = '.'.join([__package__, 'models', model])
+            module = utils.import_module(model_path)
+            tables.append(module.__table__)
+
         tables_to_create = []
         for table in reversed(models.Base.metadata.sorted_tables):
             if table in tables:
diff --git a/keystone/backends/sqlalchemy/migration.py b/keystone/backends/sqlalchemy/migration.py
index 0bd6d3a2..b1810179 100644
--- a/keystone/backends/sqlalchemy/migration.py
+++ b/keystone/backends/sqlalchemy/migration.py
@@ -29,7 +29,23 @@ except ImportError:
 from keystone.logic.types import fault
 
 
-logger = logging.getLogger('keystone.backends.sqlalchemy.migration')
+logger = logging.getLogger(__name__)
+
+
+def get_migrate_repo(repo_path):
+    return versioning_api.repository.Repository(repo_path)
+
+
+def get_schema(engine, repo_path):
+    return versioning_api.schema.ControlledSchema(engine, repo_path)
+
+
+def get_repo_version(repo_path):
+    return get_migrate_repo(repo_path).latest
+
+
+def get_db_version(engine, repo_path):
+    return get_schema(engine, repo_path).version
 
 
 def db_version(options):
@@ -41,7 +57,6 @@ def db_version(options):
     """
     repo_path = get_migrate_repo_path()
     sql_connection = options['sql_connection']
-    print repo_path, sql_connection
     try:
         return versioning_api.db_version(sql_connection, repo_path)
     except versioning_exceptions.DatabaseNotControlledError:
diff --git a/keystone/manage/__init__.py b/keystone/manage/__init__.py
index bf50d25a..f39c6f20 100644
--- a/keystone/manage/__init__.py
+++ b/keystone/manage/__init__.py
@@ -34,6 +34,9 @@ from keystone.logic.types import fault
 from keystone.manage import api
 
 
+logger = logging.getLogger(__name__)
+
+
 # CLI feature set
 OBJECTS = ['user', 'tenant', 'role', 'service',
     'endpointTemplates', 'token', 'endpoint', 'credentials', 'database']
@@ -80,7 +83,7 @@ def parse_args(args=None):
     # Initialize a parser for our configuration paramaters
     parser = RaisingOptionParser(usage, version='%%prog %s'
         % version.version())
-    _common_group = config.add_common_options(parser)
+    config.add_common_options(parser)
     config.add_log_options(parser)
 
     # Parse command-line and load config
@@ -89,7 +92,8 @@ def parse_args(args=None):
 
     config.setup_logging(options, conf)
 
-    db.configure_backends(conf.global_conf)
+    if not args or args[0] != 'database':
+        db.configure_backends(conf.global_conf)
 
     return args
 
@@ -97,7 +101,7 @@ def parse_args(args=None):
 def get_options(args=None):
     # Initialize a parser for our configuration paramaters
     parser = RaisingOptionParser()
-    _common_group = config.add_common_options(parser)
+    config.add_common_options(parser)
     config.add_log_options(parser)
 
     # Parse command-line and load config
@@ -140,6 +144,9 @@ def process(*args):
     optional_arg = (lambda args, x:
         len(args) > x and str(args[x]).strip() or None)
 
+    if object_type == 'database':
+        options = get_options(args)
+
     # Execute command
     if (object_type, action) == ('user', 'add'):
         require_args(args, 4, 'No password specified for fourth argument')
@@ -286,8 +293,6 @@ def process(*args):
 
     elif (object_type, action) == ('database', 'sync'):
         require_args(args, 1, 'Syncing database requires a version #')
-        options = get_options(args)
-        options = get_options(args)
         backend_names = options.get('backends', None)
         if backend_names:
             if 'keystone.backends.sqlalchemy' in backend_names.split(','):
@@ -299,7 +304,6 @@ def process(*args):
 
     elif (object_type, action) == ('database', 'upgrade'):
         require_args(args, 1, 'Upgrading database requires a version #')
-        options = get_options(args)
         backend_names = options.get('backends', None)
         if backend_names:
             if 'keystone.backends.sqlalchemy' in backend_names.split(','):
@@ -311,7 +315,6 @@ def process(*args):
 
     elif (object_type, action) == ('database', 'downgrade'):
         require_args(args, 1, 'Downgrading database requires a version #')
-        options = get_options(args)
         backend_names = options.get('backends', None)
         if backend_names:
             if 'keystone.backends.sqlalchemy' in backend_names.split(','):
@@ -322,7 +325,6 @@ def process(*args):
                     'SQL alchemy backend not specified in config')
 
     elif (object_type, action) == ('database', 'version_control'):
-        options = get_options(args)
         backend_names = options.get('backends', None)
         if backend_names:
             if 'keystone.backends.sqlalchemy' in backend_names.split(','):
@@ -332,7 +334,6 @@ def process(*args):
                     'SQL alchemy backend not specified in config')
 
     elif (object_type, action) == ('database', 'version'):
-        options = get_options(args)
         backend_names = options.get('backends', None)
         if backend_names:
             if 'keystone.backends.sqlalchemy' in backend_names.split(','):
@@ -457,10 +458,10 @@ def main(args=None):
             info = exc.args[1]
         except IndexError:
             print "ERROR: %s" % (exc,)
-            logging.error(str(exc))
+            logger.error(str(exc))
         else:
             print "ERROR: %s: %s" % (exc.args[0], info)
-            logging.error(exc.args[0], exc_info=info)
+            logger.error(exc.args[0], exc_info=info)
         raise exc