fleshing out architecture docs

Change-Id: I60e59ad7a8140985eddb86ece09e7d45bb3dbfa0

3 files changed, 171 insertions, 66 deletions

diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst
index d80e7289..b6498a08 100644
--- a/docs/source/architecture.rst
+++ b/docs/source/architecture.rst
@@ -24,11 +24,11 @@ for most deployments will actually be shims in front of existing user systems.
 The Services
 ------------
 
-Keystone is organized as a group of services exposed on one or many endpoints.
-Many of these services are used in a combined fashion by the frontend, for
-example an authenticate call will validate user/tenant credentials with the
-Identity service and, upon success, create and return a token with the Token
-service.
+Keystone is organized as a group of internal services exposed on one or many
+endpoints. Many of these services are used in a combined fashion by the
+frontend, for example an authenticate call will validate user/tenant
+credentials with the Identity service and, upon success, create and return a
+token with the Token service.
 
 
 Identity
@@ -64,28 +64,65 @@ Policy
 The Policy service provides a rule-based authorization engine and the
 associated rule management interface.
 
-----------
-Data Model
-----------
+------------------------
+Application Construction
+------------------------
 
-Keystone was designed from the ground up to be amenable to multiple styles of
-backends and as such many of the methods and data types will happily accept
-more data than they know what to do with and pass them on to a backend.
+Keystone is an HTTP front-end to several services. Like other OpenStack
+applications, this is done using python WSGI interfaces and applications are
+configured together using Paste_. The application's HTTP endpoints are made up
+of pipelines of WSGI middleware, such as::
 
-There are a few main data types:
+    [pipeline:public_api]
+    pipeline = token_auth admin_token_auth json_body debug ec2_extension public_service
 
- * **User**: has account credentials, is associated with one or more tenants
- * **Tenant**: unit of ownership in openstack, contains one or more users
- * **Role**: a first-class piece of metadata associated with many user-tenant pairs.
- * **Token**: identifying credential associated with a user or user and tenant
- * **Extras**: bucket of key-value metadata associated with a user-tenant pair.
- * **Rule**: describes a set of requirements for performing an action.
+These in turn use a subclass of :mod:`keystone.common.wsgi.ComposingRouter` to
+link URLs to Controllers (a subclass of
+:mod:`keystone.common.wsgi.Application`). Within each Controller, one or more
+Managers are loaded (for example, see :mod:`keystone.catalog.core.Manager`),
+which are thin wrapper classes which load the appropriate service driver based
+on the keystone configuration.
 
-While the general data model allows a many-to-many relationship between Users
-and Tenants and a many-to-one relationship between Extras and User-Tenant pairs,
-the actual backend implementations take varying levels of advantage of that
-functionality.
+* Identity
+ * :mod:`keystone.identity.core.TenantController`
+ * :mod:`keystone.identity.core.UserController`
+ * :mod:`keystone.identity.core.RoleController`
+
+* Catalog
+ * :mod:`keystone.catalog.core.ServiceController`
+ * :mod:`keystone.service.VersionController`
+
+* Token
+ * :mod:`keystone.service.TokenController`
+
+* Misc
+ * :mod:`keystone.service.ExtensionsController`
+
+At this time, the policy service and associated manager is not exposed as a URL
+frontend, and has no associated Controller class.
+
+
+.. _Paste: http://pythonpaste.org/
+
+----------------
+Service Backends
+----------------
+
+Each of the services can configured to use a backend to allow Keystone to fit a
+variety of environments and needs. The backend for each service is defined in
+the keystone.conf file with the key ``driver`` under a group associated with
+each service.
+
+A general class under each backend named ``Driver`` exists to provide an
+abstract base class for any implementations, identifying the expected service
+implementations. The drivers for the services are:
 
+* :mod:`keystone.identity.core.Driver`
+* :mod:`keystone.token.core.Driver`
+
+If you implement a backend driver for one of the keystone services, you're
+expected to subclass from these classes. The default response for the defined
+apis in these Drivers is to raise a :mod:`keystone.service.TokenController`.
 
 KVS Backend
 -----------
@@ -97,6 +134,15 @@ dict.
 Supports all features of the general data model.
 
 
+SQL Backend
+-----------
+
+A SQL based backend using SQLAlchemy to store data persistently. The
+keystone-manage command introspects the backends to identify SQL based backends
+when running "db_sync" to establish or upgrade schema. If the backend driver
+has a method db_sync(), it will be invoked to sync and/or migrate schema.
+
+
 PAM Backend
 -----------
 
@@ -121,6 +167,28 @@ interpolation)::
   catalog.RegionOne.identity.internalURL = http://localhost:$(public_port)s/v2.0
   catalog.RegionOne.identity.name = 'Identity Service'
 
+----------
+Data Model
+----------
+
+Keystone was designed from the ground up to be amenable to multiple styles of
+backends and as such many of the methods and data types will happily accept
+more data than they know what to do with and pass them on to a backend.
+
+There are a few main data types:
+
+ * **User**: has account credentials, is associated with one or more tenants
+ * **Tenant**: unit of ownership in openstack, contains one or more users
+ * **Role**: a first-class piece of metadata associated with many user-tenant pairs.
+ * **Token**: identifying credential associated with a user or user and tenant
+ * **Extras**: bucket of key-value metadata associated with a user-tenant pair.
+ * **Rule**: describes a set of requirements for performing an action.
+
+While the general data model allows a many-to-many relationship between Users
+and Tenants and a many-to-one relationship between Extras and User-Tenant pairs,
+the actual backend implementations take varying levels of advantage of that
+functionality.
+
 
 ----------------
 Approach to CRUD
@@ -130,8 +198,10 @@ While it is expected that any "real" deployment at a large company will manage
 their users, tenants and other metadata in their existing user systems, a
 variety of CRUD operations are provided for the sake of development and testing.
 
-CRUD is treated as an extension or additional feature to the core feature set in
-that it is not required that a backend support it.
+CRUD is treated as an extension or additional feature to the core feature set
+in that it is not required that a backend support it. It is expected that
+backends for services that don't support the CRUD operations will raise a
+:mod:`NotImplementedError`.
 
 
 ----------------------------------
@@ -141,15 +211,15 @@ Approach to Authorization (Policy)
 Various components in the system require that different actions are allowed
 based on whether the user is authorized to perform that action.
 
-For the purposes of Keystone Light there are only a couple levels of
-authorization being checked for:
+For the purposes of Keystone there are only a couple levels of authorization
+being checked for:
 
  * Require that the performing user is considered an admin.
  * Require that the performing user matches the user being referenced.
 
 Other systems wishing to use the policy engine will require additional styles
 of checks and will possibly write completely custom backends. Backends included
-in Keystone Light are:
+in Keystone are:
 
 
 Trivial True
diff --git a/docs/source/community.rst b/docs/source/community.rst
index 1fe500ee..975d1bdc 100644
--- a/docs/source/community.rst
+++ b/docs/source/community.rst
@@ -18,16 +18,18 @@
 Getting Involved
 ================
 
-The OpenStack community is a very friendly group and there are places online to join in with the 
-community. Feel free to ask questions. This document points you to some of the places where you can
-communicate with people.
+The OpenStack community is a very friendly group and there are places online to
+join in with the community. Feel free to ask questions. This document points
+you to some of the places where you can communicate with people.
 
 How to Join the Community
 =========================
 
-Our community welcomes all people interested in open source cloud computing, and there are no formal
-membership requirements. The best way to join the community is to talk with others online or at a meetup
-and offer contributions through Launchpad_, the wiki_, or blogs. We welcome all types of contributions,
+Our community welcomes all people interested in open source cloud computing,
+and there are no formal membership requirements. The best way to join the
+community is to talk with others online or at a meetup and offer contributions
+through Launchpad_, the wiki_, or blogs. We welcome all types of contributions,
+
 from blueprint designs to documentation to testing to deployment scripts.
 
 .. _Launchpad: https://launchpad.net/keystone
@@ -36,25 +38,32 @@ from blueprint designs to documentation to testing to deployment scripts.
 #openstack on Freenode IRC Network
 ----------------------------------
 
-There is a very active chat channel at `<irc://freenode.net/#openstack>`_.  This
-is usually the best place to ask questions and find your way around. IRC stands for Internet Relay
-Chat and it is a way to chat online in real time. You can also ask a question and come back to the 
-log files to read the answer later. Logs for the #openstack IRC channel are stored at 
+There is a very active chat channel at `<irc://freenode.net/#openstack>`_. This
+is usually the best place to ask questions and find your way around. IRC stands
+for Internet Relay Chat and it is a way to chat online in real time. You can
+also ask a question and come back to the log files to read the answer later.
+Logs for the #openstack IRC channel are stored at
 `<http://eavesdrop.openstack.org/irclogs/>`_.
 
+
 OpenStack Wiki
 --------------
 
-The wiki is a living source of knowledge.  It is edited by the community, and
-has collections of links and other sources of information. Typically the pages are a good place
-to write drafts for specs or documentation, describe a blueprint, or collaborate with others.
+The wiki is a living source of knowledge. It is edited by the community, and
+has collections of links and other sources of information. Typically the pages
+are a good place to write drafts for specs or documentation, describe a
+blueprint, or collaborate with others.
 
 `OpenStack Wiki <http://wiki.openstack.org/>`_
 
+* `useful keystone project links <http://wiki.openstack.org/keystone>`_
+
 Keystone on Launchpad
 ---------------------
 
-Launchpad is a code hosting that OpenStack is using to track bugs, feature work, and releases of OpenStack. Like other OpenStack projects, Keystone source code is hosted on GitHub
+Launchpad is a code hosting that OpenStack is using to track bugs, feature
+work, and releases of OpenStack. Like other OpenStack projects, Keystone source
+code is hosted on GitHub
 
 * `Keystone Project Page on Launchpad <http://launchpad.net/keystone>`_
 * `Keystone Source Repository on GitHub <http://github.com/openstack/keystone>`_
@@ -68,12 +77,16 @@ events and posts from OpenStack contributors.
 
 `OpenStack Blog <http://openstack.org/blog>`_
 
-See also: `Planet OpenStack <http://planet.openstack.org/>`_, an aggregation of blogs
-about OpenStack from around the internet, combined into a web site and RSS feed. If you'd like to
-contribute with your blog posts, there are instructions for `adding your blog <http://wiki.openstack.org/AddingYourBlog>`_.
+See also: `Planet OpenStack <http://planet.openstack.org/>`_, an aggregation of
+blogs about OpenStack from around the internet, combined into a web site and
+RSS feed. If you'd like to contribute with your blog posts, there are
+instructions for `adding your blog <http://wiki.openstack.org/AddingYourBlog>`_.
+
 
 Twitter
 -------
 
-Because all the cool kids do it: `@openstack <http://twitter.com/openstack>`_. Also follow the 
-`#openstack <http://search.twitter.com/search?q=%23openstack>`_ tag for relevant tweets.
+Because all the cool kids do it: `@openstack <http://twitter.com/openstack>`_.
+Also follow the `#openstack <http://search.twitter.com/search?q=%23openstack>`_
+tag for relevant tweets.
+
diff --git a/docs/source/developing.rst b/docs/source/developing.rst
index f23732be..11207a95 100644
--- a/docs/source/developing.rst
+++ b/docs/source/developing.rst
@@ -21,9 +21,11 @@ Developing with Keystone
 Contributing Code
 =================
 
-To contribute code, sign up for a Launchpad account and sign a contributor license agreement,
-available on the `<http://wiki.openstack.org/CLA>`_. Once the CLA is signed you
-can contribute code through the Gerrit version control system which is related to your Launchpad account.
+To contribute code, sign up for a Launchpad account and sign a contributor
+license agreement, available on the `<http://wiki.openstack.org/CLA>`_. Once
+the CLA is signed you can contribute code through the Gerrit version control
+system which is related to your Launchpad account.
+
 
 To contribute tests, docs, code, etc, refer to our `Gerrit-Jenkins-Github Workflow`_.
 
@@ -32,8 +34,11 @@ To contribute tests, docs, code, etc, refer to our `Gerrit-Jenkins-Github Workfl
 Setup
 -----
 
-Get your development environment set up according to :doc:`setup`. The instructions from here will
-assume that you have installed keystone into a virtualenv. If you chose not to, simply exclude "tools/with_venv.sh" from the example commands below.
+Get your development environment set up according to :doc:`setup`. The
+instructions from here will assume that you have installed keystone into a
+virtualenv. If you chose not to, simply exclude "tools/with_venv.sh" from the
+example commands below.
+
 
 Running Keystone
 ----------------
@@ -42,24 +47,30 @@ To run the keystone Admin and API server instances, use::
 
     $ tools/with_venv.sh bin/keystone-all
 
-this runs keystone with the configuration the etc/ directory of the project. See :doc:`configuration` for details on how Keystone is configured.
+this runs keystone with the configuration the etc/ directory of the project.
+See :doc:`configuration` for details on how Keystone is configured. By default,
+keystone is configured with KVS backends, so any data entered into keystone run
+in this fashion will not persist across restarts.
+
 
 Interacting with Keystone
 -------------------------
 
-You can interact with Keystone through the command line using :doc:`man/keystone-manage`
-which allows you to establish tenants, users, etc.
+You can interact with Keystone through the command line using
+:doc:`man/keystone-manage` which allows you to establish tenants, users, etc.
+
 
 You can also interact with Keystone through it's REST API. There is a python
-keystone client library `python-keystoneclient`_ which interacts exclusively through
-the REST API, and which keystone itself uses to provide it's command-line interface.
+keystone client library `python-keystoneclient`_ which interacts exclusively
+through the REST API, and which keystone itself uses to provide it's
+command-line interface.
 
 When initially getting set up, after you've configured which databases to use,
-you're probably going to need to run the following to your database schema in place ::
+you're probably going to need to run the following to your database schema in
+place::
 
     $ bin/keystone-manage db_sync
 
-
 .. _`python-keystoneclient`: https://github.com/openstack/python-keystoneclient
 
 Running Tests
@@ -107,20 +118,30 @@ built-in test runner, one migration at a time.
 
     This may leave your database in an inconsistent state; attempt this in non-production environments only!
 
-This is useful for testing the *next* migration in sequence (both forward & backward) in a database under version control::
+This is useful for testing the *next* migration in sequence (both forward &
+backward) in a database under version control::
+
 
     python keystone/common/sql/migrate_repo/manage.py test \
-	--url=sqlite:///test.db \
-	--repository=keystone/common/sql/migrate_repo/
+    --url=sqlite:///test.db \
+    --repository=keystone/common/sql/migrate_repo/
+
+This command references to a SQLite database (test.db) to be used. Depending on
+the migration, this command alone does not make assertions as to the integrity
+of your data during migration.
 
-This command references to a SQLite database (test.db) to be used. Depending on the migration, this command alone does not make assertions as to the integrity of your data during migration.
 
 Writing Tests
 -------------
 
-To add tests covering all drivers, update the base test class (``test_backend.py``, ``test_legacy_compat.py``, and ``test_keystoneclient.py``).
+To add tests covering all drivers, update the base test class
+(``test_backend.py``, ``test_legacy_compat.py``, and
+``test_keystoneclient.py``).
+
+To add new drivers, subclass the ``test_backend.py`` (look towards
+``test_backend_sql.py`` or ``test_backend_kvs.py`` for examples) and update the
+configuration of the test class in ``setUp()``.
 
-To add new drivers, subclass the ``test_backend.py`` (look towards ``test_backend_sql.py`` or ``test_backend_kvs.py`` for examples) and update the configuration of the test class in ``setUp()``.
 
 Further Testing
 ---------------
@@ -128,8 +149,9 @@ Further Testing
 devstack_ is the *best* way to quickly deploy keystone with the rest of the
 OpenStack universe and should be critical step in your development workflow!
 
-You may also be interested in either the `OpenStack Continuous Integration Project`_
-or the `OpenStack Integration Testing Project`_.
+You may also be interested in either the
+`OpenStack Continuous Integration Project`_ or the
+`OpenStack Integration Testing Project`_.
 
 .. _devstack: http://devstack.org/
 .. _OpenStack Continuous Integration Project: https://github.com/openstack/openstack-ci