Cryptographically Signed tokens

Uses CMS to create tokens that can be verified without network calls.

Tokens encapsulate authorization information.
This includes user name and roles in JSON.
The JSON document info is cryptographically signed with a private key
from Keystone, in accordance with the Cryptographic Message Syntax (CMS)
in DER format and then Base64 encoded.  The header, footer, and line breaks
are stripped to minimize the size,  and slashes which are  invalid in Base64
are converted to hyphens.

Since signed tokens are not validated against the Keystone server,  they
continue to be valid until the expiration time.  This means that even if a user
has their roles revoked or their account disabled, those changes will not take
effect until their token times out.  The prototype for this is Kerberos, which
has the same limitation, and has funtioned sucessfully with it for decades.  It
is possible to set the token time out for much shorter than the default of 8
hours, but that may mean that users tokens will time out prior to completion
of long running tasks.

This should be a drop in replacement for the current token production code.
Although  the signed token is longer than the older format, the token is still
a unique stream of Alpha-Numeric characters.

The auth token middle_ware is capable of handling both uuid and signed tokens.

To start with, the PKI functionality is disabled.  This will keep from breaking
the existing deployments.  However,  it can be enabled with the config value:

[signing]
disable_pki = False

The 'id_hash' column is added to the SQL schema because SQL alchemy insists on
each table having a primary key.  However primary keys are limited to roughly
250 Characters (768 Bytes,  but there is more than 1 varchar per byte) so the
ID field cannot be used as the primary key anymore.  id_hash is a hash of the
id column, and should be used for lookups as it is indexed.

middleware/auth_token.py needs to stand alone in the other services, and uses
keystone.common.cms in order to verify tokens.
Token needs to have all of the data from the original authenticate code
contained in the signed document, as the authenticate RPC will no longer
be called in mand cases.

The datetime of expiry is signed in the token.

The certificates are accessible via web APIs.  On the remote service side,
certificates needed to authenitcate tokens are stored in /tmp/keystone-signing
by default.  Remote systems use Paste API to read configuration values.
Certificates are retrieved only if they are not on the local system.

When authenticating in Keystone systems, it still does the Database checks for
token presence.  This allows Keystone to continue to enforce Timeout and
disabled users.

The service catalog has been added to the  signed token.  Although this greatly
increases the size of the token,  it makes it consistant with what is fetched
during the token authenticate checks

This change also fixes time variations in expiry test.  Although unrelated to
the above changes, it was making testing very frustrating.

For the database Upgrade scripts, we now only  bring 'token' up to V1 in 001
script.  This makes it possible to use the same 002 script for both upgrade
and initializing a new database.

Upon upgrade, the current UUID tokens are retained in the id_hash and id fields.
The mechanisms to verify uuid tokens work the same as before.  On downgrade,
token_ids are dropped.

Takes into account changes for "Raise unauthorized if tenant disabled"

    Bug 1003962

Change-Id: I89b5aa609143bbe09a36bfaf64758c5306e86de7

1 files changed, 152 insertions, 35 deletions

diff --git a/keystone/middleware/auth_token.py b/keystone/middleware/auth_token.py
index 3c379983..c82e5ef5 100644
--- a/keystone/middleware/auth_token.py
+++ b/keystone/middleware/auth_token.py
@@ -94,14 +94,17 @@ HTTP_X_ROLE
 """
 
 import httplib
+import json
 import logging
+import os
+import stat
+import subprocess
 import time
-
 import webob
 import webob.exc
 
 from keystone.openstack.common import jsonutils
-
+from keystone.common import cms
 
 LOG = logging.getLogger(__name__)
 
@@ -146,6 +149,22 @@ class AuthProtocol(object):
         self.cert_file = conf.get('certfile')
         self.key_file = conf.get('keyfile')
 
+        #signing
+        self.signing_dirname = conf.get('signing_dir', '/tmp/keystone-signing')
+        if (os.path.exists(self.signing_dirname) and
+                not os.access(self.signing_dirname, os.W_OK)):
+                raise "TODO: Need to find an Exception to raise here."
+
+        if not os.path.exists(self.signing_dirname):
+            os.makedirs(self.signing_dirname)
+        #will throw IOError  if it cannot change permissions
+        os.chmod(self.signing_dirname, stat.S_IRWXU)
+
+        val = '%s/signing_cert.pem' % self.signing_dirname
+        self.signing_cert_file_name = val
+        val = '%s/cacert.pem' % self.signing_dirname
+        self.ca_file_name = val
+
         # Credentials used to verify this component with the Auth service since
         # validating tokens is a privileged call
         self.admin_token = conf.get('admin_token')
@@ -271,6 +290,29 @@ class AuthProtocol(object):
                                           self.key_file,
                                           self.cert_file)
 
+    def _http_request(self, method, path):
+        """HTTP request helper used to make unspecified content type requests.
+
+        :param method: http method
+        :param path: relative request url
+        :return (http response object)
+        :raise ServerError when unable to communicate with keystone
+
+        """
+        conn = self._get_http_connection()
+
+        try:
+            conn.request(method, path)
+            response = conn.getresponse()
+            body = response.read()
+        except Exception, e:
+            LOG.error('HTTP connection exception: %s' % e)
+            raise ServiceError('Unable to communicate with keystone')
+        finally:
+            conn.close()
+
+        return response, body
+
     def _json_request(self, method, path, body=None, additional_headers=None):
         """HTTP request helper used to make json requests.
 
@@ -347,49 +389,30 @@ class AuthProtocol(object):
             raise ServiceError('invalid json response')
 
     def _validate_user_token(self, user_token, retry=True):
-        """Authenticate user token with keystone.
+        """Authenticate user using PKI
 
         :param user_token: user's token id
-        :param retry: flag that forces the middleware to retry
-                      user authentication when an indeterminate
-                      response is received. Optional.
-        :return token object received from keystone on success
+        :param retry: Ignored, as it is not longer relevant
+        :return uncrypted body of the token if the token is valid
         :raise InvalidUserToken if token is rejected
-        :raise ServiceError if unable to authenticate token
+        :no longer raises ServiceError since it no longer makes RPC
 
         """
-        cached = self._cache_get(user_token)
-        if cached:
-            return cached
-
-        headers = {'X-Auth-Token': self.get_admin_token()}
-        response, data = self._json_request('GET',
-                                            '/v2.0/tokens/%s' % user_token,
-                                            additional_headers=headers)
-
-        if response.status == 200:
+        try:
+            cached = self._cache_get(user_token)
+            if cached:
+                return cached
+            if (len(user_token) > cms.UUID_TOKEN_LENGTH):
+                verified = self.verify_signed_token(user_token)
+                data = json.loads(verified)
+            else:
+                data = self.verify_uuid_token(user_token, retry)
             self._cache_put(user_token, data)
             return data
-        if response.status == 404:
-            # FIXME(ja): I'm assuming the 404 status means that user_token is
-            #            invalid - not that the admin_token is invalid
+        except Exception as e:
             self._cache_store_invalid(user_token)
             LOG.warn("Authorization failed for token %s", user_token)
             raise InvalidUserToken('Token authorization failed')
-        if response.status == 401:
-            LOG.info('Keystone rejected admin token %s, resetting', headers)
-            self.admin_token = None
-        else:
-            LOG.error('Bad response code while validating token: %s' %
-                      response.status)
-        if retry:
-            LOG.info('Retrying validation')
-            return self._validate_user_token(user_token, False)
-        else:
-            LOG.warn("Invalid user token: %s. Keystone response: %s.",
-                     user_token, data)
-
-            raise InvalidUserToken()
 
     def _build_user_headers(self, token_info):
         """Convert token object into headers.
@@ -541,6 +564,100 @@ class AuthProtocol(object):
                             'invalid',
                             time=self.token_cache_time)
 
+    def cert_file_missing(self, called_proc_err, file_name):
+        return (called_proc_err.output.find(self.signing_cert_file_name)
+                and not os.path.exists(self.signing_cert_file_name))
+
+    def verify_uuid_token(self, user_token, retry=True):
+        """Authenticate user token with keystone.
+
+        :param user_token: user's token id
+        :param retry: flag that forces the middleware to retry
+                      user authentication when an indeterminate
+                      response is received. Optional.
+        :return token object received from keystone on success
+        :raise InvalidUserToken if token is rejected
+        :raise ServiceError if unable to authenticate token
+
+        """
+
+        headers = {'X-Auth-Token': self.get_admin_token()}
+        response, data = self._json_request('GET',
+                                            '/v2.0/tokens/%s' % user_token,
+                                            additional_headers=headers)
+
+        if response.status == 200:
+            self._cache_put(user_token, data)
+            return data
+        if response.status == 404:
+            # FIXME(ja): I'm assuming the 404 status means that user_token is
+            #            invalid - not that the admin_token is invalid
+            self._cache_store_invalid(user_token)
+            LOG.warn("Authorization failed for token %s", user_token)
+            raise InvalidUserToken('Token authorization failed')
+        if response.status == 401:
+            LOG.info('Keystone rejected admin token %s, resetting', headers)
+            self.admin_token = None
+        else:
+            LOG.error('Bad response code while validating token: %s' %
+                      response.status)
+        if retry:
+            LOG.info('Retrying validation')
+            return self._validate_user_token(user_token, False)
+        else:
+            LOG.warn("Invalid user token: %s. Keystone response: %s.",
+                     user_token, data)
+
+            raise InvalidUserToken()
+
+    def verify_signed_token(self, signed_text):
+        """
+            Converts a block of Base64 encoding to strict PEM format
+            and verifies the signature of the contensts IAW CMS syntax
+            If either of the certificate files are missing, fetch them
+            and retry
+        """
+
+        formatted = cms.token_to_cms(signed_text)
+
+        while True:
+            try:
+                output = cms.cms_verify(formatted, self.signing_cert_file_name,
+                                        self.ca_file_name)
+            except subprocess.CalledProcessError as err:
+                if self.cert_file_missing(err, self.signing_cert_file_name):
+                    self.fetch_signing_cert()
+                    continue
+                if self.cert_file_missing(err, self.ca_file_name):
+                    self.fetch_ca_cert()
+                    continue
+                raise err
+            return output
+
+    def fetch_signing_cert(self):
+        response, data = self._http_request('GET',
+                                            '/v2.0/certificates/signing')
+        try:
+            #todo check response
+            certfile = open(self.signing_cert_file_name, 'w')
+            certfile.write(data)
+            certfile.close()
+        except (AssertionError, KeyError):
+            LOG.warn("Unexpected response from keystone service: %s", data)
+            raise ServiceError('invalid json response')
+
+    def fetch_ca_cert(self):
+        response, data = self._http_request('GET',
+                                            '/v2.0/certificates/ca')
+        try:
+            #todo check response
+            certfile = open(self.ca_file_name, 'w')
+            certfile.write(data)
+            certfile.close()
+        except (AssertionError, KeyError):
+            LOG.warn("Unexpected response from keystone service: %s", data)
+            raise ServiceError('invalid json response')
+
 
 def filter_factory(global_conf, **local_conf):
     """Returns a WSGI filter app for use with paste.deploy."""