Proposing merge to get feedback on orm refactoring. I am very interested in feedback to all of these changes.

This is a huge set of changes, that touches almost all of the files.  I'm sure I have broken quite a bit, but better to take the plunge now than to postpone this until later.  The idea is to allow for pluggable backends throughout the code.

Brief Overview
For compute/volume/network, there are multiple classes
service - responsible for rpc
  this currently uses the existing cast and call in rpc.py and a little bit of magic
  to call public methods on the manager class.
  each service also reports its state into the database every 10 seconds
manager - responsible for managing respective object classes
  all the business logic for the classes go here
db (db_driver) - responsible for abstracting database access
driver (domain_driver) - responsible for executing actual shell commands and implementation

Compute hasn't been fully cleaned up, but to get an idea of how it works, take a look
at volume and network

Known issues/Things to be done:

* nova-api accesses db objects directly
  It seems cleaner to have only the managers dealing with their respective objects.  This would
  mean code for 'run_instances' would move into the manager class and it would do the initial
  setup and cast out to the remote service

* db code uses flat methods to define its interface
  In my mind this is a little prettier as an abstract base class, but driver loading code
  can load a module or a class.  It works, so I'm not sure it needs to be changed but feel
  free to debate it.

* Service classes have no code in them
  Not sure if this is a problem for people, but the magic of calling the manager's methods is
  done in the base class.  We could remove the magic from the base class and explicitly
  wrap methods that we want to make available via rpc if this seems nasty.

* AuthManager Projects/Users/Roles are not integrated into this system.
  In order for everything to live happily in the backend, we need some type
  of adaptor for LDAP

* Context is not passed properly across rabbit
  Context should probably be changed to a simple dictionary so that it can be
  passed properly through the queue

* No authorization checks on access to objects
  We need to decide on which layer auth checks should happen.

* Some of the methods in ComputeManager need to be moved into other layers/managers
* Compute driver layer should be abstracted more cleanly
* Flat networking is untested and may need to be reworked
* Some of the api commands are not working yet
* Nova Swift Authentication needs to be refactored(Todd is working on this)

6 files changed, 79 insertions, 37 deletions

diff --git a/bin/nova-compute b/bin/nova-compute
index 631d2c85c..1724e9659 100755
--- a/bin/nova-compute
+++ b/bin/nova-compute
@@ -32,12 +32,12 @@ possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
     sys.path.insert(0, possible_topdir)
 
+from nova import service
 from nova import twistd
-from nova.compute import service
 
 
 if __name__ == '__main__':
     twistd.serve(__file__)
 
 if __name__ == '__builtin__':
-    application = service.ComputeService.create() # pylint: disable-msg=C0103
+    application = service.Service.create()  # pylint: disable=C0103
diff --git a/bin/nova-dhcpbridge b/bin/nova-dhcpbridge
index 0c3d987a7..a127ed03c 100755
--- a/bin/nova-dhcpbridge
+++ b/bin/nova-dhcpbridge
@@ -33,24 +33,32 @@ possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
     sys.path.insert(0, possible_topdir)
 
+from nova import db
 from nova import flags
 from nova import rpc
 from nova import utils
 from nova.network import linux_net
-from nova.network import model
-from nova.network import service
 
 FLAGS = flags.FLAGS
+flags.DECLARE('auth_driver', 'nova.auth.manager')
+flags.DECLARE('redis_db', 'nova.datastore')
+flags.DECLARE('network_size', 'nova.network.manager')
+flags.DECLARE('num_networks', 'nova.network.manager')
+flags.DECLARE('update_dhcp_on_disassociate', 'nova.network.manager')
 
 
-def add_lease(_mac, ip_address, _hostname, _interface):
+def add_lease(mac, ip_address, _hostname, _interface):
     """Set the IP that was assigned by the DHCP server."""
     if FLAGS.fake_rabbit:
-        service.VlanNetworkService().lease_ip(ip_address)
+        logging.debug("leasing ip")
+        network_manager = utils.import_object(FLAGS.network_manager)
+        network_manager.lease_fixed_ip(None, mac, ip_address)
     else:
-        rpc.cast("%s.%s" % (FLAGS.network_topic, FLAGS.node_name),
-                 {"method": "lease_ip",
-                  "args": {"fixed_ip": ip_address}})
+        rpc.cast("%s.%s" % (FLAGS.network_topic, FLAGS.host),
+                 {"method": "lease_fixed_ip",
+                  "args": {"context": None,
+                           "mac": mac,
+                           "address": ip_address}})
 
 
 def old_lease(_mac, _ip_address, _hostname, _interface):
@@ -58,23 +66,24 @@ def old_lease(_mac, _ip_address, _hostname, _interface):
     logging.debug("Adopted old lease or got a change of mac/hostname")
 
 
-def del_lease(_mac, ip_address, _hostname, _interface):
+def del_lease(mac, ip_address, _hostname, _interface):
     """Called when a lease expires."""
     if FLAGS.fake_rabbit:
-        service.VlanNetworkService().release_ip(ip_address)
+        logging.debug("releasing ip")
+        network_manager = utils.import_object(FLAGS.network_manager)
+        network_manager.release_fixed_ip(None, mac, ip_address)
     else:
-        rpc.cast("%s.%s" % (FLAGS.network_topic, FLAGS.node_name),
-                 {"method": "release_ip",
-                  "args": {"fixed_ip": ip_address}})
+        rpc.cast("%s.%s" % (FLAGS.network_topic, FLAGS.host),
+                 {"method": "release_fixed_ip",
+                  "args": {"context": None,
+                           "mac": mac,
+                           "address": ip_address}})
 
 
 def init_leases(interface):
     """Get the list of hosts for an interface."""
-    net = model.get_network_by_interface(interface)
-    res = ""
-    for address in net.assigned_objs:
-        res += "%s\n" % linux_net.host_dhcp(address)
-    return res
+    network_ref = db.network_get_by_bridge(None, interface)
+    return linux_net.get_dhcp_hosts(None, network_ref['id'])
 
 
 def main():
@@ -86,10 +95,16 @@ def main():
     if int(os.environ.get('TESTING', '0')):
         FLAGS.fake_rabbit = True
         FLAGS.redis_db = 8
-        FLAGS.network_size = 32
+        FLAGS.network_size = 16
         FLAGS.connection_type = 'fake'
         FLAGS.fake_network = True
         FLAGS.auth_driver = 'nova.auth.ldapdriver.FakeLdapDriver'
+        FLAGS.num_networks = 5
+        path = os.path.abspath(os.path.join(os.path.dirname(__file__),
+                                            '..',
+                                            '_trial_temp',
+                                            'nova.sqlite'))
+        FLAGS.sql_connection = 'sqlite:///%s' % path
     action = argv[1]
     if action in ['add', 'del', 'old']:
         mac = argv[2]
diff --git a/bin/nova-manage b/bin/nova-manage
index 6e5266767..325245ac4 100755
--- a/bin/nova-manage
+++ b/bin/nova-manage
@@ -57,6 +57,8 @@ import os
 import sys
 import time
 
+import IPy
+
 # If ../nova/__init__.py exists, add ../ to Python search path, so that
 # it will override what happens to be installed in /usr/(local/)lib/python...
 possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
@@ -65,10 +67,10 @@ possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
     sys.path.insert(0, possible_topdir)
 
+from nova import db
 from nova import flags
 from nova import utils
 from nova.auth import manager
-from nova.compute import model
 from nova.cloudpipe import pipelib
 from nova.endpoint import cloud
 
@@ -81,7 +83,6 @@ class VpnCommands(object):
 
     def __init__(self):
         self.manager = manager.AuthManager()
-        self.instdir = model.InstanceDirectory()
         self.pipe = pipelib.CloudPipe(cloud.CloudController())
 
     def list(self):
@@ -96,8 +97,8 @@ class VpnCommands(object):
             vpn = self._vpn_for(project.id)
             if vpn:
                 command = "ping -c1 -w1 %s > /dev/null; echo $?"
-                out, _err = utils.execute(  command % vpn['private_dns_name'],
-                                            check_exit_code=False)
+                out, _err = utils.execute(command % vpn['private_dns_name'],
+                                          check_exit_code=False)
                 if out.strip() == '0':
                     net = 'up'
                 else:
@@ -113,9 +114,8 @@ class VpnCommands(object):
 
     def _vpn_for(self, project_id):
         """Get the VPN instance for a project ID."""
-        for instance in self.instdir.all:
-            if ('image_id' in instance.state
-                and instance['image_id'] == FLAGS.vpn_image_id
+        for instance in db.instance_get_all():
+            if (instance['image_id'] == FLAGS.vpn_image_id
                 and not instance['state_description'] in
                     ['shutting_down', 'shutdown']
                 and instance['project_id'] == project_id):
@@ -274,6 +274,37 @@ class ProjectCommands(object):
         with open(filename, 'w') as f:
             f.write(zip_file)
 
+class FloatingIpCommands(object):
+    """Class for managing floating ip."""
+
+    def create(self, host, range):
+        """Creates floating ips for host by range
+        arguments: host ip_range"""
+        for address in IPy.IP(range):
+            db.floating_ip_create(None, {'address': str(address),
+                                         'host': host})
+
+    def delete(self, ip_range):
+        """Deletes floating ips by range
+        arguments: range"""
+        for address in IPy.IP(ip_range):
+            db.floating_ip_destroy(None, str(address))
+
+
+    def list(self, host=None):
+        """Lists all floating ips (optionally by host)
+        arguments: [host]"""
+        if host == None:
+            floating_ips = db.floating_ip_get_all(None)
+        else:
+            floating_ips = db.floating_ip_get_all_by_host(None, host)
+        for floating_ip in floating_ips:
+            instance = None
+            if floating_ip['fixed_ip']:
+                instance = floating_ip['fixed_ip']['instance']['str_id']
+            print "%s\t%s\t%s" % (floating_ip['host'],
+                                  floating_ip['address'],
+                                  instance)
 
 CATEGORIES = [
     ('user', UserCommands),
@@ -281,6 +312,7 @@ CATEGORIES = [
     ('role', RoleCommands),
     ('shell', ShellCommands),
     ('vpn', VpnCommands),
+    ('floating', FloatingIpCommands)
 ]
 
 
diff --git a/bin/nova-network b/bin/nova-network
index 307795d7b..fa88aeb47 100755
--- a/bin/nova-network
+++ b/bin/nova-network
@@ -32,17 +32,12 @@ possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
     sys.path.insert(0, possible_topdir)
 
-from nova import flags
+from nova import service
 from nova import twistd
 
-from nova.network import service
-
-FLAGS = flags.FLAGS
-
 
 if __name__ == '__main__':
     twistd.serve(__file__)
 
 if __name__ == '__builtin__':
-    # pylint: disable-msg=C0103
-    application = service.type_to_class(FLAGS.network_type).create()
+    application = service.Service.create()  # pylint: disable-msg=C0103
diff --git a/bin/nova-objectstore b/bin/nova-objectstore
index 447ef9055..728f2ee5b 100755
--- a/bin/nova-objectstore
+++ b/bin/nova-objectstore
@@ -46,4 +46,4 @@ if __name__ == '__main__':
 
 if __name__ == '__builtin__':
     utils.default_flagfile()
-    application = handler.get_application() # pylint: disable-msg=C0103
+    application = handler.get_application()  # pylint: disable-msg=C0103
diff --git a/bin/nova-volume b/bin/nova-volume
index f8e2e1744..b9e235717 100755
--- a/bin/nova-volume
+++ b/bin/nova-volume
@@ -32,12 +32,12 @@ possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
     sys.path.insert(0, possible_topdir)
 
+from nova import service
 from nova import twistd
-from nova.volume import service
 
 
 if __name__ == '__main__':
     twistd.serve(__file__)
 
 if __name__ == '__builtin__':
-    application = service.VolumeService.create() # pylint: disable-msg=C0103
+    application = service.Service.create()  # pylint: disable-msg=C0103