Lots of work on importing rsync mirrors, including preliminary code to guess

kickstart files and assign them for distributions that we know how to minimally kickstart.

6 files changed, 162 insertions, 37 deletions

diff --git a/cobbler.pod b/cobbler.pod
index 6d2ab85..0a9673c 100644
--- a/cobbler.pod
+++ b/cobbler.pod
@@ -247,27 +247,17 @@ Usage:  B<cobbler system enchant --name=<ip|hostname> [--password=<string>]>
 
 =head2 IMPORTING TREES
 
-It is sometimes useful to import an existing directory tree containing a large amount of kernel and initrd images and make "cobbler distro add" and "cobbler profile add" calls for each of them.   The "cobbler import" command automates this, and initially names the distros and profiles based on the paths they were found in.  Profiles created in this way are very basic, and initially do not reference kickstarts.
+Cobbler can auto-add distributions and profiles, including optionally assigning basic kickstarts to them, if it can be pointed at a local filesystem tree or rsync mirror.
 
-For this to work, there can only be one kernel and initrd in each directory, anywhere down in the tree.  
+There are certain restrictions here imposed by the auto-discovery process -- the architecture (x86, i386, ia64, x86_64, etc) must be in the path name.  For kickstarts to be assigned, so must the name of the distribution.  The kernel images must be named "vmlinuz", and initrd's must be named "initrd.img".  Images must reside in a directory tree containing the string "pxe", "pxeboot", or "xen".  All of these restrictions are used to help to "automagically" set up cobbler based on a mirror.  If all of this sounds confusing, just use a Fedora rsync mirror as listed on http://fedora.redhat.com/Download/mirrors.html, and everything should be fine.
 
-Only images found in directories leaf nodes named "pxe", "pxeboot", or "xen" will be added.  Somewhere higher up in the directory tree, there needs to be a path containing something like "i386" or "x86_64" indicating the architecture.  This is because the import command tries to be smart enough to not import architectures it can't support.
-
-The import command will find new distributions and automatically add them if no existing cobbler distribution references the same files.  Existing cobbler distributions, even ones that no longer point to valid locations, will not be deleted, though warnings will be generated about orphaned distributions.
-
-Import can be run repeatedly with no adverse affect to existing configurations.
-
-Directory tree example (importing /mnt/foo)
+Usage: B<cobbler import --path=<path>>
 
-/mnt/foo/nightly/rawhide-200610111/tree-ppc <-------------- NOT imported
-/mnt/foo/nightly/rawhide-200610111/tree-i386/images/pxe <-- gets imported
-/mnt/foo/nightly/rawhide-200610111/tree-i386/images/xen <-- gets imported
-/mnt/foo/nightly/rawhide-200610111/tree-ia64/...        <-- gets imported
-/mnt/foo/nightly/rawhide-200610111/tree-x86_64/...      <-- gets imported
+Usage; B<cobbler import --mirror=<rsync_url> --mirror-name=<string>>
 
-Distributions are named after fragments of the path name seperated with underscores instead of slashes.  These names are obviously ugly so you may want to rename them after the fact.
+Example:  B<cobbler import --mirror=rsync://mirror.linux.duke.edu/fedora-linux-core/ --mirror-name=fedora>
 
-Usage: B<cobbler import --path=<path>>
+Once imported, run a "cobbler list" to see what you've added.
 
 =head2 TWEAKING
 
diff --git a/cobbler/action_import.py b/cobbler/action_import.py
index e50ffd8..3a28e6c 100644
--- a/cobbler/action_import.py
+++ b/cobbler/action_import.py
@@ -21,13 +21,26 @@ import traceback
 
 import api
 
+# MATCH_LIST uses path segments of mirror URLs to assign kickstart
+# files.  It's not all that intelligent.
+
+# FIXME: add common FC, RHEL, and Centos path segments
+MATCH_LIST = ( 
+   ( "FC-5/"  , "/etc/cobbler/kickstart_fc56.ks" ),
+   ( "FC-6/"  , "/etc/cobbler/kickstart_fc56.ks" ),
+   ( "RHEL-4/", "/etc/cobbler/kickstart_fc56.ks" ),
+   ( "6/"     , "/etc/cobbler/kickstart_fc56.ks" ),
+   ( "5/"     , "/etc/cobbler/kickstart_fc56.ks" )
+)
+
 class Importer:
 
-   def __init__(self,config,path,mirror):
+   def __init__(self,config,path,mirror,mirror_name):
        # FIXME: consider a basename for the import
        self.config = config
        self.path = path
        self.mirror = mirror
+       self.mirror_name = mirror_name
        if path is None:
            raise cexceptions.CobblerException("import_failed","no path specified")
        self.distros = config.distros()
@@ -39,12 +52,93 @@ class Importer:
            raise cexceptions.CobblerException("import_failed","no path specified")
        if not os.path.isdir(self.path):
            raise cexceptions.CobblerException("import_failed","bad path")
+       if self.mirror is not None:
+           if not self.mirror.startswith("rsync://"):
+               raise cexceptions.CobblerException("import_failed","expecting rsync:// url")
+           if self.mirror_name is None:
+               raise cexceptions.CobblerException("import_failed","must specify --mirror-name")
+           # FIXME:  --delete is a little harsh and should be a command
+           # line option and not the default (?)
+           print "This will take a while..."
+           self.path = "/var/www/cobbler/localmirror/%s" % self.mirror_name
+           cmd = "rsync -az %s /var/www/cobbler/localmirror/%s --progress" % self.mirror_name
+           sub_process.call(cmd,shell=True)
+           update_file = os.path.open(os.path.join(self.path,"update.sh"))
+           update.file.write("#!/bin/sh")
+           update_file.write(cmd)
+           update_file.close()
        if self.path is not None:
            arg = None
            os.path.walk(self.path, self.walker, arg)
+           self.scrub_orphans()
+           self.guess_kickstarts()
            return True
        return False
 
+   def scrub_orphans(self):
+       """
+       This has nothing to do with parentless children that need baths.
+       first: remove any distros with missing kernel or initrd files
+       second: remove any profiles that depend on distros that don't exist
+       systems will be left as orphans as the MAC info may be useful
+       to the sysadmin and may not be recorded elsewhere.  We will report
+       the orphaned systems.  
+       FIXME: this should also be a seperate API command!
+       """
+       print "*** SCRUBBING ORPHANS"
+       # FIXME
+       for distro in self.distros:
+           if not os.path.exists(distro.kernel):
+               print "*** ORPHANED DISTRO: %s" % distro.name
+               self.distros.remove(distro.name)
+               continue
+           if not os.path.exists(distro.initrd):
+               print "*** ORPHANED DISTRO: %s" % distro.name
+               self.distros.remove(distro.initrd)
+               continue
+           print "*** KEEPING: %s" % distro.name
+       for profile in self.profiles:
+           if not self.distros.find(profile.distro):
+               print "*** ORPHANED PROFILE: %s" % profile.name
+               self.profiles.remove(profile.name)
+               continue
+           print "*** KEEPING: %s" % profile.name
+       for system in self.systems:
+           if not self.profiles.find(system.profile):
+               print "*** ORPHANED SYSTEM (NOT DELETED): %s" % system.name
+               continue
+
+   def guess_kickstarts(self):
+       """
+       For all of the profiles in the config w/o a kickstart, look
+       at the kernel path, from that, see if we can guess the distro,
+       and if we can, assign a kickstart if one is available for it.
+       """
+       print "*** GUESSING KICKSTARTS"
+       for profile in self.profiles:
+           distro = self.distros.find(profile.name)
+           kpath = distro.kernel
+           if not kpath.startswith("/var/www/cobbler"):
+               print "*** CAN'T GUESS (kpath): %s" % kpath
+               continue 
+           for entry in MATCH_LIST:
+               (part, kickstart) = entry
+               if kpath.find(part) != -1:
+                   print "*** CONSIDERING: %s" % kickstart
+                   if os.path.exists(kickstart):
+                       print "*** ASSIGNING kickstart: %s" % kickstart
+                       profile.set_kickstart(kickstart)      
+                       # from the kernel path, the tree path is always two up.
+                       # FIXME: that's probably not always true
+                       base = os.path.basename(kpath)
+                       base = os.path.basename(base)
+                       base = base.replace("/var/www/cobbler/","")
+                       print "%s" % base
+                       tree = "tree=http://%s/localmirror/%s/" % (self.settings,server, self.mirror_name, base)
+                       print "%s" % tree
+                       print "*** ASSIGNING KS META = %s" % tree
+                       profile.set_ksmeta(tree)
+
    def walker(self,arg,dirname,fnames):
        # FIXME: requires getting an arch out of the path
        # FIXME: requires making sure the path contains "pxe" somewhere
@@ -85,26 +179,23 @@ class Importer:
 
    def get_proposed_name(self,dirname):
        # FIXME: how can this name be nicer?
-       str = "_".join(dirname.split("/"))
-       if str.startswith("_"):
-          return str[1:]
-       return str
+       temp  = "_".join(dirname.split("/"))
+       if temp.startswith("_"):
+          temp = temp[1:]
+       return temp
 
    def get_pxe_arch(self,dirname):
-       tokens = os.path.split(dirname)
-       tokens = [x.lower() for x in tokens]
-       for t in tokens:
-          if t == "i386" or t == "386" or t == "x86":
-              return "x86"
-          if t == "x86_64":
-              return "x86_64"
-          if t == "ia64":
-              return "ia64"
+       t = dirname
+       if t.find("x86_64") != -1:
+          return "x86_64"
+       if t.find("ia64") != -1:
+          return "ia64"
+       if t.find("i386") != -1 or t.find("386") != -1 or t.find("x86") != -1:
+          return "x86"
        return "x86"
 
    def is_pxe_or_xen_dir(self,dirname):
-       tokens = os.path.split(dirname)
-       for x in tokens:
-           if x.lower() == "pxe" or x.lower() == "pxeboot" or x.lower() == "xen":
-               return True
+       if dirname.find("pxe") != -1 or dirname.find("xen") != -1:
+           return True
        return False
+
diff --git a/cobbler/api.py b/cobbler/api.py
index 18ed9e0..fae5d64 100644
--- a/cobbler/api.py
+++ b/cobbler/api.py
@@ -116,13 +116,13 @@ class BootAPI:
         enchant = action_enchant.Enchant(self._config,sysname,password)
         return enchant.run()
 
-    def import_tree(self,tree_path,mirror_url):
+    def import_tree(self,tree_path,mirror_url,mirror_name):
         """
         Automatically import a directory tree full of distribution files.
         Imports either a tree (path) or mirror (ftp/http).
         Mirror support really doesn't exist yet... TBA.
         """
-        importer = action_import.Importer(self._config, tree_path, mirror_url)
+        importer = action_import.Importer(self._config, tree_path, mirror_url, mirror_name)
         return importer.run()
 
     def serialize(self):
diff --git a/cobbler/cobbler.py b/cobbler/cobbler.py
index e7a512c..d31924c 100755
--- a/cobbler/cobbler.py
+++ b/cobbler/cobbler.py
@@ -152,6 +152,9 @@ class BootCLI:
         """
         self.temp_path = None
         self.temp_mirror = None
+        self.temp_mirror_name = None
+        def set_mirror_name(a):
+            self.temp_mirror_name = a
         def set_mirror(a):
             self.temp_mirror = a
         def set_path(a):
@@ -160,7 +163,9 @@ class BootCLI:
                 return True
             return False
         def go_import():
-            return self.api.import_tree(self.temp_path,self.temp_mirror)
+            return self.api.import_tree(self.temp_path,
+                self.temp_mirror,
+                self.temp_mirror_name)
         commands = {
             '--path'    : lambda(a): set_path(a),
             '--mirror'  : lambda(a): set_mirror(a)
diff --git a/kickstart_fc5.ks b/kickstart_fc5.ks
new file mode 100644
index 0000000..ca060e7
--- /dev/null
+++ b/kickstart_fc5.ks
@@ -0,0 +1,38 @@
+#platform=x86, AMD64, or Intel EM64T
+# System authorization information
+auth  --useshadow  --enablemd5 
+# System bootloader configuration
+bootloader --location=mbr
+# Partition clearing information
+clearpart --all --initlabel 
+# Use text mode install
+text
+# Firewall configuration
+firewall --enabled   
+# Run the Setup Agent on first boot
+firstboot --disable
+# System keyboard
+keyboard us
+# System language
+lang en_US
+# Use network installation
+url --url=http://127.0.0.1/cobbler/foo
+# Network information
+network --bootproto=dhcp --device=eth0 --onboot=on
+# Reboot after installation
+reboot
+
+#Root password
+rootpw --iscrypted $1$mF86/UHC$WvcIcX2t6crBz2onWxyac.
+# SELinux configuration
+selinux --disabled
+# Do not configure the X Window System
+skipx
+# System timezone
+timezone  America/New_York
+# Install OS instead of upgrade
+install
+# Clear the Master Boot Record
+zerombr
+
+%packages
diff --git a/setup.py b/setup.py
index a7f7785..558e7ac 100644
--- a/setup.py
+++ b/setup.py
@@ -30,6 +30,7 @@ if __name__ == "__main__":
                                 # (docspath, ['README']),
                                 (wwwpath, []),
                                 (cobpath, ['elilo-3.6-ia64.efi']),
+                                (etcpath, ['kickstart_fc5.ks']),
 				(etcpath, ['dhcp.template']),
                                 (manpath, ['cobbler.1.gz'])
                              ],