Preliminary support for http:// and ftp:// mirrors via yum's reposync.

2 files changed, 113 insertions, 48 deletions

diff --git a/cobbler.pod b/cobbler.pod
index ebca9ef..37ca704 100644
--- a/cobbler.pod
+++ b/cobbler.pod
@@ -137,6 +137,11 @@ multiple images on a machine will cause increasing numbers to be appended to thi
 
 (optional) (Virt-only) how many megabytes of RAM to consume.  The default is 512 MB.
 
+=item repos
+
+(optional) a space delimited list of all the repos (created with "cobbler repo add" and "cobbler repo sync") that this profile
+can make use of during kickstart installation.  For example, an example might be --repos="fc6i386updates fc6i386extras".
+
 =back
 
 =head2 ADDING A SYSTEM
@@ -178,17 +183,21 @@ NOTE: Due to a limitation in elilo (IA64 bootloader), this parameter must ALSO b
 
 =head2 ADDING A REPOSITORY TO MIRROR
 
-Repository mirroring is one of the more complex cobbler features, though if you want to mirror
-a yum repository and integrate it with your provisioning, cobbler can help simplify the required
-knowledge a good bit.  If you're just provisioning your home system, ignore this part.
+Repository mirroring allows cobbler to mirror not only install trees ("cobbler import" does this for you) but
+also optional packages, 3rd party content, and even updates.   Mirroring all of this content locally
+on your network will result in faster, more up-to-date installations and faster updates.  If you
+are only provisioning a home setup, this will probably be overkill, though it can be very useful
+for larger setups (labs, datacenters, etc).
 
-B<cobbler repo add --mirror=url --mirror-name=string [--local-filename=string]>
+B<cobbler repo add --mirror=url --name=string [--local-filename=string]>
 
 =over
 
 =item mirror
 
-The addresss of the mirror.  This needs to be either an rsync:// url or an ssh location usable with rsync.
+The addresss of the yum mirror.  This can be an rsync:// URL, an ssh location, or
+a http:// or ftp:// mirror location.
+
 The mirror address should specify an exact repository to mirror -- just one architecture
 and just one distribution.  If you have a seperate repo to mirror for a different arch, add that
 repo seperately.  
@@ -196,30 +205,36 @@ repo seperately.
 Here's an example of what looks like a good URL:
 
 rsync://yourmirror.example.com/fedora-linux-core/updates/6/i386 (for rsync protocol)
+http://mirrors.kernel.org/fedora/extras/6/i386/ (for http://)
 user@yourmirror.example.com/fedora-linux-core/updates/6/i386  (for SSH)
 
 To put it more simply, if the content you are mirroring doesn't contain rpm's
 at the top level of the URL, this is bad, and you need to specify a different value.  Using the
 wrong mirror value here will rsync over too much data, and also the provisioning integration code
-simply won't work.  You can't pass in a Fedora mirror, or even a FC6 mirror address.  Be specific.
+simply won't work.  Be specific.
 
-=item mirror-name
+=item name
 
 This name is used as the save location for the mirror.  If the mirror represented, say, Fedora Core
-6 i386 updates, a good name would be "fc6i386updates".  Be explicit.  
+6 i386 updates, a good name would be "fc6i386updates".  Again, be specific.
 
-This name corresponds with values given to the --repo parameter of "cobbler profile add".  If a profile
-has a --repo value that matches the name here, that repo can be automatically set up during provisioning.
+This name corresponds with values given to the --repos parameter of "cobbler profile add".  If a profile
+has a --repos value that matches the name here, that repo can be automatically set up during provisioning.
 This means that, if supported by Anaconda, the repo can be used during kickstart install -- and -- either way,
 it can be automatically configured on the clients.
 
+See the documentation on "cobbler profile add" for more information.
+
 =item local-filename
 
 Local filename specifies, for kickstarts containing the template parameter "yum_config_stanza",
-what files to populate on provisioned clients in /etc/yum.repos.d.  In other words, if this value
-is "foo", the repo would be added on provisioned clients as "/etc/yum.repos.d/foo.repo".  If you don't
-want clients to have this repo installed, don't add a name for the repo, and provisioned machines
-will not configure yum to know about this repo -- you can still do it manually if you choose.
+what files to populate on provisioned clients in /etc/yum.repos.d.  
+
+In other words, if this value is "foo", the repo would be installed on provisioned clients as "/etc/yum.repos.d/foo.repo".  
+
+If you don't want clients to have this repo installed, don't add a name for the repo, and provisioned machines
+will not configure yum to know about this repo -- you can still do it manually if you choose.  The repository will
+still be used for installation, it just won't get installed automatically in /etc/yum.repos.d on the client.
 
 See /etc/cobbler/kickstart_fc6.ks for an example of how to employ this within a kickstart template.
 
@@ -240,7 +255,6 @@ B<cobbler list [--settings] [--profiles] [--distros] [--systems] [--repos]>
 
 Alternatively, you could look at the configuration files in /var/lib/cobbler to see the same information.  
 
-
 =head2 DELETING CONFIGURATION ENTRIES
 
 If you want to remove a specific object, use the remove command with the name that was used to add it.
@@ -329,16 +343,13 @@ B<cobbler check>
 
 # set up your cobbler distros here.
 
-B<cobbler repo add --mirror=rsync://repos.example.com/foo/i386/ --name=magicfooi386>
+B<cobbler repo add --mirror=http://mirrors.kernel.org/fedora/core/updates/6/i386/ --name=fc6i386updates>
 
-B<cobbler repo add --mirror=root@192.168.1.5:/foo/i386/ --name=magicbari386>
+B<cobbler repo add --mirror=http://mirrors.kernel.org/fedora/extras/6/i386/ --name=fc6i386extras>
 
 B<cobbler reposync>
 
-B<cobbler profile add --name=p1 --distro=existing_distro_name --kickstart=/etc/cobbler/kickstart_fc6.ks --repos="magicfooi386 magicbari386">
-
-See the expanded description towards the bottom of this manpage for further information about
-repo management.
+B<cobbler profile add --name=p1 --distro=existing_distro_name --kickstart=/etc/cobbler/kickstart_fc6.ks --repos="fc6i386updates fc6i386extras">
 
 =over
 
@@ -498,11 +509,14 @@ B<cobbler reposync>
 Cobbler reposync is the command to use to update repos as configured with "cobbler repo add".  Mirroring
 can take a long time, and usage of cobbler reposync prior to cobbler sync is needed to ensure
 provisioned systems have the files they need to actually use the mirrored repositories.  If you just
-add repos and never run reposync, the repos will never be mirrored.  This is probably a command
+add repos and never run "cobbler reposync", the repos will never be mirrored.  This is probably a command
 you would want to put on a crontab, though the frequency of that crontab and where the output
 goes is left up to the systems administrator.
 
-Repositories that do not need to be updated can be modifed by tweaking the values in /var/lib/cobbler/repos.
+For those familiar with yum's reposync, cobbler's reposync is a wrapper around the yum command.  Please
+use "cobbler reposync" to update cobbler mirrors, as reposync does not perform all required steps.  Also
+cobbler adds support for rsync and SSH locations, where as yum's reposync only supports what yum supports
+(http/ftp).
 
 =head2 KICKSTART TRACKING
 
diff --git a/cobbler/action_reposync.py b/cobbler/action_reposync.py
index 29bbfe2..3c36bb2 100644
--- a/cobbler/action_reposync.py
+++ b/cobbler/action_reposync.py
@@ -2,7 +2,7 @@
 Builds out and synchronizes yum repo mirrors.
 Initial support for rsync, perhaps reposync coming later.
 
-Copyright 2006, Red Hat, Inc
+Copyright 2006-2007, Red Hat, Inc
 Michael DeHaan <mdehaan@redhat.com>
 
 This software may be freely redistributed under the terms of the GNU
@@ -55,18 +55,53 @@ class RepoSync:
             repo_path = os.path.join(self.settings.webdir, "repo_mirror", repo.name)
             mirror = repo.mirror
             if not os.path.isdir(repo_path):
-                try:
-                    os.makedirs(repo_path)
-                except OSError, oe:
-                    if not oe.errno == 17: # already exists, constant for this?
-                        raise cexceptions.CobblerException("no_create", repo_path)
-            self.do_rsync_repo(repo)
+                os.makedirs(repo_path)
+            # if path contains http:// or ftp://, use with yum's reposync.
+            # else do rsync
+            if mirror.lower().find("http://") != -1 or mirror.lower().find("ftp://") != -1:
+                self.do_reposync(repo)
+            else:
+                self.do_rsync(repo)
 
         return True
-   
-    def do_rsync_repo(self,repo):
+  
+    def do_reposync(self,repo):
+
+        """
+        Handle copying of http:// and ftp:// repos.
+        FIXME: support for mirrorlist?
+        """
+
+        if not repo.keep_updated:
+            print "- %s is set to not be updated" % repo.name
+            return True
+
+        # create yum config file for use by reposync
+        store_path = os.path.join(self.settings.webdir, "repo_mirror")
+        dest_path = os.path.join(store_path, repo.name)
+        temp_path = os.path.join(store_path, ".origin")
+        if not os.path.isdir(temp_path):
+            os.makedirs(temp_path)
+        temp_file = self.create_local_file(repo, temp_path, output=False)
+ 
+        cmd = "/usr/bin/reposync --config=%s --repoid=%s --tempcache --download_path=%s" % (temp_file, repo.name, store_path)
+        print "- %s" % cmd
+        rc = sub_process.call(cmd, shell=True)
+        if rc !=0:
+            raise cexceptions.CobblerException("cobbler reposync failed")
+        arg = None
+        os.path.walk(dest_path, self.createrepo_walker, arg)
+
+        self.create_local_file(repo, dest_path)
+ 
+    def do_rsync(self,repo):
+
+        """
+        Handle copying of rsync:// and rsync-over-ssh repos.
+        """
+
         if not repo.keep_updated:
-            print "- %s is set to not be updated"
+            print "- %s is set to not be updated" % repo.name
             return True
         dest_path = os.path.join(self.settings.webdir, "repo_mirror", repo.name)
         spacer = ""
@@ -75,30 +110,46 @@ class RepoSync:
         if not repo.mirror.endswith("/"):
             repo.mirror = "%s/" % repo.mirror
         cmd = "rsync -av %s --delete --delete-excluded --exclude-from=/etc/cobbler/rsync.exclude %s %s" % (spacer, repo.mirror, dest_path)       
-        print "executing: %s" % cmd
+        print "- %s" % cmd
         rc = sub_process.call(cmd, shell=True)
+        if rc !=0:
+            raise cexceptions.CobblerException("cobbler reposync failed")
         arg = {}
         print "- walking: %s" % dest_path
         os.path.walk(dest_path, self.createrepo_walker, arg)
-        if repo.local_filename is not None and repo.local_filename != "":
-            # this is a rather primative configuration in terms of yum options, but allows
-            # for repos that were added with a value for --local-filename to provision a system
-            # using a kickstart that will write the yum config for that repo (named whatever
-            # was used for local_filename) in /etc/yum.repos.d ... see code in action_sync.py
-            # that relies on this and for more info about how this is added to %post kickstart
-            # templating.
-            config_file = open(os.path.join(dest_path,"config.repo"),"w+")
-            config_file.write("[%s]\n" % repo.local_filename)
+        self.create_local_file(repo, dest_path)
+
+    def create_local_file(self, repo, dest_path, output=True):
+        """
+        Two uses:
+        (A) Create local files that can be used with yum on provisioned clients to make use of thisi mirror.
+        (B) Create a temporary file for yum to feed into reposync
+        """
+
+        if output:
+            fname = os.path.join(dest_path,"config.repo")
+        else:
+            fname = os.path.join(dest_path, "%s.repo" % repo.name)
+        print "- creating: %s" % fname
+        config_file = open(fname, "w+")
+        config_file.write("[%s]\n" % repo.name)
+        config_file.write("name=%s\n" % repo.name)
+        if output:
             config_file.write("baseurl=http://%s/cobbler/repo_mirror/%s\n" % (self.settings.server, repo.name))
-            config_file.write("enabled=1\n")
-            config_file.write("gpgcheck=0\n")
-            config_file.close()
-        
+        else:
+            config_file.write("baseurl=%s\n" % repo.mirror)
+        config_file.write("enabled=1\n")
+        config_file.write("gpgcheck=0\n")
+        config_file.close()
+        return fname 
 
     def createrepo_walker(self, arg, dirname, fname):
+        """
+        Used to run createrepo on a copied mirror.
+        """
         target_dir = os.path.dirname(dirname).split("/")[-1]
         print "- scanning: %s" % target_dir
-        if target_dir.lower() in [ "i386", "x86_64", "ia64" ]:
+        if target_dir.lower() in [ "i386", "x86_64", "ia64" ] or (arg is None):
             try:
                 cmd = "createrepo %s" % dirname
                 print cmd