Update README

Signed-off-by: Martin Schwenke <martin@meltin.net>

1 files changed, 92 insertions, 442 deletions

diff --git a/README b/README
index 0e4a5b7..f5093ad 100644
--- a/README
+++ b/README
@@ -1,38 +1,42 @@
 INTRODUCTION
 ============
 
-Autocluster is set of scripts for building virtual clusters to test
-clustered Samba.  It uses Linux's libvirt and KVM virtualisation
-engine.
-
-Autocluster is a collection of scripts, template and configuration
-files that allow you to create a cluster of virtual nodes very
-quickly.  You can create a cluster from scratch in less than 30
-minutes.  Once you have a base image you can then recreate a cluster
-or create new virtual clusters in minutes.
-
-Autocluster has recently been tested to create virtual clusters of
-RHEL 6/7 nodes.  Older versions were tested with RHEL 5 and some
-versions of CentOS.
+Autocluster is a script for building virtual clusters to test
+clustered Samba.
 
+It uses Vagrant (with the libvirt plugin) and Ansible to build and
+configure a cluster.
 
 CONTENTS
 ========
 
+* SUPPORTED PLATFORMS
+
 * INSTALLING AUTOCLUSTER
 
 * HOST MACHINE SETUP
 
 * CREATING A CLUSTER
 
-* BOOTING A CLUSTER
+* DESTROYING A CLUSTER
+
+* DEVELOPMENT HINTS
 
-* POST-CREATION SETUP
 
-* CONFIGURATION
+SUPPORTED_PLATFORMS
+===================
 
-* DEVELOPMENT HINTS
+Tested host platforms:
+
+* CentOS 7
+
+Tested guest platforms:
 
+* CentOS 7
+
+Tested cluster filesystems:
+
+* GPFS
 
 INSTALLING AUTOCLUSTER
 ======================
@@ -42,481 +46,127 @@ autocluster. To download autocluster do this:
 
   git clone git://git.samba.org/autocluster.git
 
-Or to update it, run "git pull" in the autocluster directory
-
 You probably want to add the directory where autocluster is installed
 to your PATH, otherwise things may quickly become tedious.
 
+Packages can also be built and installed.
+
 
 HOST MACHINE SETUP
 ==================
 
-This section explains how to setup a host machine to run virtual
-clusters generated by autocluster.
-
-
- 1) Install and configure required software.
-
- a) Install kvm, libvirt and expect.
-
-    Autocluster creates virtual machines that use libvirt to run under
-    KVM.  This means that you will need to install both KVM and
-    libvirt on your host machine.  Expect is used by the waitfor()
-    function and should be available for installation from your
-    distribution.
-
-    For various distros:
-
-    * RHEL/CentOS
-
-      Autocluster should work with the standard RHEL qemu-kvm and
-      libvirt packages.  It will try to find the qemu-kvm binary.  If
-      you've done something unusual then you'll need to set the KVM
-      configuration variable.
-
-      For RHEL5/CentOS5, useful packages for both kvm and libvirt used
-      to be found here:
-
-        http://www.lfarkas.org/linux/packages/centos/5/x86_64/
-
-      However, since recent versions of RHEL5 ship with KVM, 3rd party
-      KVM RPMs for RHEL5 are now scarce.
-
-      RHEL5.4's KVM also has problems when autocluster uses virtio
-      shared disks, since multipath doesn't notice virtio disks.  This
-      is fixed in RHEL5.6 and in a recent RHEL5.5 update - you should
-      be able to use the settings recommended above for RHEL6.
-
-      If you're still running RHEL5.4, you have lots of time, you have
-      lots of disk space, and you like complexity, then see the
-      section below on "Raw IDE system disks".  :-)
-
-    * Fedora
-
-      Useful packages ship with Fedora Core 10 (Cambridge) and later.
-      Some of the above notes on RHEL might apply to Fedora's KVM.
-
-    * Ubuntu
-
-      Useful packages ship with Ubuntu 8.10 (Intrepid Ibex) and later.
-      In recent Ubuntu versions (e.g. 10.10 Maverick Meerkat) the KVM
-      package is called "qemu-kvm".  Older versions have a package
-      called "kvm".
-
-    For other distributions you'll have to backport distro sources or
-    compile from upstream source as described below.
-
-    * For KVM see the "Downloads" and "Code" sections at:
-
-        http://www.linux-kvm.org/
-
-    * For libvirt see:
-
-        http://libvirt.org/
-
- b) Install guestfish or qemu-nbd and nbd-client.
-
-    Autocluster needs a method of updating files in the disk image for
-    each node.
-
-    Recent Linux distributions, including RHEL since 6.0, contain
-    guestfish.  Guestfish (see http://libguestfs.org/ - there are
-    binary packages for several distros here) is a CLI for
-    manipulating KVM/QEMU disk images.  Autocluster supports
-    guestfish, so if guestfish is available then you should use it.
-    It should be more reliable than NBD.
-
-    Autocluster attempts to use the best available method (guestmount
-    -> guestfish -> loopback) for accessing disk image.  If it chooses
-    a suboptimal method (e.g. nodes created with guestmount sometimes
-    won't boot), you can force the method:
-
-      SYSTEM_DISK_ACCESS_METHOD=guestfish
-
-    If you can't use guestfish then you'll have to use NBD.  For this
-    you will need the qemu-nbd and nbd-client programs, which
-    autocluster uses to loopback-nbd-mount the disk images when
-    configuring each node.
-
-    NBD for various distros:
-
-    * RHEL/CentOS
-
-      qemu-nbd is only available in the old packages from lfarkas.org.
-      Recompiling the RHEL5 kvm package to support NBD is quite
-      straightforward.  RHEL6 doesn't have an NBD kernel module, so is
-      harder to retrofit for NBD support - use guestfish instead.
-
-      Unless you can find an RPM for nbd-client then you need to
-      download source from:
- 
-        http://sourceforge.net/projects/nbd/
-
-      and build it.
-
-    * Fedora Core
-
-      qemu-nbd is in the qemu-kvm or kvm package.
-
-      nbd-client is in the nbd package.
-
-    * Ubuntu
-
-      qemu-nbd is in the qemu-kvm or kvm package.  In older releases
-      it is called kvm-nbd, so you need to set the QEMU_NBD
-      configuration variable.
-
-      nbd-client is in the nbd-client package.
-
-    * As mentioned above, nbd can be found at:
-
-        http://sourceforge.net/projects/nbd/
-
- c) Environment and libvirt virtual networks
-
-    You will need to add the autocluster directory to your PATH.
-
-    You will need to configure the right libvirt networking setup. To
-    do this, run:
-
-      host_setup/setup_networks.sh [ <myconfig> ]
-
-    If you're using a network setup different to the default then pass
-    your autocluster configuration filename, which should set the
-    NETWORKS variable.  If you're using a variety of networks for
-    different clusters then you can probably run this script multiple
-    times.
+1. Install Ansible
 
-    You might also need to set:
+2. Run: autocluster host <platform> setup
 
-      VIRSH_DEFAULT_CONNECT_URI=qemu:///system
+   Currently the only supported <platform> is "centos7"
 
-    in your environment so that virsh does KVM/QEMU things by default.
+   This will
 
- 2) Configure a local web/install server to provide required YUM
-    repositories
+   * Install and configure several packages, including Vagrant
 
-    If your install server is far away then you may need a caching web
-    proxy on your local network.
+   * Assume you want to serve repositories to guests from /home/mediasets/.
 
-    If you don't have one, then you can install a squid proxy on your
-    host amd set:
+   * Create a libvirt storage pool at /virtual/autocluster/ for VM
+     images/files.
 
-      WEBPROXY="http://10.0.0.1:3128/"
+   * Create an SSH key for autocluster
 
-    See host_setup/etc/squid/squid.conf for a sample config suitable
-    for a virtual cluster. Make sure it caches large objects and has
-    plenty of space. This will be needed to make downloading all the
-    RPMs to each client sane
-
-    To test your squid setup, run a command like this:
-
-      http_proxy=http://10.0.0.1:3128/ wget <some-url>
-
-    Check your firewall setup.  If you have problems accessing the
-    proxy from your nodes (including from kickstart postinstall) then
-    check it again!  Some distributions install nice "convenient"
-    firewalls by default that might block access to the squid port
-    from the nodes.  On a current version of Fedora Core you may be
-    able to run system-config-firewall-tui to reconfigure the
-    firewall.
-
- 3) Setup a DNS server on your host. See host_setup/etc/bind/ for a
-    sample config that is suitable. It needs to redirect DNS queries
-    for your virtual domain to your windows domain controller.
-
- 4) Download a RHEL (or CentOS) install ISO.
+   For speed, you may wish to mirror the guest distribution somewhere
+   under /home/mediasets/ or on another nearby machine.
 
+Depending on how your host machine is setup, you may need to run
+autocluster commands as root.
 
 CREATING A CLUSTER
 ==================
 
-A cluster comprises a single base disk image, a copy-on-write disk
-image for each node and some XML files that tell libvirt about each
-node's virtual hardware configuration.  The copy-on-write disk images
-save a lot of disk space on the host machine because they each use the
-base disk image - without them the disk image for each cluster node
-would need to contain the entire RHEL install.
-
-The cluster creation process can be broken down into several main
-steps:
-
- 1) Create a base disk image.
-
- 2) Create per-node disk images and corresponding XML files.
-
- 3) Update /etc/hosts to include cluster nodes.
-
- 4) Boot virtual machines for the nodes.
-
- 5) Post-boot configuration.
-
-However, before you do this you will need to create a configuration
-file.  See the "CONFIGURATION" section below for more details.
-
-Here are more details on the "create cluster" process.  Note that
-unless you have done something extra special then you'll need to run
-all of this as root.
-
- 1) Create the base disk image using:
-
-      ./autocluster base create
-
-    The first thing this step does is to check that it can connect to
-    the YUM server.  If this fails make sure that there are no
-    firewalls blocking your access to the server.
-
-    The install will take about 10 to 15 minutes and you will see the
-    packages installing in your terminal
-
-    The installation process uses kickstart.  The choice of
-    postinstall script is set using the POSTINSTALL_TEMPLATE variable.
-    This can be used to install packages that will be common to all
-    nodes into the base image.  This save time later when you're
-    setting up the cluster nodes.  However, current usage (given that
-    we test many versions of CTDB) is to default POSTINSTALL_TEMPLATE
-    to "" and install packages post-boot.  This seems to be a
-    reasonable compromise between flexibility (the base image can be,
-    for example, a pristine RHEL7.0-base.qcow2, CTDB/Samba packages
-    are selected post-base creation) and speed of cluster creation.
-
-    When that has finished you should mark that base image immutable
-    like this:
-
-      chattr +i /virtual/ac-base.img
-
-    That will ensure it won't change. This is a precaution as the
-    image will be used as a basis file for the per-node images, and if
-    it changes your cluster will become corrupt
-
- 2-5)
-    Now run "autocluster cluster build", specifying a configuration
-    file. For example:
-
-      autocluster -c m1.autocluster cluster build
-
-    This will create and install the XML node descriptions and the
-    disk images for your cluster nodes, and any other nodes you have
-    configured.  Each disk image is initially created as an "empty"
-    copy-on-write image, which is linked to the base image.  Those
-    images are then attached to using guestfish or
-    loopback-nbd-mounted, and populated with system configuration
-    files and other potentially useful things (such as scripts).
-    /etc/hosts is updated, the cluster is booted and post-boot
-    setup is done.
-
-    Instead of doing all of the steps 2-5 using 1 command you call do:
-
-    2) autocluster -c m1.autocluster cluster create
- 
-    3) autocluster -c m1.autocluster cluster update_hosts
-
-    4) autocluster -c m1.autocluster cluster boot
-
-    5) autocluster -c m1.autocluster cluster setup
-
-BOOTING/DESTROY A CLUSTER
-=========================
+Configuration file
+------------------
 
-Autocluster provides a command called "vircmd", which is a thin
-wrapper around libvirt's virsh command.  vircmd takes a cluster name
-instead of a node/domain name and runs the requested command on all
-nodes in the cluster.
+The configuration file is a YAML file.  If your cluster is to be
+called "foo" then the configuration file must be "foo.yml" in the
+current directory.
 
-    The most useful vircmd commands are:
- 
-      start    : boot a cluster
-      shutdown : graceful shutdown of a cluster
-      destroy  : power off a cluster immediately
+To see what options to set, try this:
 
-    You can watch boot progress like this:
+    # autocluster cluster foo defaults
 
-       tail -f /var/log/kvm/serial.c1*
+This will show default the default configuration.  This is the only
+cluster command that doesn't need a cluster configuration.
 
-    All the nodes have serial consoles, making it easier to capture
-    kernel panic messages and watch the nodes via ssh
+It may also be worth looking at the file defaults.yml, which
+includes some useful comments.
 
+Add updated settings foo.yml.  Try to set the minimum number of
+options to keep the configuration file small.  See example.yml.
 
-POST-BOOT SETUP
-===============
+Most items are fairly obvious.  However, here are some details:
 
-Autocluster copies some scripts to cluster nodes to enable post-boot
-configuration.  These are used to configure specialised subsystems
-like GPFS or Samba, and are installed in /root/scripts/ on each node.
-The main entry point is cluster_setup.sh, which invokes specialised
-scripts depending on the cluster filesystem type or the node type.
-cluster_setup.sh is invoked by the cluster_setup() function in
-autocluster.
+* networks
 
-See cluster_setup() if you want to do things manually or if you want
-to add support for other node types and/or cluster filesystems.
+  Default: 10.0.0.0/24 10.0.1.0/24 10.0.2.0/24
 
-There are also some older scripts that haven't been used for a while
-and have probably bit-rotted, such as setup_tsm_client.sh and
-setup_tsm_server.sh.  However, they are still provided as examples.
+  There should be at least 2 networks.  The first network is a
+  private network, while the others can be used for CTDB public IP
+  addresses.
 
-CONFIGURATION
-=============
+* firstip
 
-Basics
-======
+  Default: 20
 
-Autocluster uses configuration files containing Unix shell style
-variables.  For example,
+  This is the final octet of the first IP address used on each network.
 
-  FIRSTIP=30
+* node_list
 
-indicates that the last octet of the first IP address in the cluster
-will be 30.  If an option contains multiple words then they will be
-separated by underscores ('_'), as in:
+  Default: [nas, nas, nas, nas, test]
 
-  ISO_DIR=/data/ISOs
+  Each node is offset from firstip by its position in the list.
 
-All options have an equivalent command-line option, such
-as:
+  The above default will result in 5 nodes.
 
-  --firstip=30
+  - The first 4 will be Clustered Samba NAS nodes (running CTDB,
+    Samba, NFS) with addresses on the first network from 10.0.0.20
+    to 10.0.0.23 (with similar static addresses on the other
+    networks).
 
-Command-line options are lowercase.  Words are separated by dashes
-('-'), as in:
+  - The 5th node will be a minimally installed/configured test node
+    that can be used as a CTDB test client, with address 10.0.0.24.
 
-  --iso-dir=/data/ISOs
+  Valid node types are:
 
-Normally you would use a configuration file with variables so that you
-can repeat steps easily.  The command-line equivalents are useful for
-trying things out without resorting to an editor.  You can specify a
-configuration file to use on the autocluster command-line using the -c
-option.  For example:
+  nas:     Clustered Samba node with cluster filesystem, smbd, nfsd
+  ad:      Samba Active Directory Domain Controller node
+  base:    Base operaing system node
+  build:   Build node for CTDB packages
+  cbuild:  Build node for Samba, with cluster filesystem installed
+  storage: Cluster filesystem node that doesn't directly provide NAS services
+  test:    CTDB test node, with CTDB packages installed
 
-  autocluster -c config-foo create base
+Cluster creation
+----------------
 
-If you don't provide a configuration variable then autocluster will
-look for a file called "config" in the current directory.
+In theory this is easy:
 
-You can also use environment variables to override the default values
-of configuration variables.  However, both command-line options and
-configuration file entries will override environment variables.
+    # autocluster cluster foo build
 
-Potentially useful information:
+This runs several internal steps:
 
-* Use "autocluster --help" to list all available command-line options
-  - all the items listed under "configuration options:" are the
-  equivalents of the settings for config files.  This output also
-  shows descriptions of the options.
+1. `destroy` - Destroy any existing cluster of the same name
+2. `generate` - Generate metadata (for Vagrant, Ansible, SSH) from the
+   configuration
+3. `create` - Create the cluster nodes (using Vagrant)
+4. `ssh_config` - Configure SSH to allow direct access to nodes as root
+5. `setup` - Setup each node according to its type (using Ansible)
 
-* You can use the --dump option to check the current value of
-  configuration variables.  This is most useful when used in
-  combination with grep:
-
-    autocluster --dump | grep ISO_DIR
-
-  In the past we recommended using --dump to create initial
-  configuration file.  Don't do this - it is a bad idea!  There are a
-  lot of options and you'll create a huge file that you don't
-  understand and can't debug!
-
-* Configuration options are defined in config.d/*.defconf.  You
-  shouldn't need to look in these files... but sometimes they contain
-  comments about options that are too long to fit into help strings.
-
-Keep it simple
-==============
-
-* I recommend that you aim for the smallest possible configuration file.
-  Perhaps start with:
-
-    FIRSTIP=<whatever>
-
-  and move on from there.
-
-* The NODES configuration variable controls the types of nodes that
-  are created.  At the time of writing, the default value is:
-
-    NODES="nas:0-3 base:4"
-
-  This means that you get 4 clustered NAS nodes, at IP offsets 0, 1,
-  2, & 3 from FIRSTIP, all part of the CTDB cluster.  You also get an
-  additional utility node at IP offset 4 that can be used, for
-  example, as a test client.  The base node will not be part of the
-  CTDB cluster.  It is just extra node that can be used as a test
-  client or similar.
-
-Corrupt system disks
-====================
-
-Recent versions of KVM seem to have fixed problems where the
-combination of qcow2 file format, virtio block devices and writeback
-caching would cause result in corrupt.  This means the default system
-disk bus type (a.k.a. SYSTEM_DISK_TYPE) is now virtio.
-
-If using an older version of KVM or if you experience corruption of
-the system disk, try using IDE system disks:
-
-  SYSTEM_DISK_TYPE=ide
-
-Raw IDE system disks
+DESTROYING A CLUSTER
 ====================
 
-Older RHEL versions of KVM did not support the SCSI block device
-emulation, and produced corruption when virtio disks were used with
-qcow2 disk images and writeback caching.  In this case, you can use
-either virtio system disks without any caching, accepting reduced
-performance, or you can use IDE system disks with writeback caching,
-with nice performance.
-
-For IDE disks, here are the required settings:
-
-  SYSTEM_DISK_TYPE=ide
-  SYSTEM_DISK_PREFIX=hd
-  SYSTEM_DISK_CACHE=writeback
-
-The next problem is that RHEL5's KVM does not include qemu-nbd.  The
-best solution is to build your own qemu-nbd and stop reading this
-section.
-
-If, for whatever reason, you're unable to build your own qemu-nbd,
-then you can use raw, rather than qcow2, system disks.  If you do this
-then you need significantly more disk space (since the system disks
-will be *copies* of the base image) and cluster creation time will no
-longer be pleasantly snappy (due to the copying time - the images are
-large and a single copy can take several minutes).  So, having tried
-to warn you off this option, if you really want to do this then you'll
-need these settings:
-
-  SYSTEM_DISK_FORMAT=raw
-  BASE_FORMAT=raw
+    # autocluster cluster foo destroy
 
 DEVELOPMENT HINTS
 =================
 
-The -e option provides support for executing arbitrary bash code.
-This is useful for testing and debugging.
-
-One good use of this option is to test template substitution using the
-function substitute_vars().  For example:
-
-  ./autocluster -c example.autocluster -e 'CLUSTER=foo; DISK=foo.qcow2; UUID=abcdef; NAME=foon1; set_macaddrs; substitute_vars templates/node.xml'
-
-This prints templates/node.xml with all appropriate substitutions
-done.  Some internal variables (e.g. CLUSTER, DISK, UUID, NAME) are
-given fairly arbitrary values but the various MAC address strings are
-set using the function set_macaddrs().
-
-The -e option is also useful when writing scripts that use
-autocluster.  Given the complexities of the configuration system you
-probably don't want to parse configuration files yourself to determine
-the current settings.  Instead, you can ask autocluster to tell you
-useful pieces of information.  For example, say you want to script
-creating a base disk image and you want to ensure the image is
-marked immutable:
-
-  base_image=$(autocluster -c $CONFIG -e 'echo $VIRTBASE/$BASENAME.img')
-  chattr -V -i "$base_image"
-
-  if autocluster -c $CONFIG create base ; then
-    chattr -V +i "$base_image"
-    ...
-
-Note that the command that autocluster should run is enclosed in
-single quotes.  This means that $VIRTBASE and $BASENAME will be expand
-within autocluster after the configuration file has been loaded.
+The Ansible playbook for nodes has been structured in a way that
+should make it easy to add new platforms and cluster filesystems.  Try
+to follow the pattern and keep task names as generic as possible.