updating docs to work with webgen

git-svn-id: https://reductivelabs.com/svn/puppet/trunk@1293 980ebf18-57e1-0310-9a29-db15c13687c0

1 files changed, 281 insertions, 0 deletions

diff --git a/documentation/howitworks.page b/documentation/howitworks.page
new file mode 100644
index 000000000..0e63e651f
--- /dev/null
+++ b/documentation/howitworks.page
@@ -0,0 +1,281 @@
+---
+inMenu: true
+title: How It Works
+---
+# Introduction
+
+The goal of this document is to describe how a manifest you write in Puppet
+gets converted to work being done on the system.  This process is relatively
+complex, but you seldom need to know many of the details; this document only
+exists for those who are pushing the boundaries of what Puppet can do or who
+don't understand why they are seeing a particular error.  It can also help
+those who are hoping to extend Puppet beyond its current abilities.
+
+# High Level
+
+When looked at coarsely, Puppet has three main phases of execution --
+compiling, instantiation, and configuration.
+
+## Compiling
+
+Here is where we convert from a text-based manifest into the actual code we'll
+be executing.  Any code not meant for the host in question is ignored, and any
+code that is meant for that host is fully interpolated, meaning that variables
+are expanded and all of the results are literal strings.
+
+The only connection between the compiling phase and the library of Puppet
+elements is that all resulting elements are verified that the referenced type
+is valid and that all specified attributes are valid for that type.  There is
+no value validation at this point.
+
+In a networked setup, this phase happens entirely on the server.  The output
+of this phase is a collection of very simplistic elements that closely
+resemble basic hashes and arrays.
+
+## Instantiation
+
+This phase converts the simple hashes and arrays into Puppet library objects.
+Because this phase requires so much information about the client in order to
+work correctly (e.g., what type of packaging is used, what type of services,
+etc.), this phase happens entirely on the client.
+
+The conversion from the simpler format into literal Puppet objects allows
+those objects to do greater validation on the inputs, and this is where most
+of the input validation takes place.  If you specified a valid attribute but
+an invalid value, this is where you will find it out, meaning that you will
+find it out when the config is instantiated on the client, not (unfortunately)
+on the server.
+
+The output of this phase is the machine's entire configuration in memory and
+in a form capable of modifying the local system.
+
+## Configuration
+
+This is where the Puppet library elements actually modify the system.  Each of
+them compares their specified state to the state on the machine and make any
+modifications that are necessary.  If the machine exactly matches the
+specified configuration, then no work is done.
+
+The output of this phase is a correctly configured machine, in one pass.
+
+# Lower Level
+
+These three high level phases can each be broken down into more steps.
+
+## Compile Phase 1: Parsing
+
+:Inputs: Manifests written in the Puppet language
+:Outputs: Parse trees (instances of AST_ objects)
+:Entry: `Puppet::Parser::Parser#parse`_
+
+At this point, all Puppet manifests start out as text documents, and it's the
+parser's job to understand those documents.  The parser (defined in
+``parser/grammar.ra`` and ``parser/lexer.rb``) does very little work -- it
+converts from text to a format that maps directly back to the text, building
+parse trees that are essentially equivalent to the text itself.  The only
+validation that takes place here is syntactic.
+
+This phase takes place immediately for all uses of Puppet.  Whether you are
+using nodes or no nodes, whether you are using the standalone puppet
+interpreter or the client/server system, parsing happens as soon as Puppet
+starts.
+
+## Compile Phase 2: Interpreting
+
+:Inputs: Parse trees (instances of AST_ objects) and client information
+    (collection of facts output by [Facter][])
+:Outputs: Trees of TransObject_ and TransBucket_ instances (from
+    transportable.rb)
+:Entry: `Puppet::Parser::AST#evaluate`_
+:Exit: `Puppet::Parser::Scope#to_trans`_
+
+Most configurations will rely on client information to make decisions.  When
+the Puppet client starts, it loads the Facter_ library, collects all of the
+facts that it can, and passes those facts to the interpreter.  When you use
+Puppet over a network, these facts are passed over the network to the server
+and the server uses them to compile the client's configuration.
+
+This step of passing information to the server enables the server to make
+decisions about the client based on things like operating system and hardware
+architecture, and it also enables the server to insert information about the
+client into the configuration, information like IP address and MAC address.
+
+The interpreter_ combines the parse trees and the client information into a
+tree of simple transportable_ objects which maps roughly to the configuration
+as defined in the manifests -- it is still a tree, but it is a tree of classes
+and the elements contained in those classes.
+
+### Nodes vs. No Nodes
+
+When you use Puppet, you have the option of using `node elements`_ or not.  If
+you do not use node elements, then the entire configuration is interpreted
+every time a client connects, from the top of the parse tree down.  In this
+case, you must have some kind of explicit selection mechanism for specifying
+which code goes with which node.
+
+If you do use nodes, though, the interpreter precompiles everything except the
+node-specific code.  When a node connects, the interpreter looks for the code
+associated with that node name (retrieved from the Facter facts) and compiles
+just that bit on demand.
+
+## Configuration Transport
+
+:Inputs: Transportable_ objects
+:Outputs: Transportable_ objects
+:Entry: `Puppet::Server::Master#getconfig`_
+:Exit: `Puppet::Client::MasterClient#getconfig`_
+
+If you are using the stand-alone puppet executable, there is no configuration
+transport because the client and server are in the same process.  If you are
+using the networked puppetd client and puppetmasterd server, though, the
+configuration must be sent to the client once it is entirely compiled.
+
+Puppet currently converts the Transportable objects to YAML_, which it then
+CGI-escapes and sends over the wire using XMLRPC over HTTPS.  The client
+receives the configuration, unescapes it, caches it to disk in case the server
+is not available on the next run, and then uses YAML to convert it back to
+normal Ruby Transportable objects.
+
+## Instantiation Phase
+
+:Inputs: Transportable_ objects
+:Outputs: `Puppet::Type`_ instances
+:Entry: `Puppet::Client::MasterClient#getconfig`_
+:Exit: `Puppet::Type#finalize`_
+
+To create Puppet library objects (all of which are instances of `Puppet::Type`_
+subclasses), ``to_trans`` is called on the top-level transportable object.
+All container objects get converted to `Puppet::Type::Component`_ instances,
+and all normal objects get converted into the appropriate Puppet type
+instance.
+
+This is where all input validation takes place and often where values get
+converted into more usable forms.  For instance, filesystems always return
+user IDs, not user names, so Puppet objects convert them appropriately.
+(Incidentally, sometimes Puppet is creating the user that it's chowning a file
+to, so whenever possible it ignores validation errors until the last minute.)
+
+The last phase of instantiation is the *finalization* phase.  One of the goals
+of the Puppet language is to make file order matter as little as possible;
+this means that a Puppet object needs to be able to require other objects
+listed later in the manifest, which means that the required object will be
+instantiated after the requiring object.  So, the finalization phase is used
+to actually handle all of these requirements -- Puppet objects use their
+references to objects and verify that the objects actually exist.
+
+## Configuration Phase 1: Comparison
+
+:Inputs: `Puppet::Type`_ instances
+:Outputs: `Puppet::StateChange`_ objects collected in a `Puppet::Transaction`_
+    instance
+:Entry: `Puppet::Client::MasterClient#apply`_
+:Exit: `Puppet::Type::Component#evaluate`_
+
+Before Puppet does any work at all, it compares its entire configuration to
+the state on disk (or in memory, or whatever).  To do this, it recursively
+iterates across the tree of `Puppet::Type`_ instances (which, again, still
+roughly maps to the class structure defined in the manifest) and calls
+``evaluate``.
+
+Things are a bit messier than this in real life, but the summary is that
+``evaluate`` retrieves the state of each object, compares that state to the
+desired state, and creates a Puppet::StateChange object for every individual
+bit that's out of sync (e.g., if a file has the wrong owner and wrong mode,
+then each of those are in separate StateChange instances).  The end result of
+evaluating the whole tree is a collection of StateChange objects for every bit
+that's out of sync, all sorted in order of dependencies so that objects are
+always fixed before the objects that depend on them.
+
+The top-level component (which is also responsible for this sorting) creates a
+Puppet::Transaction instance and inserts these changes into it.
+
+### Notes About Recursion
+
+Recursion muddies this phase considerably.  While it's tempting to merely
+handle recursion in the instantiation phase, the state on disk can (and will)
+change between runs, so the configured state and the on-disk state must be
+compared on every run (and it is assumed that ``puppetd`` will be a
+long-running process that only does instantiation once but does configuration
+many times).
+
+This means that there might still be objects that don't exist at the end of
+instantiation but do exist at the end of comparison.  In particular, when
+doing recursive file copies from a remote machine, Puppet creates an object in
+memory to map to every remote file, and that recursive object creation would
+not make sense at instantiation time, only at comparison time.
+
+This might introduce some strangenesses, though, and it is expected that this
+could cause interesting-in-a-not-particularly-good-way edge cases.
+
+## Configuration Phase 2: Syncing
+
+:Inputs: `Puppet::Transaction`_ instance containing `Puppet::StateChange`_
+    instances
+:Outputs: Completely configured operating system
+:Entry: `Puppet::Type::Component#evaluate`_
+:Exit: `Puppet::Transaction#evaluate`_
+
+The transaction's job is just to execute each change.  The changes themselves
+are responsible for logging everything that happens (one of the reasons that
+all work is done by StateChange objects rather than just letting the objects
+do it is to guarantee that every modification is logged).  This execution is
+done by calling ``go`` on each change in turn, and if the change does any work
+then it produces an event of some kind.  These events are collected until all
+changes have been executed.
+
+Once the transaction is complete, all of the events are checked to see if
+there are any callbacks associated with them.  Puppet currently only supports
+one type of callback and one way of specifying them:  Calling ``refresh`` on
+objects based on that object subscribing to another object.  For instance,
+take the following snippet::
+
+    file { "/etc/ssh/sshd.conf":
+        source => "puppet://puppet/config/sshd.conf"
+    }
+
+    service { sshd:
+        running => true,
+        subscribe => file["/etc/ssh/sshd.conf"]
+    }
+
+If the local file is out of sync with the remote file, then a StateChange
+instance is created reflecting this.  When that change is executed, it creates
+a ``file_changed`` event.  Because of the above subscription, the callback
+associated with this event is to call ``refresh`` on the ``sshd`` service; for
+services, ``refresh`` is equivalent to restarting, to sshd is restarted.  In
+this way, Puppet elements can react to changes that it makes to the system.
+
+While transactions are fully capable of moving both forward and backward
+(e.g., if a transaction encountered an error, it could back out all of its
+changes), there are currently no hooks within Puppet itself to specify when
+and why that would happen.  If this is a critical feature for you or you have
+a brilliant way to go about creating it, I would love to hear it, but it is
+currently a back-burner goal.
+
+# Conclusion
+
+That's the entire flow of how a Puppet manifest becomes a complete
+configuration.  There is more to the Puppet system, such as FileBuckets, but
+those are more support staff rather than the main attraction.
+
+[facter] /projects/facter
+.. _node elements: /projects/puppet/documentation/structures#nodes
+.. _yaml: http://www.yaml.org/
+.. _Puppet::Parser::Parser#parse: /downloads/puppet/apidocs/classes/Puppet/Parser/Parser.html
+.. _Puppet::Parser::AST#evaluate: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html
+.. _Puppet::Parser::Scope#to_trans: /downloads/puppet/apidocs/classes/Puppet/Parser/Scope.html
+.. _AST: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html
+.. _TransObject: /downloads/puppet/apidocs/classes/Puppet/TransObject.html
+.. _TransBucket: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html
+.. _Puppet::Server::Master#getconfig: /downloads/puppet/apidocs/classes/Puppet/Server/Master.html
+.. _Puppet::Client::MasterClient#getconfig: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html
+.. _Transportable: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html
+.. _Puppet::StateChange: /downloads/puppet/apidocs/classes/Puppet/StateChange.html
+.. _Puppet::Transaction: /downloads/puppet/apidocs/classes/Puppet/Transaction.html
+.. _Puppet::Client::MasterClient#apply: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html
+.. _Puppet::Type::Component#evaluate: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html
+.. _Puppet::Type::Component: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html
+.. _Puppet::Transaction#evaluate: /downloads/puppet/apidocs/classes/Puppet/Transaction.html
+.. _interpreter: /downloads/puppet/apidocs/classes/Puppet/Parser/Interpreter.html
+.. _Puppet::Type: /downloads/puppet/apidocs/classes/Puppet/Type.html
+.. _Puppet::Type#finalize: /downloads/puppet/apidocs/classes/Puppet/Type.html