diff options
Diffstat (limited to 'documentation/howitworks.page')
| -rw-r--r-- | documentation/howitworks.page | 283 |
1 files changed, 0 insertions, 283 deletions
diff --git a/documentation/howitworks.page b/documentation/howitworks.page deleted file mode 100644 index dfecbcbd7..000000000 --- a/documentation/howitworks.page +++ /dev/null @@ -1,283 +0,0 @@ ---- -inMenu: true -title: How It Works ---- -# Introduction - -The goal of this document is to describe how a manifest you write in Puppet -gets converted to work being done on the system. This process is relatively -complex, but you seldom need to know many of the details; this document only -exists for those who are pushing the boundaries of what Puppet can do or who -don't understand why they are seeing a particular error. It can also help -those who are hoping to extend Puppet beyond its current abilities. - -# High Level - -When looked at coarsely, Puppet has three main phases of execution -- -compiling, instantiation, and configuration. - -## Compiling - -Here is where we convert from a text-based manifest into the actual code we'll -be executing. Any code not meant for the host in question is ignored, and any -code that is meant for that host is fully interpolated, meaning that variables -are expanded and all of the results are literal strings. - -The only connection between the compiling phase and the library of Puppet -elements is that all resulting elements are verified that the referenced type -is valid and that all specified attributes are valid for that type. There is -no value validation at this point. - -In a networked setup, this phase happens entirely on the server. The output -of this phase is a collection of very simplistic elements that closely -resemble basic hashes and arrays. - -## Instantiation - -This phase converts the simple hashes and arrays into Puppet library objects. -Because this phase requires so much information about the client in order to -work correctly (e.g., what type of packaging is used, what type of services, -etc.), this phase happens entirely on the client. - -The conversion from the simpler format into literal Puppet objects allows -those objects to do greater validation on the inputs, and this is where most -of the input validation takes place. If you specified a valid attribute but -an invalid value, this is where you will find it out, meaning that you will -find it out when the config is instantiated on the client, not (unfortunately) -on the server. - -The output of this phase is the machine's entire configuration in memory and -in a form capable of modifying the local system. - -## Configuration - -This is where the Puppet library elements actually modify the system. Each of -them compares their specified state to the state on the machine and make any -modifications that are necessary. If the machine exactly matches the -specified configuration, then no work is done. - -The output of this phase is a correctly configured machine, in one pass. - -# Lower Level - -These three high level phases can each be broken down into more steps. - -## Compile Phase 1: Parsing - -* *Inputs* Manifests written in the Puppet language -* *Outputs* Parse trees (instances of [AST][ast] objects) -* *Entry* [Puppet::Parser::Parser#parse][parse] - -At this point, all Puppet manifests start out as text documents, and it's the -parser's job to understand those documents. The parser (defined in -``parser/grammar.ra`` and ``parser/lexer.rb``) does very little work -- it -converts from text to a format that maps directly back to the text, building -parse trees that are essentially equivalent to the text itself. The only -validation that takes place here is syntactic. - -This phase takes place immediately for all uses of Puppet. Whether you are -using nodes or no nodes, whether you are using the standalone puppet -interpreter or the client/server system, parsing happens as soon as Puppet -starts. - -## Compile Phase 2: Interpreting - -* *Inputs* Parse trees (instances of [AST][] objects) and client information - (collection of facts output by [Facter][]) -* *Outputs* Trees of [TransObject][] and [TransBucket][] instances (from - transportable.rb) -* *Entry* [Puppet::Parser::AST#evaluate][ast_evaluate] -* *Exit* [Puppet::Parser::Scope#to_trans][] - -Most configurations will rely on client information to make decisions. When -the Puppet client starts, it loads the [Facter][] library, collects all of the -facts that it can, and passes those facts to the interpreter. When you use -Puppet over a network, these facts are passed over the network to the server -and the server uses them to compile the client's configuration. - -This step of passing information to the server enables the server to make -decisions about the client based on things like operating system and hardware -architecture, and it also enables the server to insert information about the -client into the configuration, information like IP address and MAC address. - -The [interpreter][] combines the parse trees and the client information into a -tree of simple [transportable][] objects which maps roughly to the configuration -as defined in the manifests -- it is still a tree, but it is a tree of classes -and the elements contained in those classes. - -### Nodes vs. No Nodes - -When you use Puppet, you have the option of using [node elements][] or not. If -you do not use node elements, then the entire configuration is interpreted -every time a client connects, from the top of the parse tree down. In this -case, you must have some kind of explicit selection mechanism for specifying -which code goes with which node. - -If you do use nodes, though, the interpreter precompiles everything except the -node-specific code. When a node connects, the interpreter looks for the code -associated with that node name (retrieved from the Facter facts) and compiles -just that bit on demand. - -## Configuration Transport - -* *Inputs* [Transportable][] objects -* *Outputs* [Transportable][] objects -* *Entry* [Puppet::Server::Master#getconfig][] -* *Exit* [Puppet::Client::MasterClient#getconfig][] - -If you are using the stand-alone puppet executable, there is no configuration -transport because the client and server are in the same process. If you are -using the networked puppetd client and puppetmasterd server, though, the -configuration must be sent to the client once it is entirely compiled. - -Puppet currently converts the Transportable objects to [YAML][], which it then -CGI-escapes and sends over the wire using XMLRPC over HTTPS. The client -receives the configuration, unescapes it, caches it to disk in case the server -is not available on the next run, and then uses YAML to convert it back to -normal Ruby Transportable objects. - -## Instantiation Phase - -* *Inputs* [Transportable][] objects -* *Outputs* [Puppet::Type][] instances -* *Entry* [Puppet::Client::MasterClient#getconfig][] -* *Exit* [Puppet::Type#finalize][] - -To create Puppet library objects (all of which are instances of [Puppet::Type][] -subclasses), ``to_trans`` is called on the top-level transportable object. -All container objects get converted to [Puppet::Type::Component][] instances, -and all normal objects get converted into the appropriate Puppet type -instance. - -This is where all input validation takes place and often where values get -converted into more usable forms. For instance, filesystems always return -user IDs, not user names, so Puppet objects convert them appropriately. -(Incidentally, sometimes Puppet is creating the user that it's chowning a file -to, so whenever possible it ignores validation errors until the last minute.) - -The last phase of instantiation is the *finalization* phase. One of the goals -of the Puppet language is to make file order matter as little as possible; -this means that a Puppet object needs to be able to require other objects -listed later in the manifest, which means that the required object will be -instantiated after the requiring object. So, the finalization phase is used -to actually handle all of these requirements -- Puppet objects use their -references to objects and verify that the objects actually exist. - -## Configuration Phase 1: Comparison - -* *Inputs* [Puppet::Type][] instances -* *Outputs* [Puppet::StateChange][] objects collected in a [Puppet::Transaction][] - instance -* *Entry* [Puppet::Client::MasterClient#apply][] -* *Exit* [Puppet::Type::Component#evaluate][component_evaluate] - -Before Puppet does any work at all, it compares its entire configuration to -the state on disk (or in memory, or whatever). To do this, it recursively -iterates across the tree of [Puppet::Type][] instances (which, again, still -roughly maps to the class structure defined in the manifest) and calls -``evaluate``. - -Things are a bit messier than this in real life, but the summary is that -``evaluate`` retrieves the state of each object, compares that state to the -desired state, and creates a Puppet::StateChange object for every individual -bit that's out of sync (e.g., if a file has the wrong owner and wrong mode, -then each of those are in separate StateChange instances). The end result of -evaluating the whole tree is a collection of StateChange objects for every bit -that's out of sync, all sorted in order of dependencies so that objects are -always fixed before the objects that depend on them. - -The top-level component (which is also responsible for this sorting) creates a -Puppet::Transaction instance and inserts these changes into it. - -### Notes About Recursion - -Recursion muddies this phase considerably. While it's tempting to merely -handle recursion in the instantiation phase, the state on disk can (and will) -change between runs, so the configured state and the on-disk state must be -compared on every run (and it is assumed that ``puppetd`` will be a -long-running process that only does instantiation once but does configuration -many times). - -This means that there might still be objects that don't exist at the end of -instantiation but do exist at the end of comparison. In particular, when -doing recursive file copies from a remote machine, Puppet creates an object in -memory to map to every remote file, and that recursive object creation would -not make sense at instantiation time, only at comparison time. - -This might introduce some strangenesses, though, and it is expected that this -could cause interesting-in-a-not-particularly-good-way edge cases. - -## Configuration Phase 2: Syncing - -* *Inputs* [Puppet::Transaction][] instance containing [Puppet::StateChange][] - instances -* *Outputs* Completely configured operating system -* *Entry* [Puppet::Type::Component#evaluate][component_evaluate] -* *Exit* [Puppet::Transaction#evaluate][] - -The transaction's job is just to execute each change. The changes themselves -are responsible for logging everything that happens (one of the reasons that -all work is done by StateChange objects rather than just letting the objects -do it is to guarantee that every modification is logged). This execution is -done by calling ``go`` on each change in turn, and if the change does any work -then it produces an event of some kind. These events are collected until all -changes have been executed. - -Once the transaction is complete, all of the events are checked to see if -there are any callbacks associated with them. Puppet currently only supports -one type of callback and one way of specifying them: Calling ``refresh`` on -objects based on that object subscribing to another object. For instance, -take the following snippet: - - file { "/etc/ssh/sshd.conf": - source => "puppet://puppet/config/sshd.conf" - } - - service { sshd: - running => true, - subscribe => file["/etc/ssh/sshd.conf"] - } - -If the local file is out of sync with the remote file, then a StateChange -instance is created reflecting this. When that change is executed, it creates -a ``file_changed`` event. Because of the above subscription, the callback -associated with this event is to call ``refresh`` on the ``sshd`` service; for -services, ``refresh`` is equivalent to restarting, to sshd is restarted. In -this way, Puppet elements can react to changes that it makes to the system. - -While transactions are fully capable of moving both forward and backward -(e.g., if a transaction encountered an error, it could back out all of its -changes), there are currently no hooks within Puppet itself to specify when -and why that would happen. If this is a critical feature for you or you have -a brilliant way to go about creating it, I would love to hear it, but it is -currently a back-burner goal. - -# Conclusion - -That's the entire flow of how a Puppet manifest becomes a complete -configuration. There is more to the Puppet system, such as FileBuckets, but -those are more support staff rather than the main attraction. - -[facter]: /projects/facter -[parse]: /downloads/puppet/apidocs/classes/Puppet/Parser/Parser.html -[AST]: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html -[node elements]: /projects/puppet/documentation/structures#nodes -[yaml]: http://www.yaml.org/ -[Puppet::Parser::Parser#parse]: /downloads/puppet/apidocs/classes/Puppet/Parser/Parser.html -[ast_evaluate]: /downloads/puppet/apidocs/classes/Puppet/Parser/AST.html -[Puppet::Parser::Scope#to_trans]: /downloads/puppet/apidocs/classes/Puppet/Parser/Scope.html -[TransObject]: /downloads/puppet/apidocs/classes/Puppet/TransObject.html -[TransBucket]: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html -[Puppet::Server::Master#getconfig]: /downloads/puppet/apidocs/classes/Puppet/Server/Master.html -[Puppet::Client::MasterClient#getconfig]: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html -[Transportable]: /downloads/puppet/apidocs/classes/Puppet/TransBucket.html -[Puppet::StateChange]: /downloads/puppet/apidocs/classes/Puppet/StateChange.html -[Puppet::Transaction]: /downloads/puppet/apidocs/classes/Puppet/Transaction.html -[Puppet::Client::MasterClient#apply]: /downloads/puppet/apidocs/classes/Puppet/Client/MasterClient.html -[component_evaluate]: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html -[Puppet::Type::Component]: /downloads/puppet/apidocs/classes/Puppet/Type/Component.html -[Puppet::Transaction#evaluate]: /downloads/puppet/apidocs/classes/Puppet/Transaction.html -[interpreter]: /downloads/puppet/apidocs/classes/Puppet/Parser/Interpreter.html -[Puppet::Type]: /downloads/puppet/apidocs/classes/Puppet/Type.html -[Puppet::Type#finalize]: /downloads/puppet/apidocs/classes/Puppet/Type.html -*$Id$* |