diff options
| author | James Turnbull <james@lovedthanlost.net> | 2010-12-28 04:43:16 +1100 |
|---|---|---|
| committer | James Turnbull <james@lovedthanlost.net> | 2011-04-11 07:55:12 +1000 |
| commit | d2145d9a02ce1802ceb44f0cf99090af62cc4b71 (patch) | |
| tree | 2a1fca7225bf7ef884296dbf00a0a1ad6959f6e9 /README.queueing | |
| parent | 50756588baad21d9b399051153274a89857433ef (diff) | |
| download | puppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.tar.gz puppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.tar.xz puppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.zip | |
Fixed #5684 - Move to Apache 2.0 license
For details of why we're moving please see:
https://groups.google.com/d/topic/puppet-users/NuspYhMpE5o/discussion
Removed explicit GPL licenses where appropriate
Replaced selected GPL licenses with Apache 2.0
Replaced LICENSE with Apache 2.0
Updated README
Diffstat (limited to 'README.queueing')
| -rw-r--r-- | README.queueing | 126 |
1 files changed, 0 insertions, 126 deletions
diff --git a/README.queueing b/README.queueing deleted file mode 100644 index 83a8e19c0..000000000 --- a/README.queueing +++ /dev/null @@ -1,126 +0,0 @@ -*PUPPET QUEUEING - -Puppet Queueing is a feature which is designed to take some load -off of the PuppetMaster by transferring the task of updating the -database to a separate program which is named puppetqd (Puppet -Queue Daemon). - -Currently this is only supported for "Storeconfigs" which is -documented at: - -http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration - -In the future this feature can be extended to any new puppet -data which involves storage in a database. - -*OPERATION - -In a nutshell: - - puppetmasterd -> stomp -> service -> stomp -> puppetqd -> database - -At the moment the only messaging protocol supported is "stomp". Although -others could be implemented, stomp is considered by many as the -default queueing mechanism for Ruby and Rails applications. It is -distributed as a Ruby gem and is easily installed. - -(The queueing code inside Puppet has been written so that when other -interfaces and protocols are implemented they will be easy to use by -changing settings in puppet.conf). - -The "service" in the diagram above is any queueing service that supports -the Stomp API. For details refer to: - - http://xircles.codehaus.org/projects/stomp - -Both puppetmasterd and puppetqd subscribe to the same queueing service -using the stomp interface. As puppetmasterd posts data to the queue, -puppetqd receives it and stores it. The details of how to connect to -the service and the name of the queue to use are set in puppet.conf: - - [main] - queue_type = stomp - queue_source = stomp://localhost:61613 - [puppetmasterd] - async_storeconfigs = true - -Note: since puppetmasterd needs to recover the data being stored at a -later time, both puppetmasterd and puppetqd need to work with the same -database as defined in the STORECONFIGS setup. - -*QUEUEING SERVICES - -As mentioned previously any queueing service that supports the Stomp -protocol can be used. Which one you use depends on your needs. We have -tested with two of the most popular services - StompServer and ActiveMQ. - -+ StompServer - - http://rubyforge.org/projects/stompserver/ - -StompServer is a lightweight queueing service written in Ruby which is -suitable for testing or low volume puppet usage. Works well when both -puppetmasterd and puppetd are running on the same machine that it's running -on but we encountered some problems when using it from multiple machines. - -Just install the stompserver gem and run 'stompserver'. - -+ Apache ActiveMQ - - http://activemq.apache.org - -Considered by many to be the most popular message service in use today, -ActiveMQ has hundreds of features for scaling, persistence and so on. - -Although installation is fairly simple, the configuration can seem quite -intimidating, but for our use a one line change to the standard configuration -is all that is required and is explained at: - - http://activemq.apache.org/stomp.html - -Other customization of the internal workings of ActiveMQ, if any, will depend -on your needs and deployment. A quick skimming of the ActiveMQ documentation -will give you enough info to decide. - -Others - -We have looked at but not tried some other queuing services which are -compatible with the Stomp API: - -+ POE Component Message Queue -+ JBoss Messaging (with 3rd party support for Stomp) - -*SCALING - -For StoreConfigs you basically need to have the catalog for a node stored -in the database before the next time the node connects and asks for a -new catalog. - -If the puppetd on your nodes is set to check every 30 minutes, -then it would seem that there is no problem. However if you have 3000 -nodes you have a LOT of catalogs to store and it is possible you will -not get a catalog saved in time. - -Running puppetmaster, your queueing service and puppetqd on the same -machine means that they are all competing for the same CPU cycles. Bumping -up the power of the server they are running on may be enough to handle -even fairly large deployments. - -However since most queueing services (even StompServer) are designed to -deliver messages from a "queue" to whoever asks for the next message you -can split things up between machines: - - puppetmaster1 --\ /-- puppetqd1 -\ - puppetmaster2 ----> ActiveMQ ---> puppetqd2 ---> database - puppetmaster3 --/ \-- puppetqd33 -/ - \- puppetqd4-/ - -This is, of course a totally contrived example, but it gets the point -across. As long as the data gets to the database, it doesn't matter -which machines or services it goes through. - -Although for StoreConfigs absolute reliability is not a requirement as -a new catalog will be sent the next time a node connects, some amount -of persistence should some process crash may be desirable. Both ActiveMQ -and MySQL (and other databases) have these kind of features built in -which can be activated as needed. |
