summaryrefslogtreecommitdiffstats
path: root/README.queueing
diff options
context:
space:
mode:
authorJames Turnbull <james@lovedthanlost.net>2010-12-28 04:43:16 +1100
committerJames Turnbull <james@lovedthanlost.net>2011-04-11 07:55:12 +1000
commitd2145d9a02ce1802ceb44f0cf99090af62cc4b71 (patch)
tree2a1fca7225bf7ef884296dbf00a0a1ad6959f6e9 /README.queueing
parent50756588baad21d9b399051153274a89857433ef (diff)
downloadpuppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.tar.gz
puppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.tar.xz
puppet-d2145d9a02ce1802ceb44f0cf99090af62cc4b71.zip
Fixed #5684 - Move to Apache 2.0 license
For details of why we're moving please see: https://groups.google.com/d/topic/puppet-users/NuspYhMpE5o/discussion Removed explicit GPL licenses where appropriate Replaced selected GPL licenses with Apache 2.0 Replaced LICENSE with Apache 2.0 Updated README
Diffstat (limited to 'README.queueing')
-rw-r--r--README.queueing126
1 files changed, 0 insertions, 126 deletions
diff --git a/README.queueing b/README.queueing
deleted file mode 100644
index 83a8e19c0..000000000
--- a/README.queueing
+++ /dev/null
@@ -1,126 +0,0 @@
-*PUPPET QUEUEING
-
-Puppet Queueing is a feature which is designed to take some load
-off of the PuppetMaster by transferring the task of updating the
-database to a separate program which is named puppetqd (Puppet
-Queue Daemon).
-
-Currently this is only supported for "Storeconfigs" which is
-documented at:
-
-http://projects.puppetlabs.com/projects/1/wiki/Using_Stored_Configuration
-
-In the future this feature can be extended to any new puppet
-data which involves storage in a database.
-
-*OPERATION
-
-In a nutshell:
-
- puppetmasterd -> stomp -> service -> stomp -> puppetqd -> database
-
-At the moment the only messaging protocol supported is "stomp". Although
-others could be implemented, stomp is considered by many as the
-default queueing mechanism for Ruby and Rails applications. It is
-distributed as a Ruby gem and is easily installed.
-
-(The queueing code inside Puppet has been written so that when other
-interfaces and protocols are implemented they will be easy to use by
-changing settings in puppet.conf).
-
-The "service" in the diagram above is any queueing service that supports
-the Stomp API. For details refer to:
-
- http://xircles.codehaus.org/projects/stomp
-
-Both puppetmasterd and puppetqd subscribe to the same queueing service
-using the stomp interface. As puppetmasterd posts data to the queue,
-puppetqd receives it and stores it. The details of how to connect to
-the service and the name of the queue to use are set in puppet.conf:
-
- [main]
- queue_type = stomp
- queue_source = stomp://localhost:61613
- [puppetmasterd]
- async_storeconfigs = true
-
-Note: since puppetmasterd needs to recover the data being stored at a
-later time, both puppetmasterd and puppetqd need to work with the same
-database as defined in the STORECONFIGS setup.
-
-*QUEUEING SERVICES
-
-As mentioned previously any queueing service that supports the Stomp
-protocol can be used. Which one you use depends on your needs. We have
-tested with two of the most popular services - StompServer and ActiveMQ.
-
-+ StompServer
-
- http://rubyforge.org/projects/stompserver/
-
-StompServer is a lightweight queueing service written in Ruby which is
-suitable for testing or low volume puppet usage. Works well when both
-puppetmasterd and puppetd are running on the same machine that it's running
-on but we encountered some problems when using it from multiple machines.
-
-Just install the stompserver gem and run 'stompserver'.
-
-+ Apache ActiveMQ
-
- http://activemq.apache.org
-
-Considered by many to be the most popular message service in use today,
-ActiveMQ has hundreds of features for scaling, persistence and so on.
-
-Although installation is fairly simple, the configuration can seem quite
-intimidating, but for our use a one line change to the standard configuration
-is all that is required and is explained at:
-
- http://activemq.apache.org/stomp.html
-
-Other customization of the internal workings of ActiveMQ, if any, will depend
-on your needs and deployment. A quick skimming of the ActiveMQ documentation
-will give you enough info to decide.
-
-Others
-
-We have looked at but not tried some other queuing services which are
-compatible with the Stomp API:
-
-+ POE Component Message Queue
-+ JBoss Messaging (with 3rd party support for Stomp)
-
-*SCALING
-
-For StoreConfigs you basically need to have the catalog for a node stored
-in the database before the next time the node connects and asks for a
-new catalog.
-
-If the puppetd on your nodes is set to check every 30 minutes,
-then it would seem that there is no problem. However if you have 3000
-nodes you have a LOT of catalogs to store and it is possible you will
-not get a catalog saved in time.
-
-Running puppetmaster, your queueing service and puppetqd on the same
-machine means that they are all competing for the same CPU cycles. Bumping
-up the power of the server they are running on may be enough to handle
-even fairly large deployments.
-
-However since most queueing services (even StompServer) are designed to
-deliver messages from a "queue" to whoever asks for the next message you
-can split things up between machines:
-
- puppetmaster1 --\ /-- puppetqd1 -\
- puppetmaster2 ----> ActiveMQ ---> puppetqd2 ---> database
- puppetmaster3 --/ \-- puppetqd33 -/
- \- puppetqd4-/
-
-This is, of course a totally contrived example, but it gets the point
-across. As long as the data gets to the database, it doesn't matter
-which machines or services it goes through.
-
-Although for StoreConfigs absolute reliability is not a requirement as
-a new catalog will be sent the next time a node connects, some amount
-of persistence should some process crash may be desirable. Both ActiveMQ
-and MySQL (and other databases) have these kind of features built in
-which can be activated as needed.