summaryrefslogtreecommitdiffstats
path: root/lib/puppet/external/pson
Commit message (Collapse)AuthorAgeFilesLines
* Fix #5261 Don't escape Unicode characters in PSONJesse Wolfe2010-11-221-22/+1
| | | | | | | | | | | | | | | | | This patch removes the escaping of valid UTF-8 sequences as "\uXXXX". This code was unreliable, as it relied on Iconv's ability to convert those codepoints between UTF-8 and UTF-16, but some versions of Iconv barf on some valid codepoints. Invalid UTF-8 sequences are still passed through unchanged. We believe that this is fine; if you are concerned about complience with the JSON standard, what we are doing is equivalent to: * interpreting binary files as Latin-1 encoded character sequences * JSON-encoding those characters according to RFC 4627 * outputting the JSON as Latin-1 This allows all raw binary files to be transmitted losslessly. Paired-With: Paul Berry <paul@puppetlabs.com>
* Fix for #4832 -- Making PSON handle arbitrary binary dataMarkus Roberts2010-10-131-12/+5
| | | | | | | | | The PSON library needlessly assumed that the data to be transmitted was well- formed unicode. This made Latin-1 users (and anyone who needed to serialize arbitrary binary data) sad. This patch goes some of the way to resolving the issues, by passing through non-unicode data rather than just failing, adds tests, and cleans up a pernicious assumption about escape characters in ruby regular expressions not marked "n" (no-encoding).
* Code smell: Two space indentationMarkus Roberts2010-07-095-1068/+1068
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replaced 106806 occurances of ^( +)(.*$) with The ruby community almost universally (i.e. everyone but Luke, Markus, and the other eleven people who learned ruby in the 1900s) uses two-space indentation. 3 Examples: The code: end # Tell getopt which arguments are valid def test_get_getopt_args element = Setting.new :name => "foo", :desc => "anything", :settings => Puppet::Util::Settings.new assert_equal([["--foo", GetoptLong::REQUIRED_ARGUMENT]], element.getopt_args, "Did not produce appropriate getopt args") becomes: end # Tell getopt which arguments are valid def test_get_getopt_args element = Setting.new :name => "foo", :desc => "anything", :settings => Puppet::Util::Settings.new assert_equal([["--foo", GetoptLong::REQUIRED_ARGUMENT]], element.getopt_args, "Did not produce appropriate getopt args") The code: assert_equal(str, val) assert_instance_of(Float, result) end # Now test it with a passed object becomes: assert_equal(str, val) assert_instance_of(Float, result) end # Now test it with a passed object The code: end assert_nothing_raised do klass[:Yay] = "boo" klass["Cool"] = :yayness end becomes: end assert_nothing_raised do klass[:Yay] = "boo" klass["Cool"] = :yayness end
* Code smell: Line modifiers are preferred to one-line blocks.Markus Roberts2010-07-091-12/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Replaced 6 occurances of (while .*?) *do$ with The do is unneeded in the block header form and causes problems with the block-to-one-line transformation. 3 Examples: The code: while line = f.gets do becomes: while line = f.gets The code: while line = shadow.gets do becomes: while line = shadow.gets The code: while wrapper = zeros.pop do becomes: while wrapper = zeros.pop * Replaced 19 occurances of ((if|unless) .*?) *then$ with The then is unneeded in the block header form and causes problems with the block-to-one-line transformation. 3 Examples: The code: if f = test_files_for(failed).find { |f| failed_trace =~ Regexp.new(f) } then becomes: if f = test_files_for(failed).find { |f| failed_trace =~ Regexp.new(f) } The code: unless defined?(@spec_command) then becomes: unless defined?(@spec_command) The code: if c == ?\n then becomes: if c == ?\n * Replaced 758 occurances of ((?:if|unless|while|until) .*) (.*) end with The one-line form is preferable provided: * The condition is not used to assign a variable * The body line is not already modified * The resulting line is not too long 3 Examples: The code: if Puppet.features.libshadow? has_feature :manages_passwords end becomes: has_feature :manages_passwords if Puppet.features.libshadow? The code: unless (defined?(@current_pool) and @current_pool) @current_pool = process_zpool_data(get_pool_data) end becomes: @current_pool = process_zpool_data(get_pool_data) unless (defined?(@current_pool) and @current_pool) The code: if Puppet[:trace] puts detail.backtrace end becomes: puts detail.backtrace if Puppet[:trace]
* Code smell: English names for special globals rather than line-noiseMarkus Roberts2010-07-092-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Replaced 36 occurances of [$][?] with $CHILD_STATUS 3 Examples: The code: print "%s finished with exit code %s\n" % [host, $?.exitstatus] becomes: print "%s finished with exit code %s\n" % [host, $CHILD_STATUS.exitstatus] The code: $stderr.puts "Could not find host for PID %s with status %s" % [pid, $?.exitstatus] becomes: $stderr.puts "Could not find host for PID %s with status %s" % [pid, $CHILD_STATUS.exitstatus] The code: unless $? == 0 becomes: unless $CHILD_STATUS == 0 * Replaced 3 occurances of [$][$] with $PID 3 Examples: The code: Process.kill(:HUP, $$) if restart_requested? becomes: Process.kill(:HUP, $PID) if restart_requested? The code: if pid == $$ becomes: if pid == $PID The code: host[:name] = "!invalid.hostname.$$$" becomes: host[:name] = "!invalid.hostname.$PID$" * Replaced 7 occurances of [$]& with $MATCH 3 Examples: The code: work.slice!(0, $&.length) becomes: work.slice!(0, $MATCH.length) The code: if $& becomes: if $MATCH The code: if $& becomes: if $MATCH * Replaced 28 occurances of [$]:(?!:) with $LOAD_PATH 3 Examples: The code: sitelibdir = $:.find { |x| x =~ /site_ruby/ } becomes: sitelibdir = $LOAD_PATH.find { |x| x =~ /site_ruby/ } The code: $:.unshift "lib" becomes: $LOAD_PATH.unshift "lib" The code: $:.shift becomes: $LOAD_PATH.shift * Replaced 3 occurances of [$]! with $ERROR_INFO 3 Examples: The code: $LOG.fatal("Problem reading #{filepath}: #{$!}") becomes: $LOG.fatal("Problem reading #{filepath}: #{$ERROR_INFO}") The code: $stderr.puts "Couldn't build man pages: " + $! becomes: $stderr.puts "Couldn't build man pages: " + $ERROR_INFO The code: $stderr.puts $!.message becomes: $stderr.puts $ERROR_INFO.message * Replaced 3 occurances of ^(.*)[$]" with \1$LOADED_FEATURES 3 Examples: The code: unless $".index 'racc/parser.rb' becomes: unless $LOADED_FEATURES.index 'racc/parser.rb' The code: $".push 'racc/parser.rb' becomes: $LOADED_FEATURES.push 'racc/parser.rb' The code: $".should be_include("tmp/myfile.rb") becomes: $LOADED_FEATURES.should be_include("tmp/myfile.rb")
* Code smell: Use {} for % notation delimiters wherever practicalMarkus Roberts2010-07-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * * Replaced 16 occurances of %([qQrwWx])\((.*?)\) with %\1{\2} 3 Examples: The code: # %r(/) != /\// becomes: # %r{/} != /\// The code: ri = glob(%w(bin/*.rb sbin/* lib/**/*.rb)).reject { |e| e=~ /\.(bat|cmd)$/ } becomes: ri = glob(%w{bin/*.rb sbin/* lib/**/*.rb}).reject { |e| e=~ /\.(bat|cmd)$/ } The code: if !FileUtils.uptodate?("/var/cache/eix", %w(/usr/bin/eix /usr/portage/metadata/timestamp)) becomes: if !FileUtils.uptodate?("/var/cache/eix", %w{/usr/bin/eix /usr/portage/metadata/timestamp}) * Replaced 12 occurances of %([qQrwWx])\[(.*?)\] with %\1{\2} 3 Examples: The code: return send(command) if %w[get backup restore].include? command becomes: return send(command) if %w{get backup restore}.include? command The code: for iv in %w[indent space space_before object_nl array_nl check_circular allow_nan max_nesting] becomes: for iv in %w{indent space space_before object_nl array_nl check_circular allow_nan max_nesting} The code: @puppetd.command_line.stubs(:args).returns(%w[--logdest /my/file]) becomes: @puppetd.command_line.stubs(:args).returns(%w{--logdest /my/file}) * Replaced no occurances of %([qQrwWx])<(.*?)> with %\1{\2} * Replaced 19 occurances of %([qQrwWx])([^{\[(<])(.*?)\2 with %\1{\3} 3 Examples: The code: at.add_mapping(%r%^lib/puppet/(.*)\.rb$%) { |filename, m| becomes: at.add_mapping(%r{^lib/puppet/(.*)\.rb$}) { |filename, m| The code: at.add_mapping(%r%^spec/(unit|integration)/.*\.rb$%) { |filename, _| becomes: at.add_mapping(%r{^spec/(unit|integration)/.*\.rb$}) { |filename, _| The code: at.add_mapping(%r!^lib/puppet\.rb$!) { |filename, _| becomes: at.add_mapping(%r{^lib/puppet\.rb$}) { |filename, _|
* Code smell: Inconsistent indentation and related formatting issuesMarkus Roberts2010-07-095-1073/+1079
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Replaced 163 occurances of defined\? +([@a-zA-Z_.0-9?=]+) with defined?(\1) This makes detecting subsequent patterns easier. 3 Examples: The code: if ! defined? @parse_config becomes: if ! defined?(@parse_config) The code: return @option_parser if defined? @option_parser becomes: return @option_parser if defined?(@option_parser) The code: if defined? @local and @local becomes: if defined?(@local) and @local * Eliminate trailing spaces. Replaced 428 occurances of ^(.*?) +$ with \1 1 file was skipped. test/ral/providers/host/parsed.rb because 0 * Replace leading tabs with an appropriate number of spaces. Replaced 306 occurances of ^(\t+)(.*) with Tabs are not consistently expanded in all environments. * Don't arbitrarily wrap on sprintf (%) operator. Replaced 143 occurances of (.*['"] *%) +(.*) with Splitting the line does nothing to aid clarity and hinders further refactorings. 3 Examples: The code: raise Puppet::Error, "Cannot create %s: basedir %s is a file" % [dir, File.join(path)] becomes: raise Puppet::Error, "Cannot create %s: basedir %s is a file" % [dir, File.join(path)] The code: Puppet.err "Will not start without authorization file %s" % Puppet[:authconfig] becomes: Puppet.err "Will not start without authorization file %s" % Puppet[:authconfig] The code: $stderr.puts "Could not find host for PID %s with status %s" % [pid, $?.exitstatus] becomes: $stderr.puts "Could not find host for PID %s with status %s" % [pid, $?.exitstatus] * Don't break short arrays/parameter list in two. Replaced 228 occurances of (.*) +(.*) with 3 Examples: The code: puts @format.wrap(type.provider(prov).doc, :indent => 4, :scrub => true) becomes: puts @format.wrap(type.provider(prov).doc, :indent => 4, :scrub => true) The code: assert(FileTest.exists?(daily), "Did not make daily graph for %s" % type) becomes: assert(FileTest.exists?(daily), "Did not make daily graph for %s" % type) The code: assert(prov.target_object(:first).read !~ /^notdisk/, "Did not remove thing from disk") becomes: assert(prov.target_object(:first).read !~ /^notdisk/, "Did not remove thing from disk") * If arguments must wrap, treat them all equally Replaced 510 occurances of lines ending in things like ...(foo, or ...(bar(1,3), with \1 \2 3 Examples: The code: midscope.to_hash(false), becomes: assert_equal( The code: botscope.to_hash(true), becomes: # bottomscope, then checking that we see the right stuff. The code: :path => link, becomes: * Replaced 4516 occurances of ^( *)(.*) with The present code base is supposed to use four-space indentation. In some places we failed to maintain that standard. These should be fixed regardless of the 2 vs. 4 space question. 15 Examples: The code: def run_comp(cmd) puts cmd results = [] old_sync = $stdout.sync $stdout.sync = true line = [] begin open("| #{cmd}", "r") do |f| until f.eof? do c = f.getc becomes: def run_comp(cmd) puts cmd results = [] old_sync = $stdout.sync $stdout.sync = true line = [] begin open("| #{cmd}", "r") do |f| until f.eof? do c = f.getc The code: s.gsub!(/.{4}/n, '\\\\u\&') } string.force_encoding(Encoding::UTF_8) string rescue Iconv::Failure => e raise GeneratorError, "Caught #{e.class}: #{e}" end else def utf8_to_pson(string) # :nodoc: string = string.gsub(/["\\\x0-\x1f]/) { MAP[$&] } string.gsub!(/( becomes: s.gsub!(/.{4}/n, '\\\\u\&') } string.force_encoding(Encoding::UTF_8) string rescue Iconv::Failure => e raise GeneratorError, "Caught #{e.class}: #{e}" end else def utf8_to_pson(string) # :nodoc: string = string.gsub(/["\\\x0-\x1f]/) { MAP[$&] } string.gsub!(/( The code: end } rvalues: rvalue | rvalues comma rvalue { if val[0].instance_of?(AST::ASTArray) result = val[0].push(val[2]) else result = ast AST::ASTArray, :children => [val[0],val[2]] end } becomes: end } rvalues: rvalue | rvalues comma rvalue { if val[0].instance_of?(AST::ASTArray) result = val[0].push(val[2]) else result = ast AST::ASTArray, :children => [val[0],val[2]] end } The code: #passwdproc = proc { @password } keytext = @key.export( OpenSSL::Cipher::DES.new(:EDE3, :CBC), @password ) File.open(@keyfile, "w", 0400) { |f| f << keytext } becomes: # passwdproc = proc { @password } keytext = @key.export( OpenSSL::Cipher::DES.new(:EDE3, :CBC), @password ) File.open(@keyfile, "w", 0400) { |f| f << keytext } The code: end def to_manifest "%s { '%s':\n%s\n}" % [self.type.to_s, self.name, @params.collect { |p, v| if v.is_a? Array " #{p} => [\'#{v.join("','")}\']" else " #{p} => \'#{v}\'" end }.join(",\n") becomes: end def to_manifest "%s { '%s':\n%s\n}" % [self.type.to_s, self.name, @params.collect { |p, v| if v.is_a? Array " #{p} => [\'#{v.join("','")}\']" else " #{p} => \'#{v}\'" end }.join(",\n") The code: via the augeas tool. Requires: - augeas to be installed (http://www.augeas.net) - ruby-augeas bindings Sample usage with a string:: augeas{\"test1\" : context => \"/files/etc/sysconfig/firstboot\", changes => \"set RUN_FIRSTBOOT YES\", becomes: via the augeas tool. Requires: - augeas to be installed (http://www.augeas.net) - ruby-augeas bindings Sample usage with a string:: augeas{\"test1\" : context => \"/files/etc/sysconfig/firstboot\", changes => \"set RUN_FIRSTBOOT YES\", The code: names.should_not be_include("root") end describe "when generating a purgeable resource" do it "should be included in the generated resources" do Puppet::Type.type(:host).stubs(:instances).returns [@purgeable_resource] @resources.generate.collect { |r| r.ref }.should include(@purgeable_resource.ref) end end describe "when the instance's do not have an ensure property" do becomes: names.should_not be_include("root") end describe "when generating a purgeable resource" do it "should be included in the generated resources" do Puppet::Type.type(:host).stubs(:instances).returns [@purgeable_resource] @resources.generate.collect { |r| r.ref }.should include(@purgeable_resource.ref) end end describe "when the instance's do not have an ensure property" do The code: describe "when the instance's do not have an ensure property" do it "should not be included in the generated resources" do @no_ensure_resource = Puppet::Type.type(:exec).new(:name => '/usr/bin/env echo') Puppet::Type.type(:host).stubs(:instances).returns [@no_ensure_resource] @resources.generate.collect { |r| r.ref }.should_not include(@no_ensure_resource.ref) end end describe "when the instance's ensure property does not accept absent" do it "should not be included in the generated resources" do @no_absent_resource = Puppet::Type.type(:service).new(:name => 'foobar') becomes: describe "when the instance's do not have an ensure property" do it "should not be included in the generated resources" do @no_ensure_resource = Puppet::Type.type(:exec).new(:name => '/usr/bin/env echo') Puppet::Type.type(:host).stubs(:instances).returns [@no_ensure_resource] @resources.generate.collect { |r| r.ref }.should_not include(@no_ensure_resource.ref) end end describe "when the instance's ensure property does not accept absent" do it "should not be included in the generated resources" do @no_absent_resource = Puppet::Type.type(:service).new(:name => 'foobar') The code: func = nil assert_nothing_raised do func = Puppet::Parser::AST::Function.new( :name => "template", :ftype => :rvalue, :arguments => AST::ASTArray.new( :children => [stringobj(template)] ) becomes: func = nil assert_nothing_raised do func = Puppet::Parser::AST::Function.new( :name => "template", :ftype => :rvalue, :arguments => AST::ASTArray.new( :children => [stringobj(template)] ) The code: assert( @store.allowed?("hostname.madstop.com", "192.168.1.50"), "hostname not allowed") assert( ! @store.allowed?("name.sub.madstop.com", "192.168.0.50"), "subname name allowed") becomes: assert( @store.allowed?("hostname.madstop.com", "192.168.1.50"), "hostname not allowed") assert( ! @store.allowed?("name.sub.madstop.com", "192.168.0.50"), "subname name allowed") The code: assert_nothing_raised { server = Puppet::Network::Handler.fileserver.new( :Local => true, :Config => false ) } becomes: assert_nothing_raised { server = Puppet::Network::Handler.fileserver.new( :Local => true, :Config => false ) } The code: 'yay', { :failonfail => false, :uid => @user.uid, :gid => @user.gid } ).returns('output') output = Puppet::Util::SUIDManager.run_and_capture 'yay', @user.uid, @user.gid becomes: 'yay', { :failonfail => false, :uid => @user.uid, :gid => @user.gid } ).returns('output') output = Puppet::Util::SUIDManager.run_and_capture 'yay', @user.uid, @user.gid The code: ).times(1) pkg.provider.expects( :aptget ).with( '-y', '-q', 'remove', 'faff' becomes: ).times(1) pkg.provider.expects( :aptget ).with( '-y', '-q', 'remove', 'faff' The code: johnny one two billy three four\n" # Just parse and generate, to make sure it's isomorphic. assert_nothing_raised do assert_equal(text, @parser.to_file(@parser.parse(text)), "parsing was not isomorphic") end end def test_valid_attrs becomes: johnny one two billy three four\n" # Just parse and generate, to make sure it's isomorphic. assert_nothing_raised do assert_equal(text, @parser.to_file(@parser.parse(text)), "parsing was not isomorphic") end end def test_valid_attrs The code: "testing", :onboolean => [true, "An on bool"], :string => ["a string", "A string arg"] ) result = [] should = [] assert_nothing_raised("Add args failed") do @config.addargs(result) end @config.each do |name, element| becomes: "testing", :onboolean => [true, "An on bool"], :string => ["a string", "A string arg"] ) result = [] should = [] assert_nothing_raised("Add args failed") do @config.addargs(result) end @config.each do |name, element|
* Code smell: Miscellaneous oddity removalMarkus Roberts2010-07-091-1/+1
| | | | | | | | * Changed "string = self[1].gsub(%r((?:\\\\[\\\\bfnrt\"/]|(?:\\\\u(?:[A-Fa-f\\d]{4}))+|\\\\[\\x20-\\xff]))n) do |c|" to "string = self[1].gsub(%r{(?:\\\\[\\\\bfnrt\"/]|(?:\\\\u(?:[A-Fa-f\\d]{4}))+|\\\\[\\x20-\\xff])}n) do |c|" in lib/puppet/external/pson/pure/parser.rb * * Changed "\"$\"" to "'$'" in lib/puppet/provider/augeas/augeas.rb
* Fix for #2995 (don't fail to load PSON when UTF-8 missing)Markus Roberts2009-12-301-1/+1
| | | | | | We don't actually rely on iconv's UTF-8 support, so its absence shouldn't cause the PSON feature to fail on system (e.g. HPUX) where it isn't fully implemented.
* Ticket #2770 (deserializing Exec[...]s with "\n"s)Markus Roberts2009-11-121-2/+2
| | | | | | | | | | | The resource reference logic wasn't handling resources with "\n"s in their namevars gracefully, and detection of this was complicated by infelicitous exception reporting. Note that this patch will require a merge when combined with the patch for #2657. Signed-off-by: Markus Roberts <Markus@reality.com>
* Fix #2757 & CSR 92 (symlinks in recursively managed dirs)Markus Roberts2009-11-051-1/+1
| | | | | | | | | | | | | | | | | | The fundemental problem was that, despite what the comment said, the early bailout for file content management only applied to directories, not to links. Making links bail out at as well fixed the problem for most users. However, it would still occur for users with mixed ruby version system since there were no to_/from_pson methods for file metadata. So the second (and far larger) part of this patch adds metadata pson support. The testing is unit level only, as there's no pratical way to do the cross-ruby-version acceptance testing and no benifit to doing "integration" testing short of that. Signed-off-by: Markus Roberts <Markus@reductivelabs.com>
* Bundling of pure ruby json lib as "pson"Markus Roberts2009-10-175-0/+1150
Bundeling and renaming the pure ruby json library to addresses a number of cross version serliaization bugs (#2615, et al). This patch adds a subset of the files from the json_pure gem to lib/puppet/external/pson (renamed to avoid conflicts with rails) so that we will always have a known-good erialization format available. The pure ruby json gem as distibuted defers to the compiled version if it is installed. This is problematic in some circumstances so the files that have been brought over have been modified to always and only use the bundled version. It's a large patch, so here's a breakdown of the change categories: The majority of the lines are only marginally interesting: * The json lib itself (in lib/puppet/external/pson) make up the bulk of the lines. * Renaming of json to pson make up the second largest group. Somewhat more interesting are the following, which can be located by searching the diffs for the indicated strings: * Adjusting tests to reflect the changes * Changing the encoding/decoding behavior so that nested structures (e.g. resources) don't serialize as escaped strings. This should make it much easier to process the results with external tools, if needed. Search for "to_pson" and "to_pson_data_hash" * Cleaning up the envelope/metadata * Now provides a document_type (as opposed to a ruby class name) by using a symple registration scheme instead of constant lookup (search for "document_type") * Added an api_version (search for "api_version") * Added a hash for document metadata (search for "metadata") * Removing the yaml monkeypatch and instead disabling yaml serialization on ruby 1.8.1 in favor of pson (search for "yaml") * Cleaning up the json/rails feature interaction (they're now totally independent) (search for "feature")