Moving the string interpolation parsing to the parser/lexer

This patch moves the syntactic aspects of string interpolation up
into the lexer/parser phase, preparatory to moving the semantic
portions down to the as yet unnamed futures resolution phase.

This is an enabling move, designed to allow:

* Futures resolution in and between interpolated strings
* Interpolation of hash elements into strings
* Removal of certain order-dependent paths
* Further modularization of the lexer/parser

The key change is switching from viewing strings with interpolation
as single lexical entities (which await later special case processing)
to viewing them as formulas for constructing strings, with the internal
structure of the string exposed by the parser.

Thus a string like:

    "Hello $name, are you enjoying ${language_feature}?"

internally becomes something like:

    concat("Hello ",$name,", are you enjoying ",$language_feature,"?")

where "concat" is an internal string concatenation function.

A few test cases to show the user observable effects of this change:

    notice("string with ${'a nested single quoted string'} inside it.")
    $v2 = 3+4
    notice("string with ${['an array ',3,'+',4,'=',$v2]} in it.")
    notice("string with ${(3+5)/4} nested math ops in it.")

...and so forth.

The key changes in the internals are:

* Unification of SQTEXT and DQTEXT into a new token type STRING (since
  nothing past the lexer cares about the distinction.
* Creation of several new token types to represent the components of
  an interpolated string:

      DQPRE    The initial portion of an interpolated string
      DQMID    The portion of a string betwixt two interpolations
      DQPOST   The final portion of an interpolated string
      DQCONT   The as-yet-unlexed portion after an interpolation

  Thus, in the example above (phantom curly braces added for clarity),

      DQPRE    "Hello ${
      DQMID    }, are you enjoying ${
      DQPOST   }?"

  DQCONT is a bookkeeping token and is never generated.
* Creation of a DOLLAR_VAR token to strip the "$" off of variables
  with explicit dollar signs, so that the VARIABLEs produced from
  things like "Test ${x}" (where the "$" has already been consumed)
  do not fail for want of a "$"
* Reworking the grammar rules in the obvious way
* Introduction of a "concatenation" AST node type (which will be going
  away in a subsequent refactor).

Note finally that this is a component of a set of interrelated refactors,
and some of the changes around the edges of the above will only makes
sense in context of the other parts.

1 files changed, 11 insertions, 3 deletions

diff --git a/lib/puppet/parser/ast/leaf.rb b/lib/puppet/parser/ast/leaf.rb
index 4ad1f9ff3..caf1d13a5 100644
--- a/lib/puppet/parser/ast/leaf.rb
+++ b/lib/puppet/parser/ast/leaf.rb
@@ -52,10 +52,8 @@ class Puppet::Parser::AST
 
     # The base string class.
     class String < AST::Leaf
-        # Interpolate the string looking for variables, and then return
-        # the result.
         def evaluate(scope)
-            return scope.strinterp(@value, file, line)
+            @value
         end
 
         def to_s
@@ -74,6 +72,16 @@ class Puppet::Parser::AST
         end
     end
 
+    class Concat < AST::Leaf
+        def evaluate(scope)
+            @value.collect { |x| x.evaluate(scope) }.join
+        end
+
+        def to_s
+            "concat(#{@value.join(',')})"
+        end
+    end
+
     # The 'default' option on case statements and selectors.
     class Default < AST::Leaf; end