Proposed alternative solution for #2664 (REGEX / DIV lexing)

This is my proposed attack on the lexing problem, with a few minor
cleanups to simplify its integration.  The strategy:

* Anotate tokens with a method "acceptable?" that determines if
  they can be generated in a given context.  Have this default
  to true.
* Give the lexer the notion of a context; initialize it and
  update it as needed.  The present context records the name of
  the last significant token generated and a start_of_line flag.
* When a token is found to match, check if it is acceptable in
  the present context before generating it.

These changes don't result any any change in behaviour but they
enable:

* Give the REGEX token an acceptable? rule that only permits a
  regular expression in specific contexts.

The other changes were a fix to the scan bug Brice reported,
adjusting a test and clearing up some cluttered conditions in the
context collection path.

Added tests and subsumed change restricting REGEX to one line.

1 files changed, 29 insertions, 8 deletions

diff --git a/spec/unit/parser/lexer.rb b/spec/unit/parser/lexer.rb
index 3c73ca986..959f36026 100755
--- a/spec/unit/parser/lexer.rb
+++ b/spec/unit/parser/lexer.rb
@@ -464,18 +464,39 @@ describe Puppet::Parser::Lexer::TOKENS[:REGEX] do
         @token.regex.should_not =~ "/this is \n a regex/"
     end
 
-    describe "when including escaped slashes" do
-        before { @lexer = Puppet::Parser::Lexer.new }
-
-        it "should not consider escaped slashes to be the end of a regex" do
-            @lexer.string = "/this \\/ foo/"
+    describe "when scanning" do
+        def tokens_scanned_from(s)
+            lexer = Puppet::Parser::Lexer.new
+            lexer.string = s
             tokens = []
-            @lexer.scan do |name, value|
+            lexer.scan do |name, value|
                 tokens << value
             end
-            tokens[0][:value].should == Regexp.new("this / foo")
+            tokens[0..-2]
         end
-    end
+
+        it "should not consider escaped slashes to be the end of a regex" do
+            tokens_scanned_from("$x =~ /this \\/ foo/").last[:value].should == Regexp.new("this / foo")
+        end
+
+        it "should not lex chained division as a regex" do
+            tokens_scanned_from("$x = $a/$b/$c").any? {|t| t[:value].class == Regexp }.should == false
+        end
+
+        it "should accept a regular expression after NODE" do
+            tokens_scanned_from("node /www.*\.mysite\.org/").last[:value].should == Regexp.new("www.*\.mysite\.org")
+        end
+
+        it "should accept regular expressions in a CASE" do
+            s = %q{case $variable {
+                "something": {$othervar = 4096 / 2}
+                /regex/: {notice("this notably sucks")}
+                }
+            }
+            tokens_scanned_from(s)[12][:value].should == Regexp.new("regex")
+        end
+ 
+   end
 
 
     it "should return the REGEX token and a Regexp" do