Fix #2333 - Make sure lexer skip whitespace on non-token

Comments and multi-line comments produces no token per-se during
lexing, so the lexer loops to find another token.
The issue was that we were not skipping whitespace after finding
such non-token.

Signed-off-by: Brice Figureau <brice-puppet@daysofwonder.com>

1 files changed, 4 insertions, 0 deletions

diff --git a/spec/unit/parser/lexer.rb b/spec/unit/parser/lexer.rb
index 2946c4152..2952b73c8 100755
--- a/spec/unit/parser/lexer.rb
+++ b/spec/unit/parser/lexer.rb
@@ -472,6 +472,10 @@ describe Puppet::Parser::Lexer, "when lexing comments" do
         @lexer.getcomment.should == "2\n"
     end
 
+    it "should skip whitespace before lexing the next token after a non-token" do
+        @lexer.string = "/* 1\n\n */ \ntest"
+        @lexer.fullscan.include?([:NAME, "test"]).should be_true
+    end
 end
 
 # FIXME: We need to rewrite all of these tests, but I just don't want to take the time right now.