Proposed alternative solution for #2664 (REGEX / DIV lexing)

This is my proposed attack on the lexing problem, with a few minor
cleanups to simplify its integration.  The strategy:

* Anotate tokens with a method "acceptable?" that determines if
  they can be generated in a given context.  Have this default
  to true.
* Give the lexer the notion of a context; initialize it and
  update it as needed.  The present context records the name of
  the last significant token generated and a start_of_line flag.
* When a token is found to match, check if it is acceptable in
  the present context before generating it.

These changes don't result any any change in behaviour but they
enable:

* Give the REGEX token an acceptable? rule that only permits a
  regular expression in specific contexts.

The other changes were a fix to the scan bug Brice reported,
adjusting a test and clearing up some cluttered conditions in the
context collection path.

Added tests and subsumed change restricting REGEX to one line.

0 files changed, 0 insertions, 0 deletions