Elex scanner generator

• The homepage of Elex is here.
• Local version of the documentation is here.
• My patched version with Prolog support is here as a gzipped tar archive. The result is already quite usable: I do use it for some of the projects I am involved in. It's infinitely more flexible then doing it the ad-hoc way, or using DCGs on lists of character codes (well, Elex produces such a DCG automatically). To do: write an elex description for R.O.K's tokenizer (the `standard' Prolog tokenizer).
• A description of Elex(Prolog) can be found here (postscript), and here (html)

Tested with SICStus Prolog, BinProlog, SWI Prolog, Bprolog. Should also work with Quintus and other Edinburgh-style Prologs.

Example of using the prolog tokenizer. This uses the same example as the one in the original distribution, CalcScanner.elx (but of course I have added prolog code fragments). It contains statements such as:

  
defines

  # This section is used to define symbolic expressions that can be used
  # in later regular expressions.

  number = "0-9+"

begin
  # This is the main section, composed of a number of productions that
  # each define a symbol.  Each production is of the form
  #
  #  on [productionName] "regexp"
  #  <LANGUAGETAG
  #    code to be executed on a match of regexp
  #  >
  #


# Matches numbers like 1, -1.2, +1.1e10, -10e-2
  on Number "<number>(\.<number>)?([eE][\+\-]?<number>)?"
  <prolog
    number_chars(Number,__Token),
    __Tokens0=[number(Number)|__Tokens].
  >

  on Plus "\+"
  <prolog
    __Tokens0=[plus|__Tokens].
  >
  on WhiteSpace "[ \t\n]+"
  <prolog
    __Tokens0=__Tokens.
  >

  on error
  <prolog 
    % to continue after ignoring the next symbol:
    % the ignored symbol Sym is 'tokenized' as skip(Sym)
    __Tokens0 = [Skip|Rest],
    format(user_error,"Warning: skipping ~s~n",[[Skip]]),
    __Token = [skip(Skip)|Result],
    tokenize_all(Result,Rest,__Tokens).    
  >

After running the elex program on such a file, a Prolog file will be constructed with the tokenizer. This tokenizer can then be used as follows (here I use SICStus, but the resulting code is vanilla (Edinburgh style) Prolog):

% sicstus
SICStus 3  #3: Fri Mar 14 14:15:50 MET 1997
| ?- compile('CalcScanner.prolog').
{compiling /users1/vannoord/elex/demo/Prolog/CalcScanner.prolog...}
{compiled /users1/vannoord/elex/demo/Prolog/CalcScanner.prolog in module CalcScanner, 3570 msec 75792 bytes}

yes
| ?- 'CalcScanner':tokenize("123213.12 mod 2322.2 * * / 234.23",L).

L = [number(123213.12),mod,number(2322.2),star,star,slash,number(234.23)] ?

yes
| ?= 'CalcScanner':tokenize("1.2 3.14e-2 8+a_variable/(mod)
123.2e 1. !#$%^&",T).
Warning: skipping .
Warning: skipping !
Warning: skipping #
Warning: skipping $
Warning: skipping %
Warning: skipping ^
Warning: skipping &

T = [number(1.2),number(0.0314),number(8),plus,variable(a_variable),slash,
     lbracket,mod,rbracket,number(123.2),variable(e),number(1),skip(46),
     skip(33),skip(35),skip(36),skip(37),skip(94),skip(38)] ?
    

---

Gertjan van Noord