fri.patterns.interpreter.parsergenerator
Interface Lexer

All Known Implementing Classes:
LexerImpl

public interface Lexer

A Lexer reads input bytes (InputStream) or characters (Reader), until one of its terminals is fulfilled. This happens when the Parser calls getNextToken(). The terminals will be set by the Parser on init.

Usage:

        SyntaxSeparation separation = new SyntaxSeparation(new Syntax(myRules));
        LexerBuilder builder = new LexerBuilder(separation.getLexerSyntax(), separation.getIgnoredSymbols());
        Lexer lexer = builder.getLexer(inputStream);
        lexer.setTerminals(separation.getTokenSymbols());
        Token token;
        do	{
                token = lexer.getNextToken(null);
                System.err.println("token.symbol="+token.symbol+", text >"+token.text+"<");
        }
        while (token.symbol != null && Token.isEpsilon(token) == false);
        boolean error = token.symbol == null;
        

Author:
(c) 2000, Fritz Ritzberger
See Also:
LexerImpl

Nested Class Summary
static interface Lexer.TokenListener
          A way to receive every parsing syntax Token the Lexer reads, even it is ignored.
 
Method Summary
 void addTokenListener(Lexer.TokenListener listener)
          Installs a TokenListener that wants to know about every read Token, even it is ignored.
 void clear()
          Reset the Lexer for another pass.
 void dump(java.io.PrintStream out)
          Dump the current text and the scan position.
 Token getNextToken(java.util.Map tokenSymbols)
          Returns the next token from input.
 void removeTokenListener(Lexer.TokenListener listener)
          Removes a TokenListener from this Lexer.
 void setDebug(boolean debug)
          Turn on and off debug mode.
 void setInput(java.lang.Object text)
          Set the input to be processed by the Lexer.
 void setTerminals(java.util.List terminals)
          Tells the Lexer the terminals (tokens) to scan, called on init.
 

Method Detail

setInput

public void setInput(java.lang.Object text)
              throws java.io.IOException
Set the input to be processed by the Lexer.

Parameters:
text - can be String, StringBuffer, File, InputStream, Reader.
java.io.IOException

setTerminals

public void setTerminals(java.util.List terminals)
Tells the Lexer the terminals (tokens) to scan, called on init. Every terminal is a String that satisfies the facts defined in Token.isTerminal() (EPSILON or delimited by quotes).

Parameters:
terminals - List of String containing all terminals of the parser syntax.

getNextToken

public Token getNextToken(java.util.Map tokenSymbols)
                   throws java.io.IOException
Returns the next token from input. This is done trying to satisy the parser hints, or by using contained character consumers, which are obtained by the lexer strategy.

Parameters:
tokenSymbols - a Map that contains token symbols (in "key" field) expected by the Parser, can be null (slower).
Returns:
a Token with a terminal symbol and its instance text, or a Token with null symbol for error.
java.io.IOException

clear

public void clear()
Reset the Lexer for another pass.


addTokenListener

public void addTokenListener(Lexer.TokenListener listener)
Installs a TokenListener that wants to know about every read Token, even it is ignored.

Parameters:
listener - the Lexer.Listener implementation to install.

removeTokenListener

public void removeTokenListener(Lexer.TokenListener listener)
Removes a TokenListener from this Lexer.

Parameters:
listener - the Lexer.Listener implementation to remove.

dump

public void dump(java.io.PrintStream out)
Dump the current text and the scan position.


setDebug

public void setDebug(boolean debug)
Turn on and off debug mode.