fri.patterns.interpreter.parsergenerator
Class Token

java.lang.Object
  |
  +--fri.patterns.interpreter.parsergenerator.Token

public class Token

extends java.lang.Object

Lexer-Parser communication struct. Utility methods defining EPSILON and terminals. Definition of all global constants for parsergenerator packages.

This class defines special token identifiers: "token" (Token.TOKEN) and "ignored" (Token.IGNORED), which are needed when specifiying a syntax with mixed parser and lexer rules.

This class defines special symbols needed to define character sets with Token.UPTO (e.g. A..Z) and intersections with Token.BUTNOT (e.g. `char` - `newline`).

Author:

(c) 2000, Fritz Ritzberger

Nested Class Summary

static class	Token.Address Address stores input line number (1-n), column (0-n) and character/byte offset (0-n).
static class	Token.Range Range stores start and end Address of a token.

Field Summary

static java.lang.String	ARTIFICIAL_NONTERMINAL_START_CHARACTER The character used to mark artificial nonterminal (e.g. made from "prolog?").
static java.lang.String	BUTNOT Symbol used to define lexer character set intersections, e.g.
static char	CHAR_QUOTE Delimiter for literal terminal.
static char	COMMAND_QUOTE Delimiter for lexer rules (terminal).
static java.lang.String	EPSILON This special token symbol means "end of input" to the Parser, which stops calling getNextToken() then.
static java.lang.String	IGNORED Reserved identifier that marks tokens not to deliver to parser, e.g.
Token.Range	range The start and end Address of this Token.
static char	STRING_QUOTE Delimiter for literal terminals.
java.lang.String	symbol The syntax symbol this Token represents.
java.lang.Object	text The text that was scanned for this Token.
static java.lang.String	TOKEN Reserved identifier that marks nonterminals the lexer should deliver to parser.
static java.lang.String	UPTO Symbol used to define lexer character sets, e.g.

Constructor Summary

Token(java.lang.String symbol, java.lang.Object text, Token.Range range)

Method Summary

static boolean	isEpsilon(java.lang.String symbol) Epsilon means end of input, no more bytes available.
static boolean	isEpsilon(Token token) Epsilon means end of input, EOF, no more bytes available.
static boolean	isTerminal(java.lang.String symbol) Distinction of 'terminals' and nonterminals: terminals are either starting with digit or are enclosed in quotes '"` or equal to EPSILON.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EPSILON

public static final java.lang.String EPSILON

This special token symbol means "end of input" to the Parser, which stops calling getNextToken() then.

See Also:

Constant Field Values

STRING_QUOTE

public static final char STRING_QUOTE

Delimiter for literal terminals.

See Also:

Constant Field Values

CHAR_QUOTE

public static final char CHAR_QUOTE

Delimiter for literal terminal.

See Also:

Constant Field Values

COMMAND_QUOTE

public static final char COMMAND_QUOTE

Delimiter for lexer rules (terminal).

See Also:

Constant Field Values

UPTO

public static final java.lang.String UPTO

Symbol used to define lexer character sets, e.g. a..z.

See Also:

Constant Field Values

BUTNOT

public static final java.lang.String BUTNOT

Symbol used to define lexer character set intersections, e.g. char - "/*" - "//".

See Also:

Constant Field Values

TOKEN

public static final java.lang.String TOKEN

Reserved identifier that marks nonterminals the lexer should deliver to parser.

See Also:

Constant Field Values

IGNORED

public static final java.lang.String IGNORED

Reserved identifier that marks tokens not to deliver to parser, e.g. ignored ::= spaces ;.

See Also:

Constant Field Values

ARTIFICIAL_NONTERMINAL_START_CHARACTER

public static final java.lang.String ARTIFICIAL_NONTERMINAL_START_CHARACTER

The character used to mark artificial nonterminal (e.g. made from "prolog?"). Every artificial rule starts with it.

See Also:

Constant Field Values

symbol

public final java.lang.String symbol

The syntax symbol this Token represents.

text

public final java.lang.Object text

The text that was scanned for this Token.

range

public final Token.Range range

The start and end Address of this Token.

Constructor Detail

Token

public Token(java.lang.String symbol,
             java.lang.Object text,
             Token.Range range)

Method Detail

isEpsilon

public static boolean isEpsilon(Token token)

Epsilon means end of input, EOF, no more bytes available.

Returns:

true if passed token is the EPSILON-symbol.

isEpsilon

public static boolean isEpsilon(java.lang.String symbol)

Epsilon means end of input, no more bytes available.

Returns:

true if Token symbol not null and is the EPSILON-symbol.

isTerminal

public static boolean isTerminal(java.lang.String symbol)

Distinction of 'terminals' and nonterminals: terminals are either starting with digit or are enclosed in quotes '"` or equal to EPSILON.