Package org.antlr.runtime

Class UnbufferedTokenStream

java.lang.Object

org.antlr.runtime.misc.FastQueue<Token>

org.antlr.runtime.misc.LookaheadStream<Token>

org.antlr.runtime.UnbufferedTokenStream

All Implemented Interfaces:

IntStream, TokenStream

public class UnbufferedTokenStream
extends LookaheadStream<Token>
implements TokenStream

A token stream that pulls tokens from the code source on-demand and without tracking a complete buffer of the tokens. This stream buffers the minimum number of tokens possible. It's the same as OnDemandTokenStream except that OnDemandTokenStream buffers all tokens. You can't use this stream if you pass whitespace or other off-channel tokens to the parser. The stream can't ignore off-channel tokens. You can only look backwards 1 token: LT(-1). Use this when you need to read from a socket or other infinite stream.

See Also:

• BufferedTokenStream
• CommonTokenStream

Field Summary

Fields

Modifier and Type	Field	Description
protected int	channel	Skip tokens on any channel but this one; this is how we skip whitespace...
protected int	tokenIndex
protected TokenSource	tokenSource

Fields inherited from class org.antlr.runtime.misc.LookaheadStream

currentElementIndex, eof, lastMarker, markDepth, prevElement, UNINITIALIZED_EOF_ELEMENT_INDEX

Fields inherited from class org.antlr.runtime.misc.FastQueue

data, p, range

Constructor Summary

Constructors

Constructor	Description
UnbufferedTokenStream(TokenSource tokenSource)

Method Summary

Modifier and Type	Method	Description
Token	get(int i)	Get a token at an absolute index i; 0..n-1.
String	getSourceName()	Where are you getting symbols from? Normally, implementations will pass the buck all the way to the lexer who can ask its input stream for the file name or whatever.
TokenSource	getTokenSource()	Where is this stream pulling tokens from? This is not the name, but the object that provides Token objects.
boolean	isEOF(Token o)
int	LA(int i)	Get int at current input pointer + i ahead where i=1 is next int.
Token	nextElement()	Implement nextElement to supply a stream of elements to this lookahead buffer.
String	toString(int start, int stop)	Return the text of all tokens from start to stop, inclusive.
String	toString(Token start, Token stop)	Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location.

Methods inherited from class org.antlr.runtime.misc.LookaheadStream

consume, fill, index, LB, LT, mark, release, remove, reset, rewind, rewind, seek, size, syncAhead

Methods inherited from class org.antlr.runtime.misc.FastQueue

add, clear, elementAt, head, range, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.antlr.runtime.IntStream

consume, index, mark, release, rewind, rewind, seek, size

Methods inherited from interface org.antlr.runtime.TokenStream

LT, range

Field Details

tokenSource

protected TokenSource tokenSource

tokenIndex

protected int tokenIndex

channel

protected int channel

Skip tokens on any channel but this one; this is how we skip whitespace...

Constructor Details

UnbufferedTokenStream

public UnbufferedTokenStream(TokenSource tokenSource)

Method Details

nextElement

public Token nextElement()

Description copied from class: LookaheadStream

Implement nextElement to supply a stream of elements to this lookahead buffer. Return EOF upon end of the stream we're pulling from.

Specified by:

nextElement in class LookaheadStream<Token>

See Also:

• LookaheadStream.isEOF(T)

isEOF

public boolean isEOF(Token o)

Specified by:

isEOF in class LookaheadStream<Token>

getTokenSource

public TokenSource getTokenSource()

Description copied from interface: TokenStream

Where is this stream pulling tokens from? This is not the name, but the object that provides Token objects.

Specified by:

getTokenSource in interface TokenStream

toString

public String toString(int start,
 int stop)

Description copied from interface: TokenStream

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return "" or null; Users should not access $ruleLabel.text in an action of course in that case.

Specified by:

toString in interface TokenStream

toString

public String toString(Token start,
 Token stop)

Description copied from interface: TokenStream

Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location. Most often this will just delegate to the other toString(int,int). This is also parallel with the TreeNodeStream.toString(Object,Object).

Specified by:

toString in interface TokenStream

LA

public int LA(int i)

Description copied from interface: IntStream

Get int at current input pointer + i ahead where i=1 is next int. Negative indexes are allowed. LA(-1) is previous token (token just matched). LA(-i) where i is before first token should yield -1, invalid char / EOF.

Specified by:

LA in interface IntStream

get

public Token get(int i)

Description copied from interface: TokenStream

Get a token at an absolute index i; 0..n-1. This is really only needed for profiling and debugging and token stream rewriting. If you don't want to buffer up tokens, then this method makes no sense for you. Naturally you can't use the rewrite stream feature. I believe DebugTokenStream can easily be altered to not use this method, removing the dependency.

Specified by:

get in interface TokenStream

getSourceName

public String getSourceName()

Description copied from interface: IntStream

Where are you getting symbols from? Normally, implementations will pass the buck all the way to the lexer who can ask its input stream for the file name or whatever.

Specified by:

getSourceName in interface IntStream