Package org.fife.rsta.ac.java.rjc.lexer

Class Scanner

java.lang.Object

org.fife.rsta.ac.java.rjc.lexer.Scanner

public class Scanner
extends java.lang.Object

A scanner that allows the user to "push back" tokens. This scanner allows arbitrary lookahead.

Version:

1.0

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static class	Scanner.DocumentOffset

Field Summary

Fields

Modifier and Type	Field	Description
private int	currentResetStartOffset
private java.util.Stack<Token>	currentResetTokenStack
private static boolean	DEBUG
private javax.swing.text.Document	doc	If we are parsing text in a Swing JTextComponent, this should be the document of that component.
private Token	mostRecentToken	The most recently lexed token, or null if EOS was reached.
private java.util.Stack<java.util.Stack<Token>>	resetPositions
private SourceCodeScanner	s	The scanner we delegate to.
private java.util.Stack<Token>	stack	Stack of tokens that have been "pushed back".
private int	typeArgLevel	The depth in which we're in TypeArguments or TypeParameters.

Constructor Summary

Constructors

Constructor	Description
Scanner()	Constructor.
Scanner(java.io.Reader r)	Constructor.
Scanner(java.util.List<Token> tokens)	Constructor.

Method Summary

Modifier and Type	Method	Description
void	clearResetPosition()	Clears the position to reset to.
Offset	createOffset(int offs)	Returns an offset into the source being parsed.
private void	debugPrintToken(Token t)
void	decreaseTypeArgumentsLevel()	Decreases the depth in which we're in TypeArguments or TypeParameters.
void	eatParenPairs()	Eats through (possibly nested) paren pairs, e.g.:
void	eatThroughNext(int tokenType)	Eats all tokens up to (and including) the next token of the specified type.
void	eatThroughNextSkippingBlocks(int tokenType)	Eats all tokens up to (and including) the next token of the specified type.
Token	eatThroughNextSkippingBlocks(int tokenType1, int tokenType2)	Eats all tokens up to (and including) the next token of one of the specified types.
Token	eatThroughNextSkippingBlocksAndStuffInParens(int tokenType1, int tokenType2)	Eats all tokens up to (and including) the next token of one of the specified types.
void	eatUntilNext(int type1, int type2)	Eats tokens from the stream until one of the given types is found.
void	eatUntilNext(int type1, int type2, int type3)	Eats tokens from the stream until one of the given types is found.
int	getColumn()	Returns the current column into the current line.
java.lang.String	getLastDocComment()	Returns the last documentation comment parsed.
int	getLine()	Returns the current line into the document.
Token	getMostRecentToken()	Returns the most recently-lexed token.
int	getOffset()	Returns the current offset into the document.
int	getTypeArgumentsLevel()	Returns the current TypeArgument/TypeParameter level.
void	increaseTypeArgumentsLevel()	Increases the depth in which we're in TypeArguments or TypeParameters.
void	markResetPosition()	Marks a position to reset to.
private void	pushOntoStack(Token t)	This method is just here for debugging purposes to make sure our parser is sound.
void	resetToLastMarkedPosition()	Resets to the last marked position.
void	setDocument(javax.swing.text.Document doc)	Sets the Swing Document whose content is being parsed.
int	skipBracketPairs()	Skips all bracket pairs ('[' followed by ']') in the stream.
Token	yylex()	Returns the next token from the input stream.
Token	yylexNonNull(int type1, int type2, int type3, java.lang.String error)	Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of three given types.
Token	yylexNonNull(int type1, int type2, java.lang.String error)	Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of two given types.
Token	yylexNonNull(int type, java.lang.String error)	Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of a given type.
Token	yylexNonNull(java.lang.String error)	Returns the next token from the input stream, or throws an exception if the end of stream is reached.
Token	yyPeek()	Returns the next token, but does not take it off of the stream.
Token	yyPeek(int depth)	Returns the depth-th token, but does not anything off of the stream.
int	yyPeekCheckType()	Peeks at and returns the type of the next token on the stream.
int	yyPeekCheckType(int index)	Peeks at and returns the type of the specified token on the stream.
Token	yyPeekNonNull(int type1, int type2, int type3, java.lang.String error)	Returns the next token, but does not take it off of the stream.
Token	yyPeekNonNull(int type1, int type2, java.lang.String error)	Returns the next token, but does not take it off of the stream.
Token	yyPeekNonNull(int type, java.lang.String error)	Returns the next token, but does not take it off of the stream.
Token	yyPeekNonNull(java.lang.String error)	Returns the next token, but does not take it off of the stream.
void	yyPushback(Token t)	Pushes a token back onto the stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEBUG

private static final boolean DEBUG

See Also:

Constant Field Values

s

private SourceCodeScanner s

The scanner we delegate to.

stack

private java.util.Stack<Token> stack

Stack of tokens that have been "pushed back".

typeArgLevel

private int typeArgLevel

The depth in which we're in TypeArguments or TypeParameters.

doc

private javax.swing.text.Document doc

If we are parsing text in a Swing JTextComponent, this should be the document of that component.

mostRecentToken

private Token mostRecentToken

The most recently lexed token, or null if EOS was reached.

resetPositions

private java.util.Stack<java.util.Stack<Token>> resetPositions

currentResetTokenStack

private java.util.Stack<Token> currentResetTokenStack

currentResetStartOffset

private int currentResetStartOffset

Constructor Detail

Scanner

public Scanner()

Constructor. This scanner will return no tokens unless some are pushed onto it via yyPushback(Token).

Scanner

public Scanner(java.util.List<Token> tokens)

Constructor. This scanner will only return those tokens pushed onto it.

Parameters:

tokens - Tokens to return.

Scanner

public Scanner(java.io.Reader r)

Constructor.

Parameters:

r - The stream to read from.

Method Detail

pushOntoStack

private void pushOntoStack(Token t)

This method is just here for debugging purposes to make sure our parser is sound.

Parameters:

t - A token to push onto the stack (non-null).

decreaseTypeArgumentsLevel

public void decreaseTypeArgumentsLevel()

Decreases the depth in which we're in TypeArguments or TypeParameters.

See Also:

increaseTypeArgumentsLevel(), getTypeArgumentsLevel()

createOffset

public Offset createOffset(int offs)

Returns an offset into the source being parsed. This offset will be tracked if we are parsing code from a Swing JTextComponent.

Parameters:

offs - The offset.

Returns:

An object representing the offset.

See Also:

setDocument(Document)

debugPrintToken

private void debugPrintToken(Token t)

getColumn

public int getColumn()

Returns the current column into the current line.

Returns:

The current column.

See Also:

getLine()

getLastDocComment

public java.lang.String getLastDocComment()

Returns the last documentation comment parsed. The "last documentation comment" is cleared when this method returns.

Returns:

The last documentation comment parsed, or null if there was none.

getLine

public int getLine()

Returns the current line into the document.

Returns:

The current line.

See Also:

getColumn()

getMostRecentToken

public Token getMostRecentToken()

Returns the most recently-lexed token.

Returns:

The token, or null if EOS was reached.

getOffset

public int getOffset()

Returns the current offset into the document.

Returns:

The offset.

eatParenPairs

public void eatParenPairs()
                   throws java.io.IOException

Eats through (possibly nested) paren pairs, e.g.:

(int i=0; i<getFoo(getParam()); i++)

. Blocks nested inside the paren pairs are also skipped.

Throws:

java.io.IOException - If an IO error occurs.

java.lang.InternalError - If the next token is not a '('.

eatThroughNext

public void eatThroughNext(int tokenType)
                    throws java.io.IOException

Eats all tokens up to (and including) the next token of the specified type. This is useful, for example, to eat until the next semicolon.

Parameters:

tokenType - The type of token to eat through.

Throws:

java.io.IOException - If an IO error occurs.

eatThroughNextSkippingBlocks

public void eatThroughNextSkippingBlocks(int tokenType)
                                  throws java.io.IOException

Eats all tokens up to (and including) the next token of the specified type. This is useful, for example, to eat until the next semicolon.

Parameters:

tokenType - The type of token to eat through.

Throws:

java.io.IOException - If an IO error occurs.

See Also:

eatThroughNextSkippingBlocks(int, int), eatThroughNextSkippingBlocksAndStuffInParens(int, int)

eatThroughNextSkippingBlocks

public Token eatThroughNextSkippingBlocks(int tokenType1,
                                          int tokenType2)
                                   throws java.io.IOException

Eats all tokens up to (and including) the next token of one of the specified types. This is useful, for example, to eat until the next equal sign or semicolon.

Parameters:

tokenType1 - The type of token to eat through.

tokenType2 - Another type of token to eat through.

Returns:

The last token read. This will either be one of the two token types passed in, or null if the end of the stream is reached.

Throws:

java.io.IOException - If an IO error occurs.

See Also:

eatThroughNextSkippingBlocksAndStuffInParens(int, int)

eatThroughNextSkippingBlocksAndStuffInParens

public Token eatThroughNextSkippingBlocksAndStuffInParens(int tokenType1,
                                                          int tokenType2)
                                                   throws java.io.IOException

Eats all tokens up to (and including) the next token of one of the specified types. This is useful, for example, to eat until the next equal sign or semicolon.

Parameters:

tokenType1 - The type of token to eat through.

tokenType2 - Another type of token to eat through.

Returns:

The last token read. This will either be one of the two token types passed in, or null if the end of the stream is reached.

Throws:

java.io.IOException - If an IO error occurs.

See Also:

eatThroughNextSkippingBlocks(int, int)

eatUntilNext

public void eatUntilNext(int type1,
                         int type2)
                  throws java.io.IOException

Eats tokens from the stream until one of the given types is found. That token is kept in the stream and will be the next one returned.

Parameters:

type1 - The first desired type.

type2 - The second desired type.

Throws:

java.io.IOException - If an IO error occurs.

See Also:

eatUntilNext(int, int, int)

eatUntilNext

public void eatUntilNext(int type1,
                         int type2,
                         int type3)
                  throws java.io.IOException

Eats tokens from the stream until one of the given types is found. That token is kept in the stream and will be the next one returned.

Parameters:

type1 - The first desired type.

type2 - The second desired type.

type3 - The third desired type

Throws:

java.io.IOException - If an IO error occurs.

See Also:

eatUntilNext(int, int)

getTypeArgumentsLevel

public int getTypeArgumentsLevel()

Returns the current TypeArgument/TypeParameter level.

Returns:

The current level.

See Also:

increaseTypeArgumentsLevel(), decreaseTypeArgumentsLevel()

increaseTypeArgumentsLevel

public void increaseTypeArgumentsLevel()

Increases the depth in which we're in TypeArguments or TypeParameters.

See Also:

decreaseTypeArgumentsLevel(), getTypeArgumentsLevel()

markResetPosition

public void markResetPosition()

Marks a position to reset to.

See Also:

resetToLastMarkedPosition(), clearResetPosition()

resetToLastMarkedPosition

public void resetToLastMarkedPosition()

Resets to the last marked position.

See Also:

markResetPosition(), clearResetPosition()

clearResetPosition

public void clearResetPosition()

Clears the position to reset to.

See Also:

markResetPosition(), resetToLastMarkedPosition()

setDocument

public void setDocument(javax.swing.text.Document doc)

Sets the Swing Document whose content is being parsed. This method should be called if we are parsing code inside a JTextComponent, as it will help our parsed code to track changes when the document is modified. If we are parsing source from a flat file, this method shouldn't be called.

Parameters:

doc - The document being parsed.

skipBracketPairs

public int skipBracketPairs()
                     throws java.io.IOException

Skips all bracket pairs ('[' followed by ']') in the stream.

Returns:

The number of bracket pairs skipped.

Throws:

java.io.IOException - If an IO error occurs.

yylex

public Token yylex()
            throws java.io.IOException

Returns the next token from the input stream.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs.

yylexNonNull

public Token yylexNonNull(java.lang.String error)
                   throws java.io.IOException

Returns the next token from the input stream, or throws an exception if the end of stream is reached.

Parameters:

error - The error description for the exception if the end of stream is reached.

Returns:

The token.

Throws:

java.io.IOException - If an IO error occurs or the end of stream is reached.

yylexNonNull

public Token yylexNonNull(int type,
                          java.lang.String error)
                   throws java.io.IOException

Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of a given type.

Parameters:

type - The type the token must be.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The token.

Throws:

java.io.IOException - If an IO error occurs or the end of stream is reached, or if the token is of the wrong type.

yylexNonNull

public Token yylexNonNull(int type1,
                          int type2,
                          java.lang.String error)
                   throws java.io.IOException

Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of two given types.

Parameters:

type1 - One type the token can be.

type2 - Another type the token can be, or -1 if we should only check against type1.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The token.

Throws:

java.io.IOException - If an IO error occurs or the end of stream is reached, or if the token is of a wrong type.

yylexNonNull

public Token yylexNonNull(int type1,
                          int type2,
                          int type3,
                          java.lang.String error)
                   throws java.io.IOException

Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of three given types.

Parameters:

type1 - One type the token can be.

type2 - Another type the token can be, or -1 if we should only check against type1.

type3 - Another type the token can be, or -1 if we should only check against type1 and type2.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The token.

Throws:

java.io.IOException - If an IO error occurs or the end of stream is reached, or if the token is of a wrong type.

yyPeek

public Token yyPeek()
             throws java.io.IOException

Returns the next token, but does not take it off of the stream. This is useful for lookahead.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs.

yyPeek

public Token yyPeek(int depth)
             throws java.io.IOException

Returns the depth-th token, but does not anything off of the stream. This is useful for lookahead.

Parameters:

depth - The token to peek at, from 1 forward.

Returns:

The token, or null if that token index is past the end of the stream.

Throws:

java.io.IOException - If an IO error occurs.

yyPeekCheckType

public int yyPeekCheckType()
                    throws java.io.IOException

Peeks at and returns the type of the next token on the stream.

Returns:

The type of the next token, or -1 if the end of stream has been reached.

Throws:

java.io.IOException - If an IO error occurs.

yyPeekCheckType

public int yyPeekCheckType(int index)
                    throws java.io.IOException

Peeks at and returns the type of the specified token on the stream.

Parameters:

index - The index of the token to retrieve.

Returns:

The type of the token, or -1 if the end of stream was reached first.

Throws:

java.io.IOException - If an IO error occurs.

yyPeekNonNull

public Token yyPeekNonNull(java.lang.String error)
                    throws java.io.IOException

Returns the next token, but does not take it off of the stream. This is useful for lookahead.

Parameters:

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs.

yyPeekNonNull

public Token yyPeekNonNull(int type,
                           java.lang.String error)
                    throws java.io.IOException

Returns the next token, but does not take it off of the stream. This is useful for lookahead.

Parameters:

type - The type the token must be.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

yyPeekNonNull

public Token yyPeekNonNull(int type1,
                           int type2,
                           java.lang.String error)
                    throws java.io.IOException

Returns the next token, but does not take it off of the stream. This is useful for lookahead.

Parameters:

type1 - One of the two types the token must be.

type2 - The other of the two types the token must be.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

yyPeekNonNull

public Token yyPeekNonNull(int type1,
                           int type2,
                           int type3,
                           java.lang.String error)
                    throws java.io.IOException

Returns the next token, but does not take it off of the stream. This is useful for lookahead.

Parameters:

type1 - One of the three types the token must be.

type2 - Another of the three types the token must be.

type3 - The third of the types the token must be.

error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.

Returns:

The next token.

Throws:

java.io.IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

yyPushback

public void yyPushback(Token t)

Pushes a token back onto the stream.

Parameters:

t - The token.