Package org.fife.rsta.ac.java.rjc.lexer

Class Scanner

java.lang.Object
org.fife.rsta.ac.java.rjc.lexer.Scanner
public class Scanner extends Object
A scanner that allows the user to "push back" tokens. This scanner allows arbitrary lookahead.
Version:
1.0

  • Field Details

    • DEBUG

      private static final boolean DEBUG
      See Also:

    • s

      private SourceCodeScanner s
      The scanner we delegate to.

    • stack

      private Stack<Token> stack
      Stack of tokens that have been "pushed back".

    • typeArgLevel

      private int typeArgLevel
      The depth in which we're in TypeArguments or TypeParameters.

    • doc

      private Document doc
      If we are parsing text in a Swing JTextComponent, this should be the document of that component.

    • mostRecentToken

      private Token mostRecentToken
      The most recently lexed token, or null if EOS was reached.

    • resetPositions

      private Stack<Stack<Token>> resetPositions

    • currentResetTokenStack

      private Stack<Token> currentResetTokenStack

    • currentResetStartOffset

      private int currentResetStartOffset

  • Constructor Details

    • Scanner

      public Scanner()
      Constructor. This scanner will return no tokens unless some are pushed onto it via yyPushback(Token).

    • Scanner

      public Scanner(List<Token> tokens)
      Constructor. This scanner will only return those tokens pushed onto it.
      Parameters:
      tokens - Tokens to return.

    • Scanner

      public Scanner(Reader r)
      Constructor.
      Parameters:
      r - The stream to read from.

  • Method Details

    • pushOntoStack

      private void pushOntoStack(Token t)
      This method is just here for debugging purposes to make sure our parser is sound.
      Parameters:
      t - A token to push onto the stack (non-null).

    • decreaseTypeArgumentsLevel

      public void decreaseTypeArgumentsLevel()
      Decreases the depth in which we're in TypeArguments or TypeParameters.
      See Also:

    • createOffset

      public Offset createOffset(int offs)
      Returns an offset into the source being parsed. This offset will be tracked if we are parsing code from a Swing JTextComponent.
      Parameters:
      offs - The offset.
      Returns:
      An object representing the offset.
      See Also:

    • debugPrintToken

      private void debugPrintToken(Token t)

    • getColumn

      public int getColumn()
      Returns the current column into the current line.
      Returns:
      The current column.
      See Also:

    • getLastDocComment

      public String getLastDocComment()
      Returns the last documentation comment parsed. The "last documentation comment" is cleared when this method returns.
      Returns:
      The last documentation comment parsed, or null if there was none.

    • getLine

      public int getLine()
      Returns the current line into the document.
      Returns:
      The current line.
      See Also:

    • getMostRecentToken

      public Token getMostRecentToken()
      Returns the most recently-lexed token.
      Returns:
      The token, or null if EOS was reached.

    • getOffset

      public int getOffset()
      Returns the current offset into the document.
      Returns:
      The offset.

    • eatParenPairs

      public void eatParenPairs() throws IOException
      Eats through (possibly nested) paren pairs, e.g.: 
      (int i=0; i<getFoo(getParam()); i++)
      . Blocks nested inside the paren pairs are also skipped.
      Throws:
      IOException - If an IO error occurs.
      InternalError - If the next token is not a '('.

    • eatThroughNext

      public void eatThroughNext(int tokenType) throws IOException
      Eats all tokens up to (and including) the next token of the specified type. This is useful, for example, to eat until the next semicolon.
      Parameters:
      tokenType - The type of token to eat through.
      Throws:
      IOException - If an IO error occurs.

    • eatThroughNextSkippingBlocks

      public void eatThroughNextSkippingBlocks(int tokenType) throws IOException
      Eats all tokens up to (and including) the next token of the specified type. This is useful, for example, to eat until the next semicolon.
      Parameters:
      tokenType - The type of token to eat through.
      Throws:
      IOException - If an IO error occurs.
      See Also:

    • eatThroughNextSkippingBlocks

      public Token eatThroughNextSkippingBlocks(int tokenType1, int tokenType2) throws IOException
      Eats all tokens up to (and including) the next token of one of the specified types. This is useful, for example, to eat until the next equal sign or semicolon.
      Parameters:
      tokenType1 - The type of token to eat through.
      tokenType2 - Another type of token to eat through.
      Returns:
      The last token read. This will either be one of the two token types passed in, or null if the end of the stream is reached.
      Throws:
      IOException - If an IO error occurs.
      See Also:

    • eatThroughNextSkippingBlocksAndStuffInParens

      public Token eatThroughNextSkippingBlocksAndStuffInParens(int tokenType1, int tokenType2) throws IOException
      Eats all tokens up to (and including) the next token of one of the specified types. This is useful, for example, to eat until the next equal sign or semicolon.
      Parameters:
      tokenType1 - The type of token to eat through.
      tokenType2 - Another type of token to eat through.
      Returns:
      The last token read. This will either be one of the two token types passed in, or null if the end of the stream is reached.
      Throws:
      IOException - If an IO error occurs.
      See Also:

    • eatUntilNext

      public void eatUntilNext(int type1, int type2) throws IOException
      Eats tokens from the stream until one of the given types is found. That token is kept in the stream and will be the next one returned.
      Parameters:
      type1 - The first desired type.
      type2 - The second desired type.
      Throws:
      IOException - If an IO error occurs.
      See Also:

    • eatUntilNext

      public void eatUntilNext(int type1, int type2, int type3) throws IOException
      Eats tokens from the stream until one of the given types is found. That token is kept in the stream and will be the next one returned.
      Parameters:
      type1 - The first desired type.
      type2 - The second desired type.
      type3 - The third desired type
      Throws:
      IOException - If an IO error occurs.
      See Also:

    • getTypeArgumentsLevel

      public int getTypeArgumentsLevel()
      Returns the current TypeArgument/TypeParameter level.
      Returns:
      The current level.
      See Also:

    • increaseTypeArgumentsLevel

      public void increaseTypeArgumentsLevel()
      Increases the depth in which we're in TypeArguments or TypeParameters.
      See Also:

    • markResetPosition

      public void markResetPosition()
      Marks a position to reset to.
      See Also:

    • resetToLastMarkedPosition

      public void resetToLastMarkedPosition()
      Resets to the last marked position.
      See Also:

    • clearResetPosition

      public void clearResetPosition()
      Clears the position to reset to.
      See Also:

    • setDocument

      public void setDocument(Document doc)
      Sets the Swing Document whose content is being parsed. This method should be called if we are parsing code inside a JTextComponent, as it will help our parsed code to track changes when the document is modified. If we are parsing source from a flat file, this method shouldn't be called.
      Parameters:
      doc - The document being parsed.

    • skipBracketPairs

      public int skipBracketPairs() throws IOException
      Skips all bracket pairs ('[' followed by ']') in the stream.
      Returns:
      The number of bracket pairs skipped.
      Throws:
      IOException - If an IO error occurs.

    • yylex

      public Token yylex() throws IOException
      Returns the next token from the input stream.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs.

    • yylexNonNull

      public Token yylexNonNull(String error) throws IOException
      Returns the next token from the input stream, or throws an exception if the end of stream is reached.
      Parameters:
      error - The error description for the exception if the end of stream is reached.
      Returns:
      The token.
      Throws:
      IOException - If an IO error occurs or the end of stream is reached.

    • yylexNonNull

      public Token yylexNonNull(int type, String error) throws IOException
      Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of a given type.
      Parameters:
      type - The type the token must be.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The token.
      Throws:
      IOException - If an IO error occurs or the end of stream is reached, or if the token is of the wrong type.

    • yylexNonNull

      public Token yylexNonNull(int type1, int type2, String error) throws IOException
      Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of two given types.
      Parameters:
      type1 - One type the token can be.
      type2 - Another type the token can be, or -1 if we should only check against type1.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The token.
      Throws:
      IOException - If an IO error occurs or the end of stream is reached, or if the token is of a wrong type.

    • yylexNonNull

      public Token yylexNonNull(int type1, int type2, int type3, String error) throws IOException
      Returns the next token from the input stream, or throws an exception if the end of stream is reached or if the token is not of three given types.
      Parameters:
      type1 - One type the token can be.
      type2 - Another type the token can be, or -1 if we should only check against type1.
      type3 - Another type the token can be, or -1 if we should only check against type1 and type2.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The token.
      Throws:
      IOException - If an IO error occurs or the end of stream is reached, or if the token is of a wrong type.

    • yyPeek

      public Token yyPeek() throws IOException
      Returns the next token, but does not take it off of the stream. This is useful for lookahead.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs.

    • yyPeek

      public Token yyPeek(int depth) throws IOException
      Returns the depth-th token, but does not anything off of the stream. This is useful for lookahead.
      Parameters:
      depth - The token to peek at, from 1 forward.
      Returns:
      The token, or null if that token index is past the end of the stream.
      Throws:
      IOException - If an IO error occurs.

    • yyPeekCheckType

      public int yyPeekCheckType() throws IOException
      Peeks at and returns the type of the next token on the stream.
      Returns:
      The type of the next token, or -1 if the end of stream has been reached.
      Throws:
      IOException - If an IO error occurs.

    • yyPeekCheckType

      public int yyPeekCheckType(int index) throws IOException
      Peeks at and returns the type of the specified token on the stream.
      Parameters:
      index - The index of the token to retrieve.
      Returns:
      The type of the token, or -1 if the end of stream was reached first.
      Throws:
      IOException - If an IO error occurs.

    • yyPeekNonNull

      public Token yyPeekNonNull(String error) throws IOException
      Returns the next token, but does not take it off of the stream. This is useful for lookahead.
      Parameters:
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs.

    • yyPeekNonNull

      public Token yyPeekNonNull(int type, String error) throws IOException
      Returns the next token, but does not take it off of the stream. This is useful for lookahead.
      Parameters:
      type - The type the token must be.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

    • yyPeekNonNull

      public Token yyPeekNonNull(int type1, int type2, String error) throws IOException
      Returns the next token, but does not take it off of the stream. This is useful for lookahead.
      Parameters:
      type1 - One of the two types the token must be.
      type2 - The other of the two types the token must be.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

    • yyPeekNonNull

      public Token yyPeekNonNull(int type1, int type2, int type3, String error) throws IOException
      Returns the next token, but does not take it off of the stream. This is useful for lookahead.
      Parameters:
      type1 - One of the three types the token must be.
      type2 - Another of the three types the token must be.
      type3 - The third of the types the token must be.
      error - The error description for the exception if the end of stream is reached, or if the token is of an unexpected type.
      Returns:
      The next token.
      Throws:
      IOException - If an IO error occurs, or if EOS is reached, or if the token is not of the specified type.

    • yyPushback

      public void yyPushback(Token t)
      Pushes a token back onto the stream.
      Parameters:
      t - The token.