Class Scanner

java.lang.Object

org.codehaus.janino.Scanner

public class Scanner
extends Object

Splits up a character stream into tokens and returns them as String objects.

Field Summary

Fields

Modifier and Type	Field	Description
private boolean	crLfPending
private final String	fileName
private boolean	ignoreWhiteSpace
private final Reader	in
private static final Set<String>	JAVA_KEYWORDS
private static final Set<String>	JAVA_OPERATORS
private int	nextButOneChar
private int	nextChar
private int	nextCharColumnNumber
private int	nextCharLineNumber
private final StringBuilder	sb	Holds the characters of the currently scanned token.
static final String	SYSTEM_PROPERTY_SOURCE_DEBUGGING_DIR	If the source code is not read from a file, debuggers have a hard time locating the source file for source-level debugging.
static final String	SYSTEM_PROPERTY_SOURCE_DEBUGGING_ENABLE	Setting this system property to 'true' enables source-level debugging.
static final String	SYSTEM_PROPERTY_SOURCE_DEBUGGING_KEEP	If set to "true", then the temporary source code files are not deleted on exit.
private int	tokenColumnNumber	Column number of the first character of the previously produced token (1 if token is immediately preceded by a line break).
private int	tokenLineNumber	Line number of the previously produced token (typically starting at one).

Constructor Summary

Constructors

Constructor	Description
Scanner(File file)	Deprecated. This method is deprecated because it leaves the input file open
Scanner(File file, String encoding)	Deprecated. This method is deprecated because it leaves the input file open
Scanner(String fileName)	Deprecated. This method is deprecated because it leaves the input file open
Scanner(String fileName, InputStream is)	Sets up a scanner that reads tokens from the given InputStream in the platform default encoding.
Scanner(String fileName, InputStream is, String encoding)	Sets up a scanner that reads tokens from the given InputStream with the given encoding (null means platform default encoding).
Scanner(String fileName, Reader in)	Sets up a scanner that reads tokens from the given Reader.
Scanner(String fileName, Reader in, int initialLineNumber, int initialColumnNumber)	Creates a Scanner that counts lines and columns from non-default initial values.
Scanner(String fileName, String encoding)	Deprecated. This method is deprecated because it leaves the input file open

Method Summary

Modifier and Type	Method	Description
void	close()	Deprecated. This method is deprecated, because the concept described above is confusing.
String	getFileName()
private int	internalRead()
private static boolean	isBinaryDigit(int c)
private static boolean	isDecimalDigit(int c)	To comply with the JLS, this method does not allow for non-latin digit (like Character.isDigit(char) does).
private static boolean	isHexDigit(int c)	To comply with the JLS, this method does not allow for non-latin digit (like Character.isDigit(char) does).
private static boolean	isOctalDigit(int c)
Location	location()
private int	peek()	Returns the next character, but does not consume it.
private boolean	peek(String expectedCharacters)	Checks whether the next character is one of the expectedCharacters.
private int	peekButOne()	Returns the next-but-one character, but does not consume any characters.
private boolean	peekRead(int expected)	Consumes the next character iff it equals the expected character.
private boolean	peekRead(String expectedCharacters)	Consumes the next character iff it is one of the expectedCharacters
Token	produce()	Produces and returns the next token.
private char	read()	Consumes and returns the next character.
private TokenType	scan()
private void	scanLiteralCharacter()	Scans the next literal character into a StringBuilder.
private TokenType	scanNumericLiteral()
void	setIgnoreWhiteSpace(boolean value)	If value is true, then white space in the input stream is ignored, rather than scanned as a TokenType.WHITE_SPACE token.
private Token	token(TokenType type, String value)

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

SYSTEM_PROPERTY_SOURCE_DEBUGGING_ENABLE

public static final String SYSTEM_PROPERTY_SOURCE_DEBUGGING_ENABLE

Setting this system property to 'true' enables source-level debugging. Typically, this means that compilation is executed with "-g:all" instead of "-g:none".

See Also:

• Constant Field Values

SYSTEM_PROPERTY_SOURCE_DEBUGGING_DIR

public static final String SYSTEM_PROPERTY_SOURCE_DEBUGGING_DIR

If the source code is not read from a file, debuggers have a hard time locating the source file for source-level debugging. As a workaround, a copy of the source code is written to a temporary file, which must be included in the debugger's source path. If this system property is set, the temporary source file is created in that directory, otherwise in the default temporary-file directory.

See Also:

• File.createTempFile(String, String, File)
• Constant Field Values

SYSTEM_PROPERTY_SOURCE_DEBUGGING_KEEP

public static final String SYSTEM_PROPERTY_SOURCE_DEBUGGING_KEEP

If set to "true", then the temporary source code files are not deleted on exit. That may be useful to interpret stack traces offline.

See Also:

• Constant Field Values

sb

private final StringBuilder sb

Holds the characters of the currently scanned token.

fileName

@Nullable
private final String fileName

in

private final Reader in

ignoreWhiteSpace

private boolean ignoreWhiteSpace

nextChar

private int nextChar

nextButOneChar

private int nextButOneChar

crLfPending

private boolean crLfPending

nextCharLineNumber

private int nextCharLineNumber

nextCharColumnNumber

private int nextCharColumnNumber

tokenLineNumber

private int tokenLineNumber

Line number of the previously produced token (typically starting at one).

tokenColumnNumber

private int tokenColumnNumber

Column number of the first character of the previously produced token (1 if token is immediately preceded by a line break).

JAVA_KEYWORDS

private static final Set<String> JAVA_KEYWORDS

JAVA_OPERATORS

private static final Set<String> JAVA_OPERATORS

Constructor Details

Scanner

@Deprecated
public Scanner(String fileName)
        throws IOException

Deprecated.

This method is deprecated because it leaves the input file open

Throws:

IOException

Scanner

@Deprecated
public Scanner(String fileName,
 String encoding)
        throws IOException

Deprecated.

This method is deprecated because it leaves the input file open

Throws:

IOException

Scanner

@Deprecated
public Scanner(File file)
        throws IOException

Deprecated.

This method is deprecated because it leaves the input file open

Throws:

IOException

Scanner

@Deprecated
public Scanner(File file,
 @Nullable
 String encoding)
        throws IOException

Deprecated.

This method is deprecated because it leaves the input file open

Throws:

IOException

Scanner

public Scanner(@Nullable
 String fileName,
 InputStream is)
        throws IOException

Sets up a scanner that reads tokens from the given InputStream in the platform default encoding.

The fileName is solely used for reporting in thrown exceptions.

Throws:

IOException

Scanner

public Scanner(@Nullable
 String fileName,
 InputStream is,
 @Nullable
 String encoding)
        throws IOException

Sets up a scanner that reads tokens from the given InputStream with the given encoding (null means platform default encoding).

The fileName is used for reporting errors during compilation and for source level debugging, and should name an existing file. If null is passed, and the system property org.codehaus.janino.source_debugging.enable is set to "true", then a temporary file in org.codehaus.janino.source_debugging.dir or the system's default temp dir is created in order to make the source code available to a debugger.

Throws:

IOException

Scanner

public Scanner(@Nullable
 String fileName,
 Reader in)
        throws IOException

Sets up a scanner that reads tokens from the given Reader.

The fileName is used for reporting errors during compilation and for source level debugging, and should name an existing file. If null is passed, and the system property org.codehaus.janino.source_debugging.enable is set to "true", then a temporary file in org.codehaus.janino.source_debugging.dir or the system's default temp dir is created in order to make the source code available to a debugger.

Throws:

IOException

Scanner

public Scanner(@Nullable
 String fileName,
 Reader in,
 int initialLineNumber,
 int initialColumnNumber)
        throws IOException

Creates a Scanner that counts lines and columns from non-default initial values.

Throws:

IOException

Method Details

setIgnoreWhiteSpace

public void setIgnoreWhiteSpace(boolean value)

If value is true, then white space in the input stream is ignored, rather than scanned as a TokenType.WHITE_SPACE token. Since white space is typically quite numerous, this optimization may save considerable overhead.

getFileName

@Nullable
public String getFileName()

Returns:

The file name optionally passed to the constructor

close

@Deprecated
public void close()
           throws IOException

Deprecated.

This method is deprecated, because the concept described above is confusing. An application should close the underlying InputStream or Reader itself

Closes the character source (file, InputStream, Reader) associated with this object. The results of future calls to produce() are undefined.

Throws:

IOException

location

public Location location()

Returns:

The Location of the previously read (or peeked) token.

token

private Token token(TokenType type,
 String value)

produce

public Token produce()
              throws CompileException,
IOException

Produces and returns the next token. Notice that end-of-input is not signalized with a null product, but by an TokenType.END_OF_INPUT-type token.

Throws:

CompileException

IOException

scan

private TokenType scan()
                throws CompileException,
IOException

Throws:

CompileException

IOException

scanNumericLiteral

private TokenType scanNumericLiteral()
                              throws CompileException,
IOException

Throws:

CompileException

IOException

isDecimalDigit

private static boolean isDecimalDigit(int c)

To comply with the JLS, this method does not allow for non-latin digit (like Character.isDigit(char) does).

isHexDigit

private static boolean isHexDigit(int c)

To comply with the JLS, this method does not allow for non-latin digit (like Character.isDigit(char) does).

isOctalDigit

private static boolean isOctalDigit(int c)

isBinaryDigit

private static boolean isBinaryDigit(int c)

scanLiteralCharacter

private void scanLiteralCharacter()
                           throws CompileException,
IOException

Scans the next literal character into a StringBuilder.

Throws:

CompileException

IOException

peek

private int peek()
          throws CompileException,
IOException

Returns the next character, but does not consume it.

Returns:

-1 iff there is no next character, i.e the input stream is at end-of-input

Throws:

CompileException

IOException

peek

private boolean peek(String expectedCharacters)
              throws CompileException,
IOException

Checks whether the next character is one of the expectedCharacters.

It is not possible to peek exactly for "end-of-input"; use peekRead(-1) instead.

Throws:

CompileException

IOException

peekButOne

private int peekButOne()
                throws CompileException,
IOException

Returns the next-but-one character, but does not consume any characters.

Throws:

CompileException

IOException

read

private char read()
           throws CompileException,
IOException

Consumes and returns the next character.

Throws:

CompileException - There is no next charcter, i.e. the input stream is at end-of-input

IOException

peekRead

private boolean peekRead(int expected)
                  throws CompileException,
IOException

Consumes the next character iff it equals the expected character.

Returns:

Whether the next character equalled the expected character

Throws:

CompileException

IOException

peekRead

private boolean peekRead(String expectedCharacters)
                  throws CompileException,
IOException

Consumes the next character iff it is one of the expectedCharacters

Returns:

Whether the next character was one of the expectedCharacters

Throws:

CompileException

IOException

internalRead

private int internalRead()
                  throws IOException,
CompileException

Returns:

-1 iff there is no next character, i.e the input stream is at end-of-input

Throws:

IOException

CompileException