Class Tokenizer

java.lang.Object

org.sonatype.maven.polyglot.atom.parsing.Tokenizer

public class Tokenizer
extends Object

Taken from the Loop programming language compiler pipeline.

Field Summary

Fields

Modifier and Type	Field	Description
private static final int[]	DELIMITERS
private final String	input
private static final int	NON
private static final int	SEQUENCE_TOKEN
private static final int	SINGLE_TOKEN
private static final boolean[]	STRING_TERMINATORS

Constructor Summary

Constructors

Constructor	Description
Tokenizer(String input)

Method Summary

Modifier and Type	Method	Description
private static void	bakeToken(List<Token> tokens, char[] input, int i, int start)
static String	detokenize(List<Token> tokens)
private static boolean	isDelimiter(char c)
(package private) static boolean	isSingleTokenDelimiter(char c)
private static boolean	isWhitespace(char c)
List<Token>	tokenize()

Methods inherited from class Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

input

private final String input

NON

private static final int NON

See Also:

• Constant Field Values

SINGLE_TOKEN

private static final int SINGLE_TOKEN

See Also:

• Constant Field Values

SEQUENCE_TOKEN

private static final int SEQUENCE_TOKEN

See Also:

• Constant Field Values

DELIMITERS

private static final int[] DELIMITERS

STRING_TERMINATORS

private static final boolean[] STRING_TERMINATORS

Constructor Details

Tokenizer

public Tokenizer(String input)

Method Details

tokenize

public List<Token> tokenize()

isWhitespace

private static boolean isWhitespace(char c)

isSingleTokenDelimiter

static boolean isSingleTokenDelimiter(char c)

detokenize

public static String detokenize(List<Token> tokens)

isDelimiter

private static boolean isDelimiter(char c)

bakeToken

private static void bakeToken(List<Token> tokens,
 char[] input,
 int i,
 int start)