java.lang.Object
- com.igormaznitsa.prologparser.tokenizer.Tokenizer

public final class Tokenizer
extends java.lang.Object

Internal tokenizer to extract next token from reader.

Field Summary

Fields
Modifier and Type	Field	Description
`private boolean`	`blockCommentsAllowed`
`private StringBuilderEx`	`insideCharBuffer`
`private TokenizerResult`	`lastPushedTerm`
`private int`	`lastTokenLine`
`private int`	`lastTokenPos`
`private int`	`line`
`private Koi7CharOpMap`	`metaOperators`
`private PrologParser`	`parser`
`private int`	`pos`
`private int`	`prevLine`
`private int`	`prevPos`
`private int`	`prevTokenLine`
`private int`	`prevTokenPos`
`private java.io.Reader`	`reader`
`private StringBuilderEx`	`specCharBuf`
`private StringBuilderEx`	`strBuf`
`private boolean`	`zeroQuotationAllowsWhitespaceChar`
`private boolean`	`zeroSingleQuotationAllowed`

Constructor Summary

Constructors
Constructor Description

Tokenizer(PrologParser parser, Koi7CharOpMap metaOperators, java.io.Reader reader)

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`void`	`calcDiffAndPushResultBack(java.lang.String etalon, StringBuilderEx buffer)`
`void`	`close(boolean closeReader)`
`private int`	`doReadChar()`
`OpContainer`	`findOperatorForName(java.lang.String operatorName)`
`OpContainer`	`findOperatorForSingleChar(char c)`
`void`	`fixPosition()`
`(package private) TokenizerResult`	`getLastPushed()`
`int`	`getLastTokenLine()`
`int`	`getLastTokenPos()`
`int`	`getLine()`
`int`	`getPos()`
`boolean`	`hasOperatorStartsWith(java.lang.String operatorNameStartSubstring)`
`static boolean`	`isCharAllowedForRadix(char chr, int radix)`
`PrologTerm`	`makeTermFromString(java.lang.String str, int radix, Quotation quotingType, TokenizerState state)`
`TokenizerResult`	`peek()`
`TokenizerResult`	`pop()`
`void`	`push(char ch)`
`void`	`push(TokenizerResult object)`
`TokenizerResult`	`readNextToken()`	Read next token
`private void`	`skipUntilBlockCommentEnd()`
`private void`	`skipUntilNextString()`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

strBuf
```
private final StringBuilderEx strBuf
```

specCharBuf

private final StringBuilderEx specCharBuf

insideCharBuffer

private final StringBuilderEx insideCharBuffer

blockCommentsAllowed

private final boolean blockCommentsAllowed

zeroSingleQuotationAllowed

private final boolean zeroSingleQuotationAllowed

zeroQuotationAllowsWhitespaceChar

private final boolean zeroQuotationAllowsWhitespaceChar

reader
```
private final java.io.Reader reader
```

parser
```
private final PrologParser parser
```

metaOperators

private final Koi7CharOpMap metaOperators

lastPushedTerm
```
private TokenizerResult lastPushedTerm
```

prevTokenLine
```
private int prevTokenLine
```

prevTokenPos
```
private int prevTokenPos
```

lastTokenLine
```
private int lastTokenLine
```

lastTokenPos
```
private int lastTokenPos
```

prevPos
```
private int prevPos
```

prevLine
```
private int prevLine
```

pos
```
private int pos
```

line
```
private int line
```

Constructor Detail

Tokenizer

public Tokenizer(PrologParser parser,
                 Koi7CharOpMap metaOperators,
                 java.io.Reader reader)

Method Detail

isCharAllowedForRadix

public static boolean isCharAllowedForRadix(char chr,
                                            int radix)

getLastPushed
```
TokenizerResult getLastPushed()
```

doReadChar

private int doReadChar()
                throws java.io.IOException

Throws:: java.io.IOException

calcDiffAndPushResultBack

public void calcDiffAndPushResultBack(java.lang.String etalon,
                                      StringBuilderEx buffer)

push
```
public void push(char ch)
```

close

public void close(boolean closeReader)
           throws java.io.IOException

Throws:: java.io.IOException

hasOperatorStartsWith

public boolean hasOperatorStartsWith(java.lang.String operatorNameStartSubstring)

findOperatorForName

public OpContainer findOperatorForName(java.lang.String operatorName)

findOperatorForSingleChar

public OpContainer findOperatorForSingleChar(char c)

push

public void push(TokenizerResult object)

peek
```
public TokenizerResult peek()
```

getLastTokenPos
```
public int getLastTokenPos()
```

getLastTokenLine
```
public int getLastTokenLine()
```

fixPosition
```
public void fixPosition()
```

skipUntilBlockCommentEnd

private void skipUntilBlockCommentEnd()
                               throws java.io.IOException

Throws:: java.io.IOException

skipUntilNextString

private void skipUntilNextString()
                          throws java.io.IOException

Throws:: java.io.IOException

pop
```
public TokenizerResult pop()
```

readNextToken
```
public TokenizerResult readNextToken()
```
Read next token

Returns:

next token or null if not found or stream ended

makeTermFromString

public PrologTerm makeTermFromString(java.lang.String str,
                                     int radix,
                                     Quotation quotingType,
                                     TokenizerState state)

getLine
```
public int getLine()
```

getPos
```
public int getPos()
```

Class Tokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

strBuf

specCharBuf

insideCharBuffer

blockCommentsAllowed

zeroSingleQuotationAllowed

zeroQuotationAllowsWhitespaceChar

reader

parser

metaOperators

lastPushedTerm

prevTokenLine

prevTokenPos

lastTokenLine

lastTokenPos

prevPos

prevLine

pos

line

Constructor Detail

Tokenizer

Method Detail

isCharAllowedForRadix

getLastPushed

doReadChar

calcDiffAndPushResultBack

push

close

hasOperatorStartsWith

findOperatorForName

findOperatorForSingleChar

push

peek

getLastTokenPos

getLastTokenLine

fixPosition

skipUntilBlockCommentEnd

skipUntilNextString

pop

readNextToken

makeTermFromString

getLine

getPos