java.lang.Object
- org.htmlcleaner.HtmlTokenizer

```
public class HtmlTokenizer
extends java.lang.Object
```
Main HTML tokenizer.
It's task is to parse HTML and produce list of valid tokens: open tag tokens, end tag tokens, contents (text) and comments. As soon as new item is added to token list, cleaner is invoked to clean current list at the end.
Created by: Vladimir Nikic.
Date: November, 2006

Field Summary

Fields
Modifier and Type	Field	Description
`private boolean`	`_asExpected`
`private int`	`_col`
`private TagToken`	`_currentTagToken`
`private DoctypeToken`	`_docType`
`private boolean`	`_isLateForDoctype`
`private boolean`	`_isSpecialContext`
`private java.lang.String`	`_isSpecialContextName`
`private int`	`_len`
`private java.util.Set<java.lang.String>`	`_namespacePrefixes`
`private int`	`_pos`
`private java.io.BufferedReader`	`_reader`
`private int`	`_row`
`private java.lang.StringBuffer`	`_saved`
`private java.util.List<BaseToken>`	`_tokenList`
`private char[]`	`_working`
`private HtmlCleaner`	`cleaner`
`private CleanTimeValues`	`cleanTimeValues`
`private CleanerProperties`	`props`
`private CleanerTransformations`	`transformations`
`private static int`	`WORKING_BUFFER_SIZE`

Constructor Summary

Constructors
Constructor	Description
`HtmlTokenizer(HtmlCleaner cleaner, java.io.Reader reader, CleanTimeValues cleanTimeValues)`	Constructor - creates instance of the parser with specified content.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`private boolean`	`addSavedAsContent()`
`private void`	`addToken(BaseToken token)`
`private java.lang.String`	`attributeValue()`	Parses a single tag attribute - it is expected to be in one of the forms: name=value name="value" name='value' name
`private void`	`cdata()`
`private void`	`comment()`
`private boolean`	`containsEndCData()`
`private boolean`	`content()`
`private void`	`doctype()`
`DoctypeToken`	`getDocType()`
`(package private) java.util.Set<java.lang.String>`	`getNamespacePrefixes()`
`(package private) java.util.List<BaseToken>`	`getTokenList()`
`private void`	`go()`
`private void`	`go(int step)`
`private void`	`handleInterruption()`	Called whenver the thread is interrupted.
`private java.lang.String`	`identifier(boolean attribute)`	Parses an identifier from the current position.
`private void`	`ignoreUntil(char ch)`
`private boolean`	`isAllRead()`	Checks if end of the content is reached.
`private boolean`	`isChar(char ch)`	Checks if character at current runtime position is equal to specified char.
`private boolean`	`isChar(int position, char ch)`	Checks if character at specified position is equal to specified char.
`private boolean`	`isElementIdentifierStartChar(int position)`	Checks if character at specified position can be identifier start.
`private boolean`	`isHtmlAttributeIdentifierChar()`
`private boolean`	`isHtmlAttributeIdentifierChar(int position)`	Check whether the character at the specified position in the stream is a valid character for part of an attribute identifier in HTML
`private boolean`	`isHtmlAttributeIdentifierStartChar()`	Checks if character at current runtime position can be identifier start.
`private boolean`	`isHtmlElementIdentifier()`
`private boolean`	`isHtmlElementIdentifier(int position)`
`private boolean`	`isReservedTag(java.lang.String tagName)`	Checks if specified tag name is one of the reserved tags: HTML, HEAD or BODY
`private boolean`	`isTagStartOrEnd()`	Not all '<' (lt) symbols mean tag start or end.
`private boolean`	`isWhitespace()`	Checks if character at current runtime position is whitespace.
`private boolean`	`isWhitespace(int position)`	Checks if character at specified position is whitespace.
`private void`	`readIfNeeded(int neededChars)`
`private void`	`save(char ch)`	Saves specified character to the temporary buffer.
`private void`	`saveCurrent()`	Saves character at current runtime position to the temporary buffer.
`private void`	`saveCurrent(int size)`	Saves specified number of characters at current runtime position to the temporary buffer.
`private void`	`skipWhitespaces()`	Skips whitespaces at current position and moves foreward until non-whitespace character is found or the end of content is reached.
`(package private) void`	`start()`	Starts parsing HTML.
`private boolean`	`startsWith(java.lang.String value)`	Checks if content starts with specified value at the current position.
`private void`	`tagAttributes()`	Parses list tag attributes from the current position.
`private void`	`tagEnd()`	Parses end of the tag.
`private void`	`tagStart()`	Parses start of the tag.
`private void`	`updateCoordinates(char ch)`	Looks onto the char passed and updates current position coordinates.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - WORKING_BUFFER_SIZE
```
private static final int WORKING_BUFFER_SIZE
```
    See Also:
    
    Constant Field Values
  - _reader
```
private java.io.BufferedReader _reader
```
  - _working
```
private char[] _working
```
  - _pos
```
private transient int _pos
```
  - _len
```
private transient int _len
```
  - _row
```
private transient int _row
```
  - _col
```
private transient int _col
```
  - _saved
```
private transient java.lang.StringBuffer _saved
```
  - _isLateForDoctype
```
private transient boolean _isLateForDoctype
```
  - _docType
```
private transient DoctypeToken _docType
```
  - _currentTagToken
```
private transient TagToken _currentTagToken
```
  - _tokenList
```
private transient java.util.List<BaseToken> _tokenList
```
  - _namespacePrefixes
```
private transient java.util.Set<java.lang.String> _namespacePrefixes
```
  - _asExpected
```
private boolean _asExpected
```
  - _isSpecialContext
```
private boolean _isSpecialContext
```
  - _isSpecialContextName
```
private java.lang.String _isSpecialContextName
```
  - cleaner
```
private HtmlCleaner cleaner
```
  - props
```
private CleanerProperties props
```
  - transformations
```
private CleanerTransformations transformations
```
  - cleanTimeValues
```
private CleanTimeValues cleanTimeValues
```
- Constructor Detail
  - HtmlTokenizer
```
public HtmlTokenizer(HtmlCleaner cleaner,
                     java.io.Reader reader,
                     CleanTimeValues cleanTimeValues)
```
    Constructor - creates instance of the parser with specified content.
    
    Parameters:
    
    cleaner -
    
    reader -
- Method Detail
  - addToken
```
private void addToken(BaseToken token)
```
  - readIfNeeded
```
private void readIfNeeded(int neededChars)
                   throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - getTokenList
```
java.util.List<BaseToken> getTokenList()
```
  - getNamespacePrefixes
```
java.util.Set<java.lang.String> getNamespacePrefixes()
```
  - go
```
private void go()
         throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - go
```
private void go(int step)
         throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - startsWith
```
private boolean startsWith(java.lang.String value)
                    throws java.io.IOException
```
    Checks if content starts with specified value at the current position.
    
    Parameters:
    
    value -
    
    Returns:
    
    true if starts with specified value, false otherwise.
    
    Throws:
    
    java.io.IOException
  - isWhitespace
```
private boolean isWhitespace(int position)
```
    Checks if character at specified position is whitespace.
    
    Parameters:
    
    position -
    
    Returns:
    
    true is whitespace, false otherwise.
  - isWhitespace
```
private boolean isWhitespace()
```
    Checks if character at current runtime position is whitespace.
    
    Returns:
    
    true is whitespace, false otherwise.
  - isChar
```
private boolean isChar(int position,
                       char ch)
```
    Checks if character at specified position is equal to specified char.
    
    Parameters:
    
    position -
    
    ch -
    
    Returns:
    
    true is equals, false otherwise.
  - isChar
```
private boolean isChar(char ch)
```
    Checks if character at current runtime position is equal to specified char.
    
    Parameters:
    
    ch -
    
    Returns:
    
    true is equal, false otherwise.
  - isElementIdentifierStartChar
```
private boolean isElementIdentifierStartChar(int position)
```
    Checks if character at specified position can be identifier start.
    
    Parameters:
    
    position -
    
    Returns:
    
    true is may be identifier start, false otherwise.
  - isHtmlAttributeIdentifierStartChar
```
private boolean isHtmlAttributeIdentifierStartChar()
```
    Checks if character at current runtime position can be identifier start.
    
    Returns:
    
    true is may be identifier start, false otherwise.
  - isHtmlAttributeIdentifierChar
```
private boolean isHtmlAttributeIdentifierChar()
```
  - isHtmlElementIdentifier
```
private boolean isHtmlElementIdentifier()
```
  - isHtmlElementIdentifier
```
private boolean isHtmlElementIdentifier(int position)
```
  - isHtmlAttributeIdentifierChar
```
private boolean isHtmlAttributeIdentifierChar(int position)
```
    Check whether the character at the specified position in the stream is a valid character for part of an attribute identifier in HTML
    
    Parameters:
    
    position -
    
    Returns:
  - isAllRead
```
private boolean isAllRead()
```
    Checks if end of the content is reached.
  - save
```
private void save(char ch)
```
    Saves specified character to the temporary buffer.
    
    Parameters:
    
    ch -
  - updateCoordinates
```
private void updateCoordinates(char ch)
```
    Looks onto the char passed and updates current position coordinates. If char is a line break, increments row coordinate, if not -- col coordinate.
    
    Parameters:
    
    ch - - char to analyze.
  - saveCurrent
```
private void saveCurrent()
```
    Saves character at current runtime position to the temporary buffer.
  - saveCurrent
```
private void saveCurrent(int size)
                  throws java.io.IOException
```
    Saves specified number of characters at current runtime position to the temporary buffer.
    
    Throws:
    
    java.io.IOException
  - skipWhitespaces
```
private void skipWhitespaces()
                      throws java.io.IOException
```
    Skips whitespaces at current position and moves foreward until non-whitespace character is found or the end of content is reached.
    
    Throws:
    
    java.io.IOException
  - addSavedAsContent
```
private boolean addSavedAsContent()
```
  - start
```
void start()
    throws java.io.IOException
```
    Starts parsing HTML.
    
    Throws:
    
    java.io.IOException
  - isReservedTag
```
private boolean isReservedTag(java.lang.String tagName)
```
    Checks if specified tag name is one of the reserved tags: HTML, HEAD or BODY
    
    Parameters:
    
    tagName -
    
    Returns:
  - tagStart
```
private void tagStart()
               throws java.io.IOException
```
    Parses start of the tag. It expects that current position is at the "<" after which the tag's name follows.
    
    Throws:
    
    java.io.IOException
  - tagEnd
```
private void tagEnd()
             throws java.io.IOException
```
    Parses end of the tag. It expects that current position is at the "<" after which "/" and the tag's name follows.
    
    Throws:
    
    java.io.IOException
  - identifier
```
private java.lang.String identifier(boolean attribute)
                             throws java.io.IOException
```
    Parses an identifier from the current position.
    
    Throws:
    
    java.io.IOException
  - tagAttributes
```
private void tagAttributes()
                    throws java.io.IOException
```
    Parses list tag attributes from the current position.
    
    Throws:
    
    java.io.IOException
  - attributeValue
```
private java.lang.String attributeValue()
                                 throws java.io.IOException
```
    Parses a single tag attribute - it is expected to be in one of the forms: name=value name="value" name='value' name
    
    Throws:
    
    java.io.IOException
  - content
```
private boolean content()
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - isTagStartOrEnd
```
private boolean isTagStartOrEnd()
                         throws java.io.IOException
```
    Not all '<' (lt) symbols mean tag start or end. For example '<' can be part of mathematical expression. To avoid false breaks of content tags use this method to determine content tag end.
    
    Returns:
    
    true if current position is tag start or end.
    
    Throws:
    
    java.io.IOException
  - ignoreUntil
```
private void ignoreUntil(char ch)
                  throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - comment
```
private void comment()
              throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - cdata
```
private void cdata()
            throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - doctype
```
private void doctype()
              throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - getDocType
```
public DoctypeToken getDocType()
```
  - handleInterruption
```
private void handleInterruption()
```
    Called whenver the thread is interrupted. Currently this is a placeholder, but could hold cleanup methods and user interaction
  - containsEndCData
```
private boolean containsEndCData()
                          throws java.io.IOException
```
    Throws:
    
    java.io.IOException

Class HtmlTokenizer

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

WORKING_BUFFER_SIZE

_reader

_working

_pos

_len

_row

_col

_saved

_isLateForDoctype

_docType

_currentTagToken

_tokenList

_namespacePrefixes

_asExpected

_isSpecialContext

_isSpecialContextName

cleaner

props

transformations

cleanTimeValues

Constructor Detail

HtmlTokenizer

Method Detail

addToken

readIfNeeded

getTokenList

getNamespacePrefixes

go

go

startsWith

isWhitespace

isWhitespace

isChar

isChar

isElementIdentifierStartChar

isHtmlAttributeIdentifierStartChar

isHtmlAttributeIdentifierChar

isHtmlElementIdentifier

isHtmlElementIdentifier

isHtmlAttributeIdentifierChar

isAllRead

save

updateCoordinates

saveCurrent

saveCurrent

skipWhitespaces

addSavedAsContent

start

isReservedTag

tagStart

tagEnd

identifier

tagAttributes

attributeValue

content

isTagStartOrEnd

ignoreUntil

comment

cdata

doctype

getDocType

handleInterruption

containsEndCData