Lexer Struct Reference

#include <lexer.h>

Public Types
enum	Token {   ERROR , BUILD , COLON , DEFAULT ,   EQUALS , IDENT , INCLUDE , INDENT ,   NEWLINE , PIPE , PIPE2 , PIPEAT ,   POOL , RULE , SUBNINJA , TEOF }

Public Member Functions
std::string	DescribeLastError ()
	If the last token read was an ERROR token, provide more info or the empty string. More...
bool	Error (const std::string &message, std::string *err)
	Construct an error message with context. More...
	Lexer ()
	Lexer (const char *input)
	Helper ctor useful for tests. More...
bool	PeekToken (Token token)
	If the next token is token, read it and return true. More...
bool	ReadIdent (std::string *out)
	Read a simple identifier (a rule or variable name). More...
bool	ReadPath (EvalString *path, std::string *err)
	Read a path (complete with $escapes). More...
Token	ReadToken ()
	Read a Token from the Token enum. More...
bool	ReadVarValue (EvalString *value, std::string *err)
	Read the value side of a var = value line (complete with $escapes). More...
void	Start (StringPiece filename, StringPiece input)
	Start parsing some input. More...
void	UnreadToken ()
	Rewind to the last read Token. More...

Static Public Member Functions
static const char *	TokenErrorHint (Token expected)
	Return a human-readable token hint, used in error messages. More...
static const char *	TokenName (Token t)
	Return a human-readable form of a token, used in error messages. More...

Private Member Functions
void	EatWhitespace ()
	Skip past whitespace (called after each read token/ident/etc.). More...
bool	ReadEvalString (EvalString *eval, bool path, std::string *err)
	Read a $-escaped string. More...

Private Attributes
StringPiece	filename_
StringPiece	input_
const char *	last_token_
const char *	ofs_

Detailed Description

Definition at line 27 of file lexer.h.

Member Enumeration Documentation

Token

enum Lexer::Token

Enumerator
ERROR
BUILD
COLON
DEFAULT
EQUALS
IDENT
INCLUDE
INDENT
NEWLINE
PIPE
PIPE2
PIPEAT
POOL
RULE
SUBNINJA
TEOF

Definition at line 32 of file lexer.h.

Constructor & Destructor Documentation

Lexer() [1/2]

Lexer::Lexer ( )

inline

Definition at line 28 of file lexer.h.

Lexer() [2/2]

Lexer::Lexer ( const char *  input )

explicit

Helper ctor useful for tests.

Definition at line 64 of file lexer.cc.

Member Function Documentation

DescribeLastError()

string Lexer::DescribeLastError

(

)

If the last token read was an ERROR token, provide more info or the empty string.

Definition at line 106 of file lexer.cc.

Referenced by DyndepParser::Parse(), ManifestParser::Parse(), and TEST().

EatWhitespace()

void Lexer::EatWhitespace ( )

private

Skip past whitespace (called after each read token/ident/etc.).

re2c [ ]+ { continue; } "$\r\n" { continue; } "$\n" { continue; } nul { break; } [^] { break; }

Definition at line 471 of file lexer.cc.

Error()

bool Lexer::Error	(	const std::string &	message,
		std::string *	err
	)

Construct an error message with context.

Definition at line 25 of file lexer.cc.

Referenced by Parser::Load(), DyndepParser::Parse(), ManifestParser::Parse(), ManifestParser::ParseDefault(), DyndepParser::ParseDyndepVersion(), DyndepParser::ParseEdge(), ManifestParser::ParseEdge(), DyndepParser::ParseLet(), ManifestParser::ParseLet(), ManifestParser::ParsePool(), and ManifestParser::ParseRule().

PeekToken()

bool Lexer::PeekToken

(

Token

token

)

If the next token is token, read it and return true.

Definition at line 463 of file lexer.cc.

Referenced by DyndepParser::ParseEdge(), ManifestParser::ParseEdge(), ManifestParser::ParsePool(), and ManifestParser::ParseRule().

ReadEvalString()

bool Lexer::ReadEvalString ( EvalString *  eval, bool  path, std::string *  err  )

private

Read a $-escaped string.

re2c [^$ :\r
|\000]+ { eval->AddText(StringPiece(start, p - start)); continue; } "\r\n" { if (path) p = start; break; } [ :|
] { if (path) { p = start; break; } else { if (*start == '
') break; eval->AddText(StringPiece(start, 1)); continue; } } "$$" { eval->AddText(StringPiece("$", 1)); continue; } "$ " { eval->AddText(StringPiece(" ", 1)); continue; } "$\r\n"[ ]* { continue; } "$\n"[ ]* { continue; } "${"varname"}" { eval->AddSpecial(StringPiece(start + 2, p - start - 3)); continue; } "$"simple_varname { eval->AddSpecial(StringPiece(start + 1, p - start - 1)); continue; } "$:" { eval->AddText(StringPiece(":", 1)); continue; } "$". { last_token_ = start; return Error("bad $-escape (literal $ must be written as $$)", err); } nul { last_token_ = start; return Error("unexpected EOF", err); } [^] { last_token_ = start; return Error(DescribeLastError(), err); }

Definition at line 623 of file lexer.cc.

References EvalString::AddSpecial(), EvalString::AddText(), and Error().

Referenced by ReadPath(), and ReadVarValue().

ReadIdent()

bool Lexer::ReadIdent

(

std::string *

out

)

Read a simple identifier (a rule or variable name).

Returns false if a name can't be read.

re2c varname { out->assign(start, p - start); break; } [^] { last_token_ = start; return false; }

Definition at line 554 of file lexer.cc.

Referenced by DyndepParser::ParseEdge(), ManifestParser::ParseEdge(), DyndepParser::ParseLet(), ManifestParser::ParseLet(), ManifestParser::ParsePool(), ManifestParser::ParseRule(), and TEST().

ReadPath()

bool Lexer::ReadPath ( EvalString *  path, std::string *  err  )

inline

Read a path (complete with $escapes).

Returns false only on error, returned path may be empty if a delimiter (space, newline) is hit.

Definition at line 80 of file lexer.h.

References ReadEvalString().

Referenced by ManifestParser::ParseDefault(), DyndepParser::ParseEdge(), ManifestParser::ParseEdge(), and ManifestParser::ParseFileInclude().

ReadToken()

Lexer::Token Lexer::ReadToken

(

)

Read a Token from the Token enum.

re2c re2c:define:YYCTYPE = "unsigned char"; re2c:define:YYCURSOR = p; re2c:define:YYMARKER = q; re2c:yyfill:enable = 0;

nul = "\000"; simple_varname = [a-zA-Z0-9_-]+; varname = [a-zA-Z0-9_.-]+;

[ ]*"#"[^\000
]*"\n" { continue; } [ ]*"\r\n" { token = NEWLINE; break; } [ ]*"\n" { token = NEWLINE; break; } [ ]+ { token = INDENT; break; } "build" { token = BUILD; break; } "pool" { token = POOL; break; } "rule" { token = RULE; break; } "default" { token = DEFAULT; break; } "=" { token = EQUALS; break; } ":" { token = COLON; break; } "|@" { token = PIPEAT; break; } "||" { token = PIPE2; break; } "|" { token = PIPE; break; } "include" { token = INCLUDE; break; } "subninja" { token = SUBNINJA; break; } varname { token = IDENT; break; } nul { token = TEOF; break; } [^] { token = ERROR; break; }

Definition at line 120 of file lexer.cc.

Referenced by DyndepParser::Parse(), ManifestParser::Parse(), and TEST().

ReadVarValue()

bool Lexer::ReadVarValue ( EvalString *  value, std::string *  err  )

inline

Read the value side of a var = value line (complete with $escapes).

Returns false only on error.

Definition at line 86 of file lexer.h.

References ReadEvalString().

Referenced by DyndepParser::ParseLet(), ManifestParser::ParseLet(), and TEST().

Start()

void Lexer::Start	(	StringPiece	filename,
		StringPiece	input
	)

Start parsing some input.

Definition at line 68 of file lexer.cc.

References StringPiece::str_.

Referenced by DyndepParser::Parse(), and ManifestParser::Parse().

TokenErrorHint()

const char * Lexer::TokenErrorHint ( Token  expected )

static

Return a human-readable token hint, used in error messages.

Definition at line 97 of file lexer.cc.

Referenced by Parser::ExpectToken().

TokenName()

const char * Lexer::TokenName ( Token  t )

static

Return a human-readable form of a token, used in error messages.

Definition at line 75 of file lexer.cc.

Referenced by Parser::ExpectToken(), DyndepParser::Parse(), and ManifestParser::Parse().

UnreadToken()

void Lexer::UnreadToken

(

)

Rewind to the last read Token.

Definition at line 116 of file lexer.cc.

Referenced by DyndepParser::Parse(), and ManifestParser::Parse().

Member Data Documentation

filename_

StringPiece Lexer::filename_

private

Definition at line 100 of file lexer.h.

input_

StringPiece Lexer::input_

private

Definition at line 101 of file lexer.h.

last_token_

const char* Lexer::last_token_

private

Definition at line 103 of file lexer.h.

ofs_

const char* Lexer::ofs_

private

Definition at line 102 of file lexer.h.

---

The documentation for this struct was generated from the following files:

• lexer.h
• lexer.cc
• lexer.in.cc