clang::TokenLexer Class Reference

TokenLexer - This implements a lexer that returns tokens from a macro body or token stream instead of lexing from a character buffer. More...

#include "clang/Lex/TokenLexer.h"

Public Member Functions
	TokenLexer (Token &Tok, SourceLocation ILEnd, MacroInfo *MI, MacroArgs *ActualArgs, Preprocessor &pp)
	Create a TokenLexer for the specified macro with the specified actual arguments.
	TokenLexer (const Token *TokArray, unsigned NumToks, bool DisableExpansion, bool ownsTokens, bool isReinject, Preprocessor &pp)
	Create a TokenLexer for the specified token stream.
	TokenLexer (const TokenLexer &)=delete
TokenLexer &	operator= (const TokenLexer &)=delete
	~TokenLexer ()
void	Init (Token &Tok, SourceLocation ELEnd, MacroInfo *MI, MacroArgs *Actuals)
	Initialize this TokenLexer to expand from the specified macro with the specified argument information.
void	Init (const Token *TokArray, unsigned NumToks, bool DisableMacroExpansion, bool OwnsTokens, bool IsReinject)
	Initialize this TokenLexer with the specified token stream.
std::optional< Token >	peekNextPPToken () const
	If TokenLexer::isAtEnd returns true(the next token lexed will pop this macro off the expansion stack), return std::nullopt, otherwise return the next unexpanded token.
bool	Lex (Token &Tok)
	Lex and return a token from this macro stream.
bool	isParsingPreprocessorDirective () const
	isParsingPreprocessorDirective - Return true if we are in the middle of a preprocessor directive.
void	setLexingCXXModuleDirective (bool Val=true)
	setLexingCXXModuleDirective - This is set to true if this TokenLexer is created when handling a C++ module directive.
bool	isLexingCXXModuleDirective () const
	isLexingCXXModuleDirective - Return true if we are lexing a C++ module or import directive.

Friends
class	Preprocessor

Detailed Description

TokenLexer - This implements a lexer that returns tokens from a macro body or token stream instead of lexing from a character buffer.

This is used for macro expansion and _Pragma handling, for example.

Definition at line 30 of file TokenLexer.h.

Constructor & Destructor Documentation

TokenLexer() [1/3]

clang::TokenLexer::TokenLexer ( Token & Tok, SourceLocation ILEnd, MacroInfo * MI, MacroArgs * ActualArgs, Preprocessor & pp )

inline

Create a TokenLexer for the specified macro with the specified actual arguments.

Note that this ctor takes ownership of the ActualArgs pointer. ILEnd specifies the location of the ')' for a function-like macro or the identifier for an object-like macro.

Definition at line 112 of file TokenLexer.h.

References false, clang::Init, Preprocessor, and Tok.

Referenced by operator=(), and TokenLexer().

TokenLexer() [2/3]

clang::TokenLexer::TokenLexer ( const Token * TokArray, unsigned NumToks, bool DisableExpansion, bool ownsTokens, bool isReinject, Preprocessor & pp )

inline

Create a TokenLexer for the specified token stream.

If 'OwnsTokens' is specified, this takes ownership of the tokens and delete[]'s them when the token lexer is empty.

Definition at line 121 of file TokenLexer.h.

References false, clang::Init, and Preprocessor.

TokenLexer() [3/3]

clang::TokenLexer::TokenLexer ( const TokenLexer & )

delete

References TokenLexer().

~TokenLexer()

clang::TokenLexer::~TokenLexer ( )

inline

Definition at line 129 of file TokenLexer.h.

Member Function Documentation

Init() [1/2]

void TokenLexer::Init	(	const Token *	TokArray,
		unsigned	NumToks,
		bool	disableMacroExpansion,
		bool	ownsTokens,
		bool	isReinject )

Initialize this TokenLexer with the specified token stream.

Create a TokenLexer for the specified token stream.

This does not take ownership of the specified token vector.

DisableExpansion is true when macro expansion of tokens lexed from this stream should be disabled.

This does not take ownership of the specified token vector.

Definition at line 95 of file TokenLexer.cpp.

References clang::Token::hasLeadingSpace(), and clang::Token::isAtStartOfLine().

Init() [2/2]

void TokenLexer::Init	(	Token &	Tok,
		SourceLocation	ELEnd,
		MacroInfo *	MI,
		MacroArgs *	Actuals )

Initialize this TokenLexer to expand from the specified macro with the specified argument information.

Create a TokenLexer for the specified macro with the specified actual arguments.

Note that this ctor takes ownership of the ActualArgs pointer. ILEnd specifies the location of the ')' for a function-like macro or the identifier for an object-like macro.

Note that this ctor takes ownership of the ActualArgs pointer.

Definition at line 39 of file TokenLexer.cpp.

References is(), SM, and Tok.

isLexingCXXModuleDirective()

bool TokenLexer::isLexingCXXModuleDirective

(

)

const

isLexingCXXModuleDirective - Return true if we are lexing a C++ module or import directive.

Definition at line 972 of file TokenLexer.cpp.

Referenced by Lex().

isParsingPreprocessorDirective()

bool TokenLexer::isParsingPreprocessorDirective

(

)

const

isParsingPreprocessorDirective - Return true if we are in the middle of a preprocessor directive.

Definition at line 960 of file TokenLexer.cpp.

Lex()

bool TokenLexer::Lex

(

Token &

Tok

)

Lex and return a token from this macro stream.

Lex - Lex and return a token from this macro stream.

Definition at line 621 of file TokenLexer.cpp.

References clang::IdentifierInfo::getTokenID(), is(), clang::IdentifierInfo::isHandleIdentifierCase(), isLexingCXXModuleDirective(), clang::IdentifierInfo::isPoisoned(), clang::Token::IsReinjected, isWideStringLiteralFromMacro(), clang::Token::LeadingEmptyMacro, clang::Token::LeadingSpace, setLexingCXXModuleDirective(), SM, clang::Token::StartOfLine, and Tok.

operator=()

TokenLexer & clang::TokenLexer::operator= ( const TokenLexer & )

delete

References TokenLexer().

peekNextPPToken()

std::optional< Token > TokenLexer::peekNextPPToken

(

)

const

If TokenLexer::isAtEnd returns true(the next token lexed will pop this macro off the expansion stack), return std::nullopt, otherwise return the next unexpanded token.

isNextTokenLParen - If the next token lexed will pop this macro off the expansion stack, return std::nullopt, otherwise return the next unexpanded token.

Definition at line 951 of file TokenLexer.cpp.

setLexingCXXModuleDirective()

void TokenLexer::setLexingCXXModuleDirective

(

bool

Val = true

)

setLexingCXXModuleDirective - This is set to true if this TokenLexer is created when handling a C++ module directive.

setLexingCXXModuleDirective - This is set to true if this TokenLexer is created when handling C++ module directive.

Definition at line 966 of file TokenLexer.cpp.

Referenced by Lex().

Friends And Related Symbol Documentation

Preprocessor

friend class Preprocessor

friend

Definition at line 31 of file TokenLexer.h.

References Preprocessor.

Referenced by Preprocessor, TokenLexer(), and TokenLexer().

---

The documentation for this class was generated from the following files:

• include/clang/Lex/TokenLexer.h
• lib/Lex/TokenLexer.cpp