Token - This structure provides full information about a lexed token. More...

#include "clang/Lex/Token.h"

Public Types
enum	TokenFlags { StartOfLine = 0x01 , LeadingSpace = 0x02 , DisableExpand = 0x04 , NeedsCleaning = 0x08 , LeadingEmptyMacro = 0x10 , HasUDSuffix = 0x20 , HasUCN = 0x40 , IgnoredComma = 0x80 , StringifiedInMacro = 0x100 , CommaAfterElided = 0x200 , IsEditorPlaceholder = 0x400 , IsReinjected = 0x800 }

Public Member Functions
tok::TokenKind	getKind () const

void	setKind (tok::TokenKind K)

bool	is (tok::TokenKind K) const
	is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

bool	isNot (tok::TokenKind K) const

bool	isOneOf (tok::TokenKind K1, tok::TokenKind K2) const

template<typename... Ts>
bool	isOneOf (tok::TokenKind K1, Ts... Ks) const

bool	isAnyIdentifier () const
	Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).

bool	isLiteral () const
	Return true if this is a "literal", like a numeric constant, string, etc.

bool	isAnnotation () const
	Return true if this is any of tok::annot_* kind tokens.

bool	isRegularKeywordAttribute () const
	Return true if the token is a keyword that is parsed in the same position as a standard attribute, but that has semantic meaning and so cannot be a true attribute.

SourceLocation	getLocation () const
	Return a source location identifier for the specified offset in the current file.

unsigned	getLength () const

void	setLocation (SourceLocation L)

void	setLength (unsigned Len)

SourceLocation	getAnnotationEndLoc () const

void	setAnnotationEndLoc (SourceLocation L)

SourceLocation	getLastLoc () const

SourceLocation	getEndLoc () const

SourceRange	getAnnotationRange () const
	SourceRange of the group of tokens that this annotation token represents.

void	setAnnotationRange (SourceRange R)

const char *	getName () const

void	startToken ()
	Reset all flags to cleared.

bool	hasPtrData () const

IdentifierInfo *	getIdentifierInfo () const

void	setIdentifierInfo (IdentifierInfo *II)

const void *	getEofData () const

void	setEofData (const void *D)

StringRef	getRawIdentifier () const
	getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.

void	setRawIdentifierData (const char *Ptr)

const char *	getLiteralData () const
	getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

void	setLiteralData (const char *Ptr)

void *	getAnnotationValue () const

void	setAnnotationValue (void *val)

void	setFlag (TokenFlags Flag)
	Set the specified flag.

bool	getFlag (TokenFlags Flag) const
	Get the specified flag.

void	clearFlag (TokenFlags Flag)
	Unset the specified flag.

unsigned	getFlags () const
	Return the internal represtation of the flags.

void	setFlagValue (TokenFlags Flag, bool Val)
	Set a flag to either true or false.

bool	isAtStartOfLine () const
	isAtStartOfLine - Return true if this token is at the start of a line.

bool	hasLeadingSpace () const
	Return true if this token has whitespace before it.

bool	isExpandDisabled () const
	Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.

bool	isObjCAtKeyword (tok::ObjCKeywordKind objcKey) const
	Return true if we have an ObjC keyword identifier.

tok::ObjCKeywordKind	getObjCKeywordID () const
	Return the ObjC keyword kind.

bool	isSimpleTypeSpecifier (const LangOptions &LangOpts) const
	Determine whether the token kind starts a simple-type-specifier.

bool	needsCleaning () const
	Return true if this token has trigraphs or escaped newlines in it.

bool	hasLeadingEmptyMacro () const
	Return true if this token has an empty macro before it.

bool	hasUDSuffix () const
	Return true if this token is a string or character literal which has a ud-suffix.

bool	hasUCN () const
	Returns true if this token contains a universal character name.

bool	stringifiedInMacro () const
	Returns true if this token is formed by macro by stringizing or charizing operator.

bool	commaAfterElided () const
	Returns true if the comma after this token was elided.

bool	isEditorPlaceholder () const
	Returns true if this token is an editor placeholder.

Detailed Description

Token - This structure provides full information about a lexed token.

It is not intended to be space efficient, it is intended to return as much information as possible about each returned token. This is expected to be compressed into a smaller form if memory footprint is important.

The parser can create a special "annotation token" representing a stream of tokens that were parsed and semantically resolved, e.g.: "foo::MyClass<int>" can be represented by a single typename annotation token that carries information about the SourceRange of the tokens and the type object.

Definition at line 36 of file Token.h.

Member Enumeration Documentation

◆ TokenFlags

enum clang::Token::TokenFlags

Enumerator
StartOfLine
LeadingSpace
DisableExpand
NeedsCleaning
LeadingEmptyMacro
HasUDSuffix
HasUCN
IgnoredComma
StringifiedInMacro
CommaAfterElided
IsEditorPlaceholder
IsReinjected

Definition at line 74 of file Token.h.

Member Function Documentation

◆ clearFlag()

void clang::Token::clearFlag ( TokenFlags Flag )

inline

Unset the specified flag.

Definition at line 254 of file Token.h.

Referenced by setFlagValue().

◆ commaAfterElided()

bool clang::Token::commaAfterElided ( ) const

inline

Returns true if the comma after this token was elided.

Definition at line 313 of file Token.h.

References CommaAfterElided, and getFlag().

◆ getAnnotationEndLoc()

SourceLocation clang::Token::getAnnotationEndLoc ( ) const

inline

Definition at line 146 of file Token.h.

References clang::SourceLocation::getFromRawEncoding(), isAnnotation(), and Loc.

Referenced by clang::ASTWriter::AddToken(), getAnnotationRange(), getEndLoc(), and getLastLoc().

◆ getAnnotationRange()

SourceRange clang::Token::getAnnotationRange ( ) const

inline

SourceRange of the group of tokens that this annotation token represents.

Definition at line 166 of file Token.h.

References getAnnotationEndLoc(), and getLocation().

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

◆ getAnnotationValue()

void * clang::Token::getAnnotationValue ( ) const

inline

Definition at line 234 of file Token.h.

References isAnnotation().

Referenced by clang::ASTWriter::AddToken(), clang::Parser::getTypeAnnotation(), clang::Parser::ParseTopLevelDecl(), and PrintPreprocessedTokens().

◆ getEndLoc()

SourceLocation clang::Token::getEndLoc ( ) const

inline

Definition at line 159 of file Token.h.

References getAnnotationEndLoc(), getLength(), getLocation(), clang::SourceLocation::getLocWithOffset(), and isAnnotation().

Referenced by fixSeparateAttrArgAndNumber(), getEntityEndLoc(), parseDeclareSimdClauses(), and clang::format::IntegerLiteralSeparatorFixer::process().

◆ getEofData()

const void * clang::Token::getEofData ( ) const

inline

Definition at line 200 of file Token.h.

References is().

◆ getFlag()

bool clang::Token::getFlag ( TokenFlags Flag ) const

inline

Get the specified flag.

Definition at line 249 of file Token.h.

Referenced by commaAfterElided(), hasLeadingEmptyMacro(), hasLeadingSpace(), hasUCN(), hasUDSuffix(), isAtStartOfLine(), isEditorPlaceholder(), isExpandDisabled(), needsCleaning(), and stringifiedInMacro().

◆ getFlags()

unsigned clang::Token::getFlags ( ) const

inline

Return the internal represtation of the flags.

This is only intended for low-level operations such as writing tokens to disk.

Definition at line 262 of file Token.h.

Referenced by clang::ASTWriter::AddToken().

◆ getIdentifierInfo()

IdentifierInfo * clang::Token::getIdentifierInfo ( ) const

inline

Definition at line 187 of file Token.h.

References is(), isAnnotation(), isLiteral(), and isNot().

Referenced by clang::Sema::ActOnPragmaUnused(), AddDefinedMacroToHash(), clang::ASTWriter::AddToken(), clang::MacroArgs::ArgNeedsPreexpansion(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::CheckMacroName(), diagnoseAutoModuleImport(), dumpTokenInto(), EvaluateDefined(), EvaluateFeatureLikeBuiltinMacro(), EvaluateValue(), ExpectFeatureIdentifierInfo(), GetFirstChar(), getIdentifier(), getObjCKeywordID(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpelling(), clang::Preprocessor::HandleDirective(), clang::PragmaNamespace::HandlePragma(), isConfigurationPattern(), clang::format::isFunctionDeclarationName(), clang::MacroInfo::isIdenticalTo(), clang::format::AdditionalKeywords::isJavaScriptIdentifier(), isObjCAtKeyword(), isSameToken(), isSimpleTypeSpecifier(), isTrivialSingleTokenExpansion(), isWideStringLiteralFromMacro(), clang::format::AdditionalKeywords::isWordLike(), clang::TokenLexer::Lex(), clang::Preprocessor::LexEmbedParameters(), LexModuleNameComponent(), clang::Preprocessor::LexOnOffSwitch(), clang::MacroPPCallbacks::MacroDefined(), clang::detail::MacroExpansionRangeRecorder::MacroExpands(), clang::MacroPPCallbacks::MacroUndefined(), clang::format::mustBeJSIdent(), clang::TokenValue::operator==(), ParseAlignPragma(), parseDeclareSimdClauses(), clang::Parser::parseMapperModifier(), clang::Parser::parseMapTypeModifiers(), clang::Parser::ParseMSAsmIdentifier(), parseOpenMPReductionId(), clang::Parser::ParseOpenMPReservedLocator(), clang::Parser::ParseOpenMPVarList(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PragmaLoopHintString(), PrintPreprocessedTokens(), clang::Parser::TryAnnotateTypeOrScopeToken(), clang::Parser::TryAnnotateTypeOrScopeTokenAfterScopeSpec(), and clang::Preprocessor::TypoCorrectToken().

◆ getKind()

tok::TokenKind clang::Token::getKind ( ) const

inline

Definition at line 94 of file Token.h.

Referenced by clang::Sema::ActOnCharacterConstant(), clang::Sema::ActOnNumericConstant(), clang::ASTWriter::AddToken(), atOrBeforeSeparation(), clang::TokenConcatenation::AvoidConcat(), clang::Sema::CheckConstraintExpression(), clang::Preprocessor::CheckMacroName(), clang::Lexer::ComputePreamble(), contains(), clang::Preprocessor::DumpToken(), EvaluateDirectiveSubExpr(), EvaluateFeatureLikeBuiltinMacro(), EvaluateHasIncludeCommon(), EvaluateValue(), getBeginningOfFileToken(), getEntityEndLoc(), getIdentifier(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), clang::Preprocessor::HandlePragmaModuleBuild(), isAnnotation(), isAnyIdentifier(), isConfigurationPattern(), clang::format::isConfiguredQualifier(), clang::MacroInfo::isIdenticalTo(), clang::format::AdditionalKeywords::isJavaScriptIdentifier(), isLiteral(), clang::format::FormatToken::isObjCAtKeyword(), clang::Preprocessor::IsPreviousCachedToken(), clang::format::isQualifier(), isRegularKeywordAttribute(), isSameToken(), isSimpleTypeSpecifier(), clang::Preprocessor::LexEmbedParameters(), clang::TokenValue::operator==(), clang::Parser::ParseConstraintLogicalAndExpression(), parseOpenMPReductionId(), clang::Parser::ParseTopLevelDecl(), clang::Parser::ParseUnqualifiedId(), PrintPreprocessedTokens(), clang::ASTReader::ReadToken(), clang::CodeGen::CoverageMappingModuleGen::setUpCoverageCallbacks(), clang::format::ShouldBreakBeforeBrace(), clang::Parser::SkipMalformedDecl(), clang::Parser::SkipUntil(), clang::MacroArgs::StringifyArgument(), and clang::tokenIsLikeStringLiteral().

◆ getLastLoc()

SourceLocation clang::Token::getLastLoc ( ) const

inline

Definition at line 155 of file Token.h.

References getAnnotationEndLoc(), getLocation(), and isAnnotation().

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

◆ getLength()

unsigned clang::Token::getLength ( ) const

inline

Definition at line 135 of file Token.h.

References isAnnotation().

Referenced by clang::Sema::ActOnNumericConstant(), clang::ASTWriter::AddToken(), clang::TokenConcatenation::AvoidConcat(), clang::Preprocessor::DumpToken(), dumpTokenInto(), EmitAlphaCounter(), fixSeparateAttrArgAndNumber(), getBeginningOfFileToken(), getEndLoc(), GetFirstChar(), GetLineValue(), clang::StringLiteralParser::getOffsetOfStringByte(), getRawIdentifier(), clang::Preprocessor::getSpelling(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), getSpellingSlow(), clang::TokenLexer::Lex(), clang::Preprocessor::LexHeaderName(), clang::detail::MacroExpansionRangeRecorder::MacroExpands(), clang::Lexer::MeasureTokenLength(), NormalizeLine(), PrintPreprocessedTokens(), clang::format::IntegerLiteralSeparatorFixer::process(), clang::RewriteMacrosInInput(), clang::MacroArgs::StringifyArgument(), and UseNSOptionsMacro().

◆ getLiteralData()

const char * clang::Token::getLiteralData ( ) const

inline

getLiteralData - For a literal token (numeric constant, string, etc), this returns a pointer to the start of it in the text buffer if known, null otherwise.

Definition at line 225 of file Token.h.

References isLiteral().

Referenced by dumpTokenInto(), fixSeparateAttrArgAndNumber(), GetFirstChar(), clang::Lexer::getSpelling(), clang::Preprocessor::getSpellingOfSingleCharacterNumericConstant(), PrintPreprocessedTokens(), and UseNSOptionsMacro().

◆ getLocation()

SourceLocation clang::Token::getLocation ( ) const

inline

Return a source location identifier for the specified offset in the current file.

Definition at line 132 of file Token.h.

References clang::SourceLocation::getFromRawEncoding(), and Loc.

◆ getName()

const char * clang::Token::getName ( ) const

inline

Definition at line 174 of file Token.h.

References clang::tok::getTokenName().

◆ getObjCKeywordID()

tok::ObjCKeywordKind Token::getObjCKeywordID ( ) const

Return the ObjC keyword kind.

getObjCKeywordID - Return the ObjC keyword kind.

Definition at line 70 of file Lexer.cpp.

References getIdentifierInfo(), clang::IdentifierInfo::getObjCKeywordID(), and isAnnotation().

◆ getRawIdentifier()

StringRef clang::Token::getRawIdentifier ( ) const

inline

getRawIdentifier - For a raw identifier token (i.e., an identifier lexed in raw mode), returns a reference to the text substring in the buffer if known.

Definition at line 213 of file Token.h.

References getLength(), and is().

Referenced by clang::arcmt::trans::MigrationContext::addPropertyAttribute(), clang::Lexer::ComputePreamble(), clang::Lexer::getSpelling(), clang::Preprocessor::HandlePragmaModuleBuild(), LocPropertyAttribute(), and clang::arcmt::trans::MigrationContext::rewritePropertyAttribute().

◆ hasLeadingEmptyMacro()

bool clang::Token::hasLeadingEmptyMacro ( ) const

inline

Return true if this token has an empty macro before it.

Definition at line 299 of file Token.h.

References getFlag(), and LeadingEmptyMacro.

Referenced by clang::Preprocessor::LexHeaderName().

◆ hasLeadingSpace()

bool clang::Token::hasLeadingSpace ( ) const

inline

Return true if this token has whitespace before it.

Definition at line 280 of file Token.h.

References getFlag(), and LeadingSpace.

Referenced by buildMSAsmString(), clang::Preprocessor::DumpToken(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), clang::Preprocessor::LexHeaderName(), PrintMacroDefinition(), clang::RewriteMacrosInInput(), and clang::MacroArgs::StringifyArgument().

◆ hasPtrData()

bool clang::Token::hasPtrData ( ) const

inline

Definition at line 185 of file Token.h.

◆ hasUCN()

bool clang::Token::hasUCN ( ) const

inline

Returns true if this token contains a universal character name.

Definition at line 306 of file Token.h.

References getFlag(), and HasUCN.

Referenced by clang::Lexer::getSpelling(), and clang::Preprocessor::getSpelling().

◆ hasUDSuffix()

bool clang::Token::hasUDSuffix ( ) const

inline

Return true if this token is a string or character literal which has a ud-suffix.

Definition at line 303 of file Token.h.

References getFlag(), and HasUDSuffix.

Referenced by clang::TokenConcatenation::AvoidConcat(), EvaluateValue(), LexModuleNameComponent(), and clang::Preprocessor::ParsePragmaPushOrPopMacro().

◆ is()

bool clang::Token::is ( tok::TokenKind K ) const

inline

is/isNot - Predicates to check if this token is a specific kind, as in "if (Tok.is(tok::l_brace)) {...}".

Definition at line 99 of file Token.h.

◆ isAnnotation()

bool clang::Token::isAnnotation ( ) const

inline

Return true if this is any of tok::annot_* kind tokens.

Definition at line 121 of file Token.h.

References getKind(), and clang::tok::isAnnotation().

Referenced by clang::ASTWriter::AddToken(), clang::Preprocessor::AnnotateCachedTokens(), clang::TokenConcatenation::AvoidConcat(), clang::Parser::ConsumeAnyToken(), clang::Preprocessor::DumpToken(), dumpTokenInto(), ExpectFeatureIdentifierInfo(), getAnnotationEndLoc(), getAnnotationValue(), getEndLoc(), getIdentifierInfo(), getLastLoc(), getLength(), getObjCKeywordID(), isObjCAtKeyword(), clang::TokenLexer::Lex(), LexModuleNameComponent(), parseOpenMPDirectiveKind(), parseOpenMPSimpleClause(), clang::Parser::ParseOpenMPVarList(), PrintPreprocessedTokens(), clang::ASTReader::ReadToken(), clang::Preprocessor::ReplaceLastTokenWithAnnotation(), setAnnotationEndLoc(), setAnnotationValue(), setLength(), and clang::Parser::TryAnnotateTypeOrScopeToken().

◆ isAnyIdentifier()

bool clang::Token::isAnyIdentifier ( ) const

inline

Return true if this is a raw identifier (when lexing in raw mode) or a non-keyword identifier (when lexing in non-raw mode).

Definition at line 110 of file Token.h.

References getKind(), and clang::tok::isAnyIdentifier().

◆ isAtStartOfLine()

bool clang::Token::isAtStartOfLine ( ) const

inline

isAtStartOfLine - Return true if this token is at the start of a line.

Definition at line 276 of file Token.h.

References getFlag(), and StartOfLine.

Referenced by buildMSAsmString(), clang::Lexer::ComputePreamble(), clang::Preprocessor::DumpToken(), clang::Preprocessor::HandlePragmaModuleBuild(), clang::TokenLexer::Init(), clang::MacroInfo::isIdenticalTo(), clang::Preprocessor::LexHeaderName(), NormalizeLine(), PrintPreprocessedTokens(), clang::RewriteMacrosInInput(), clang::Parser::SkipMalformedDecl(), and clang::MacroArgs::StringifyArgument().

◆ isEditorPlaceholder()

bool clang::Token::isEditorPlaceholder ( ) const

inline

Returns true if this token is an editor placeholder.

Editor placeholders are produced by the code-completion engine and are represented as characters between '<#' and '#>' in the source code. The lexer uses identifier tokens to represent placeholders.

Definition at line 320 of file Token.h.

References getFlag(), and IsEditorPlaceholder.

Referenced by clang::Parser::TryAnnotateTypeOrScopeToken().

◆ isExpandDisabled()

bool clang::Token::isExpandDisabled ( ) const

inline

Return true if this identifier token should never be expanded in the future, due to C99 6.10.3.4p2.

Definition at line 284 of file Token.h.

References DisableExpand, and getFlag().

Referenced by clang::Preprocessor::DumpToken().

◆ isLiteral()

bool clang::Token::isLiteral ( ) const

inline

Return true if this is a "literal", like a numeric constant, string, etc.

Definition at line 116 of file Token.h.

References getKind(), and clang::tok::isLiteral().

Referenced by clang::Preprocessor::CreateString(), dumpTokenInto(), GetFirstChar(), getIdentifierInfo(), getLiteralData(), clang::Lexer::getSpelling(), isWideStringLiteralFromMacro(), clang::format::mustBeJSIdentOrValue(), PrintPreprocessedTokens(), setLiteralData(), and UseNSOptionsMacro().

◆ isNot()

bool clang::Token::isNot ( tok::TokenKind K ) const

inline

Definition at line 100 of file Token.h.

◆ isObjCAtKeyword()

bool Token::isObjCAtKeyword ( tok::ObjCKeywordKind objcKey ) const

Return true if we have an ObjC keyword identifier.

isObjCAtKeyword - Return true if we have an ObjC keyword identifier.

Definition at line 61 of file Lexer.cpp.

References getIdentifierInfo(), and isAnnotation().

◆ isOneOf() [1/2]

bool clang::Token::isOneOf	(	tok::TokenKind	K1,
		tok::TokenKind	K2
	)		const

inline

Definition at line 101 of file Token.h.

References is().

Referenced by clang::TokenConcatenation::AvoidConcat(), clang::Sema::ClassifyName(), EvaluateFeatureLikeBuiltinMacro(), isConfigurationPattern(), isEndOfTemplateArgument(), isMapType(), clang::Preprocessor::LexEmbedParameters(), clang::Preprocessor::LexHeaderName(), clang::Preprocessor::LexTokensUntilEOF(), clang::Parser::ParseConstraintLogicalAndExpression(), clang::format::CommaSeparatedList::precomputeFormattingInfos(), and clang::Parser::SkipMalformedDecl().

◆ isOneOf() [2/2]

template<typename... Ts>

bool clang::Token::isOneOf	(	tok::TokenKind	K1,
		Ts...	Ks
	)		const

inline

Definition at line 104 of file Token.h.

References is(), and isOneOf().

◆ isRegularKeywordAttribute()

bool clang::Token::isRegularKeywordAttribute ( ) const

inline

Return true if the token is a keyword that is parsed in the same position as a standard attribute, but that has semantic meaning and so cannot be a true attribute.

Definition at line 126 of file Token.h.

References getKind(), and clang::tok::isRegularKeywordAttribute().