tokenize

@safe @trusted
tokenize
(
Flag!"splitMultiLineTokens" splitTokens = No.splitMultiLineTokens
)
(
ref from.libclang_ast.context.ClangContext ctx
,)

Return Value

Type: auto

the file content as an array of tokens.

This is a bit slow, I think. Optimize by reducing the created strings. trusted: none of the unsafe accessed data escape this function.

Meta