Function rustc_lexer::tokenize

pub fn tokenize(input: &str) -> impl Iterator<Item = Token> + '_

Creates an iterator that produces tokens from the input string.