Public/nushell
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add doc comment to lex_n_tokens

Browse Source
This commit is contained in:
Kira 2024-08-02 23:26:07 +02:00
parent b3bdd4ac64
commit b26dd50006
1 changed files with 5 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
crates/nu-parser/src/lex.rs
View File

@ -472,9 +472,11 @@ pub struct LexState<'a> {
pub span_offset: usize,
}
// Lex until the output is max_tokens longer than before the call, or until the input is exhausted.
// The behaviour here is non-obvious (maybe non-useful) iff your additional_whitespace doesn't include newline:
// If you pass `output` in a state where the last token is an Eol, this might *remove* tokens.
/// Lex until the output is `max_tokens` longer than before the call, or until the input is exhausted.
/// The return value indicates how many tokens the call added to / removed from the output.
///
/// The behaviour here is non-obvious when `additional_whitespace` doesn't include newline:
/// If you pass a `state` where the last token in the output is an Eol, this might *remove* tokens.
pub fn lex_n_tokens(
state: &mut LexState,
additional_whitespace: &[u8],