mtmd_tokenize_from_parts per-segment tokenization for marked input

Facts

• `mtmd_input_part` holds a `const mtmd_input_text *` and a `const mtmd_bitmap *`, with the comment that only one of them can be set. [source]
• The header documents two uses for `mtmd_tokenize_from_parts`: when you do not want media markers (markers are tokenized as normal text) and when you want to control `parse_special` for each text part. [source]
• The header says the per-part `add_special` is ignored; the single `add_special` argument of the function applies. [source]
• `mtmd_tokenize_from_parts` returns 1 if a part has both text and bitmap set or neither, or if a text part's text pointer is null, and returns 2 when the tokenizer throws. [source]
• The parts constructor copies each text part with its own `parse_special` and sets the tokenizer-level `parse_special` to true, used only for text returned by lazy bitmaps. [source]
• `mtmd_tokenize` returns 0 on success, 1 when the number of bitmaps does not match the number of markers, and 2 on media preprocessing error; the marker-count mismatch is a thrown `number of media markers in text (%zu) does not match number of bitmaps (%zu)`. [source]
• A lazy bitmap's callback is called with an increasing chunk index until it returns -1 (EOF) or -2 (error); it may return a bitmap or a heap-allocated text string, not both, and not another lazy bitmap. [source]
• `tokenize` computes `n_merge_frames` from `clip_model_n_temporal_merge`, asserts it is at most 2, and groups bitmaps flagged mergeable with `mtmd_group_mergeable_bitmaps` before adding media. [source]
• When `add_special` is true and the vocab adds BOS, BOS goes at the start of the first text chunk, or a new BOS-only text chunk is inserted at the front when the first chunk is media; the same block checks the vocab's add-EOS flag. [source]
• An `mtmd_bitmap` can carry an optional ID for KV cache tracking, and a lazy bitmap is meant to be tracked by one ID such as the file hash. [source]