ByteDance study finds that asking LMMs questions beats making it transcribe text for long document training

The researchers tested both approaches head-to-head. In one setup, the model had to perform text recognition either across all pages of a document or for a few selected pages, while the remaining pages stayed in context as distractions.

In the other setup, the researchers used a separate model (Seed 2.0 from ByteDance) to generate question-answer pairs for individual sections of a document. The question then went into training alongside the entire document, forcing the model to locate the relevant passage within a long context.

Pure text recognition as a training task actually worsened performance compared to the starting point. Question-answer training, on the other hand, brought clear gains. The model only learns to navigate long texts when it has to filter out and categorize information with a specific goal.

Diversity beats specialization

Three more findings turned up in the experiments. Feeding the model mainly very long documents at the top end of the context window isn't worth it. A broader mix of shorter and longer examples works more reliably. Long-context ability isn't a skill tied to a specific length but requires flexible searching across different distances.

The real bottleneck also turns out to be finding the relevant passage, not reasoning about it. A mix weighted toward extraction tasks with a smaller share of calculation tasks delivered the best results.

The third finding is surprising because it contradicts common practice with text-only language models. Adding short training examples doesn't appear strictly necessary. The model largely kept its short-task abilities even when trained only on long question-answer data. The format of the data itself probably helps: even when the context is very long, the task is still framed as a question-answer interaction in the familiar instruction-following format.

Small but stable up to 512,000 tokens

With this recipe and a fairly modest training budget, MMProLong beats several much larger open models like InternVL3-38B and Gemma3-27B. The model was trained on only 128,000 tokens but stays stable at 256,000 and even 512,000 token input lengths, while the original model falls apart sharply at those ranges.

This ability also transfers to tasks the model was never specifically trained on, like understanding long videos. In an extra transfer experiment, the recipe proved effective on the stronger Qwen3-VL-8B too, even though that model is already built for long contexts.

The study is also interesting because it comes from an entirely different camp than Deepseek's widely discussed work on the same problem. Deepseek tries to extend the long memory of AI models by processing texts as images and compressing them heavily, most recently with an encoder that re-sorts visual information by content. ByteDance Seed takes the opposite approach: optimize the training data instead of the architecture.