<aside> 💡

Authors: Haonan He$^$, Haodi Lei$^$, Yun Luo$^{\diamond \ \dagger}$, Haoran Zhang, Shuankai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu$^\dagger$, Yafu Li$^\dagger$, Peng Ye$^\dagger$, Ning Ding$^\dagger$, Yu Cheng$^\dagger$,

$^$Co-First Authors.* **$^\dagger$**Corresponding Authors. $^{\diamond}$Project Lead.

SU-01 Team, Shanghai AI Lab.

</aside>

<aside> 💡

TL;DR：

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, yet facing challenges including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study a setting where OPD transfers proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. We observe that naive OPD leads to excessive generation length and frequent truncation, which destabilizes training and hurts model performance. To mitigate this, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem, and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2-Preview, GLM-4.7-Flash and Gemma-4-26B-A4B, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

</aside>

<aside> 💡

Takeaway

Cross-tokenizer on-policy distillation can improve student models without full token-level alignment by using student-generated text as the shared space and aligning only tokens that occupy identical text spans under the student and teacher tokenizers.
Direct cross-tokenizer OPD training transfers long-context proof-reasoning ability but often causes excessive generation-length growth, frequent truncation, and unstable training.
Masking the advantages of special termination tokens and adding a student reference KL loss could stabilize training, reduces truncation, and improves final performance. When the teacher and student models differ more substantially, a larger student reference KL weight is needed to better constrain training.
SimpleOPD could work well across different model families without supervised fine-tuning on the teacher trajectory, and a moderate increase in distillation length can further benefit long-reasoning distillation. </aside>

<aside> 💡

Citation

@online{SU-01-OPD-2026,
  title = {SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning},
  author = {SU-01 Team},
  year = {2026},
  month = July,
  url = {<https://relic-care-6c0.notion.site/SimpleOPD-Simple-Tokenizer-Agnostic-On-Policy-Distillation-for-Long-Context-Reasoning-38ec37a600c080919c4bd481ee8305ff?source=copy_link>}
}

We will release the code and the arXiv version soon.

</aside>

1. Cross-tokenizer On-policy Distillation

1.1 Setup

Let $\mathbf{x}$ denote the input conversation represented as a list of messages. The student and teacher with different tokenizers may use different chat templates, denoted by $\mathcal C_\theta$ and $\mathcal C_\phi$, respectively. We first construct the student input context text $c_\theta=\mathcal C_\theta(\mathbf{x})$ and sample a response-token sequence with the student policy executed by the rollout engine $\pi_{\theta_{\mathrm{roll}}}$.

$$ y_{1:n}=(y_1,\ldots,y_n) \sim \pi_{\theta_{\mathrm{roll}}}(\cdot\mid c_\theta). $$

Decoding $y_{1:n}$ via the tokenizer decoder of the student gives the response surface string $s=\mathcal D_\theta(y_{1:n})$. Instead of passing the student context to the teacher, we reconstruct the teacher context text $c_\phi$ using its own chat template and append the student response:

$$ c_\phi=\mathcal C_\phi(\mathbf{x}), \quad u_\phi=c_\phi\oplus s. $$

where $\oplus$ denotes string concatenation. The complete text $u_\phi$ is then provided to the teacher, which tokenizes it using the tokenizer encoder$\mathcal E_\phi$of the teacher. Let $\mathcal E_\phi(s|c_\phi)=(z_1,\ldots,z_m)$ denote token list by encoding the student response given the input context $c_\phi$using the teacher tokenizer. This procedure allows the teacher to evaluate the student response under its native tokenizer and chat-template. Although $c_\theta$ and $c_\phi$ may differ, the response text being evaluated is identical on both sides. For token alignment, let $\tau_\theta(y_t)$ and $\tau_\phi(z_i)$ denote the incremental text spans contributed by the corresponding response tokens. They satisfy

$$ \bigoplus_{t=1}^{n}\tau_\theta(y_t)

\bigoplus_{i=1}^{m}\tau_\phi(z_i)

s. $$

We perform no additional cleanup or normalization, so these spans define consistent offsets in the shared response string. The teacher log-probability associated with token $z_i$ is

$$ \log\pi_\phi(z_i\mid c_\phi,z_{<i}). $$

1.2 Cross-tokenizer Alignment

Define the cumulative response text preceding each student and teacher token as

$$ P_\theta(t)=\bigoplus_{k=1}^{t-1}\tau_\theta(y_k),\qquad P_\phi(i)=\bigoplus_{k=1}^{i-1}\tau_\phi(z_k), $$

with $P_\theta(1)=P_\phi(1)=\varepsilon$, where $\varepsilon$ denotes the empty string. A teacher position $i$ is aligned with a student position $t$ if both tokenizations have consumed the same response prefix and the current tokens contribute the same text span: