CORE results

1. \mathbf{x}_t = \mathrm{Embedding}(s_t)
2. \mathbf{H}_t = \mathbf{H}_{t-1}\,\mathrm{diag}(\boldsymbol{\alpha}_t) + \mathrm{diag}(\boldsymbol{\beta}_t)\,\mathbf{v}_t\,\mathbf{k}_t^{\top}
3. \boldsymbol{\alpha}_t = \sigma(\mathbf{W}^{\alpha}\mathbf{x}_t)
4. \mathbf{k}_t = \phi(\mathbf{W}^{k}\mathbf{x}_t)
5. \mathbf{y}_t = \mathbf{H}_t\,\mathbf{q}_t
6. \mathbf{q}_t = \phi(\mathbf{W}^{q}\mathbf{x}_t)
7. \mathbf{v}_t = \mathbf{x}_t - \mathbf{H}_{t-1}\,\mathrm{diag}(\boldsymbol{\alpha}_t)\,\mathbf{k}_t
8. \boldsymbol{\beta}_t = \sigma(\mathbf{W}^{\beta}\mathbf{x}_t)
9. \mathbf{x}_t^{(\ell+1)} = \mathbf{x}_t^{(\ell)} + \mathbf{y}_t^{(\ell)}
10. p(r_{t+1}) = \mathrm{softmax}\bigl(\mathbf{W}^{p}\mathbf{x}_t^{(L+1)}\bigr)

1. \mathbf{v}_t = \mathbf{x}_t

Everything else as in CORE.

1. s_t \in \text{BPE tokens}

Everything else as in CORE.

1. \mathbf{H}_t = \mathbf{H}_{t-1} + \mathrm{diag}(\boldsymbol{\beta}_t)\,\mathbf{v}_t\,\mathbf{k}_t^{\top}
2. \mathbf{v}_t = \mathbf{x}_t - \mathbf{H}_{t-1}\mathbf{k}_t

Everything else as in CORE.

1. L = 1

Everything else as in CORE.

1. \mathbf{x}_t^{(1)} = \mathrm{Embedding}(s_t)
2. \mathbf{q}_t = \mathbf{R}_t\,\mathbf{W}^{q}\,\mathrm{RMSNorm}(\mathbf{x}_t^{(\ell)})
3. \mathbf{k}_t = \mathbf{R}_t\,\mathbf{W}^{k}\,\mathrm{RMSNorm}(\mathbf{x}_t^{(\ell)})
4. \mathbf{v}_t = \mathbf{W}^{v}\,\mathrm{RMSNorm}(\mathbf{x}_t^{(\ell)})
5. \mathbf{y}_t = \sum_{i \le t} \mathrm{softmax}_i\!\left(\frac{\mathbf{q}_t^{\top}\mathbf{k}_i}{\sqrt{d_h}}\right)\mathbf{v}_i
6. \mathbf{h}_t^{(\ell)} = \mathbf{x}_t^{(\ell)} + \mathbf{W}^{o}\,\mathbf{y}_t
7. \mathbf{x}_t^{(\ell+1)} = \mathbf{h}_t^{(\ell)} + \mathrm{MLP}\bigl(\mathrm{RMSNorm}(\mathbf{h}_t^{(\ell)})\bigr)
8. \mathrm{MLP}(\mathbf{z}) = \mathbf{W}^{\mathrm{down}}\bigl(\mathrm{SiLU}(\mathbf{W}^{\mathrm{gate}}\mathbf{z}) \odot \mathbf{W}^{\mathrm{up}}\mathbf{z}\bigr)
9. p(r_{t+1}) = \mathrm{softmax}\bigl(\mathbf{W}^{p}\,\mathrm{RMSNorm}(\mathbf{x}_t^{(L+1)})\bigr)