Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It's true that the last layer's output for a given input token only affects the corresponding output token and is discarded afterwards. But the penultimate layer's output affects the computation of the last layer for all future tokens, so it is not discarded, but stored (in the KV cache). Similarly for the antepenultimate layer affecting the penultimate layer and so on.

So there's plenty of space in intermediate layers to store a plan between tokens without starting from scratch every time.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: