When I say models will plateau I don't mean there will be no progress. I mean progress will slow down since we'll be scraping the bottom of the barrel for training data. We might never quite run out but once we've sampled every novel, web site, scientific paper, chat log, broadcast transcript, and so on, we've exhausted the rich sources for easy gains.
Chat logs don’t run out. We may run out of novelty in those logs, at which point we may have ran out of human knowledge.
Or not - there still knowledge in people heads that is not bleeding into ai chat.
One implication here is that chats will morph to elicit more conversation to keep mining that mine. Which may lead to the need to enrage users to keep engagement.
