But you need to pass data from the access to the stream_filter, from the stream_filter to the demuxer, from the demuxer to several decoders, from the decoders to potentially a few video-filters and chroma-converters, and then finally to the output. Each of them need different access policies, and several of them require FS access.
The last part is just one of the issues, very far from all of them.
Seriously, stop thinking that noone has given a thought to the question...
These shouldn't require IPC at GB/s speed either. Modern sandboxes, like the one in Chrome, have a broker process which can open filesystem objects, device objects and sockets (file descriptors or handles) and pass them to a sandboxed decoder/renderer process, so there would be no need to stream filesystem data to the sandbox when the sandbox could do the file I/O itself. Even for Matroska ordered chapters, where the demuxer would have to tell the broker which files to open, the broker could enforce certain rules, such as enforcing that local mkv files only reference other local files, the files are all in the same directory, and that the files are always opened in read-only mode.
As for isolating decoders from video filters and chroma conversion, I'm not sure why that would be necessary, since those shouldn't require any additional privileges. I understand that retrofitting an existing program to use a multi-process sandboxing model is far from easy, and I'm definitely not volunteering to do it, but I don't think there is anything specific about a video player that is harder to sandbox than a web browser.
> I understand that retrofitting an existing program to use a multi-process sandboxing model is far from easy, and I'm definitely not volunteering to do it,
I don't think nobody has thought about it, but since you were apparently unaware that there was an alternative to performing several GB/s of IPC for moving buffers around there's obviously some options that haven't been considered. The Chromium sandbox has to deal with every issue you've listed (it's even calibrated to run ffmpeg inside the sandbox, since that's something Chromium needs to do).
It's not an attack. Each platform's methods of GPU IPC are pretty sparsely documented. Two months ago I wouldn't have known about them; I only learned by working on integrating Chromium's sandbox into an application that needed to work with the GPU within a sandboxed process.
That doesn't change the fact that none of the things you listed are unsupported by Chrome's sandbox model, and if you only need to establish a barrier around the video pipeline (and not e.g. VLC's ability to notice device status or interact with webcams) you don't even need 3/4 of what Chromium's sandbox has implemented. Like I said, I've actually walked the walk when it comes to using their sandbox for Windows and Linux with a process that needed to access certain user files, the GPU, and even each platform's font server equivalent, so this isn't me just spitballing about some theoretical solution.
The last part is just one of the issues, very far from all of them.
Seriously, stop thinking that noone has given a thought to the question...