This is why I drifted towards game development for most of my career. Consoles, until the penultimate (antipenultimate?) generation, ran software bare or nearly bare on the host machine.
I also spent time in integrate display controller development and such; it was all very similar.
Nowadays it feels like everything rides on top some ugly and opaque stack.
For context of what this guy is saying: the modern Xbox's (after 360) are actually running VMs for each game. This is part of why despite the hardware being technically (marginally) superior: Xbox tends to have lower graphical fidelity.
The 360 has a Type 1 (bare metal) hypervisor. So there's not much, if any, performance impact to having it since the software runs natively on the hardware.
Microsoft used a hypervisor primarily for security. They wanted to ensure that only signed code could be executed and wanted to prevent an exploit in a game from allowing the execution of unsigned code with kernel privileges on the system itself.
Every ounce of performance lost to the hypervisor is money Microsoft wasted in hardware costs. So they had an incentive to make the HyperV as performant as possible.
The CPU had an emulator that you could run on x86 Windows, but it was not itself a hypervisor.
The hypervisor in the XB1 served a more important purpose: to provide developers a way of shipping the custom SDK to clients, and not forcing them to update it. This was quite important for software stability and in fact we made a few patches to MS's XB1 SDK (Durango) to optimise it for our games.
VM's are VM's, there are performance trade-offs.
I know this because I worked on AAA games before in this area, do you also work in games and are repeating something you think. you heard?
This detailed article on the above exploit goes into detail on how the memory model works on the XB360, including how main memory addressing works differently in hypervisor mode than in real mode:
Yes, the 128KB of key storage and W^X. That's not a hypervisor in the sense that the XB1/HyperV or VMWare have a hypervisor, they shouldn't even share a name it's not the same thing at all.
It's like calling the JVM is a virtual machine in the same way QEMU is.
The 360 "Hypervisor" is more akin to a software T2 chip than anything that actually virtualises.
I don’t think you are showing respect when you simplistically repeat your assertion without effort, after two people expended their precious time to tell you in detail that you are wrong with examples. I don’t know anything, but a few minutes following the provided links and I find https://cxsecurity.com/issue/WLB-2007030065 which says:
The Xbox 360 security system is designed around a hypervisor concept. All games and other applications, which must be cryptographically signed with Microsoft's private key, run in non-privileged mode, while only a small hypervisor runs in privileged ("hypervisor") mode. The hypervisor controls access to memory and provides encryption and decryption services.
The policy implemented in the hypervisor forces all executable code to be read-only and encrypted. Therefore, unprivileged code cannot change executable code. A physical memory attack could modify code; however, code memory is encrypted with a unique per-session key, making meaningful modification of code memory in a broadly distributable fashion difficult. In addition, the stack and heap are always marked as non-executable, and therefore data loaded there can never be jumped to by unpriviledged code.
Unprivileged code interacts with the hypervisor via the "sc" ("syscall") instruction, which causes the machine to enter hypervisor mode.
You can argue your own definition of what a hypervisor is, but I suspect you won’t get any respect for doing so.
360 indeed uses hypervisor [0], but uses it only for security, to make the app signature verification run on a higher level.
Windows on PCs also runs under hypervisor if you enable some security features (e.g. VBS/HVCI which are on by default since Windows 11 2022 update, or Windows Sandbox, or WDAG) or enable Hyper-V itself (e.g. to use WSL2/Docker).
The performance losses are indeed there, but by purely running the hypervisor you lose just around 1% [1], because the only overhead is added latency due to accessing memory through SLAT and accessing devices through IOMMU...
I'd imagine XB1 is running with all the security stuff enabled though, which demands additional performance losses [2]..
Again with the caveat that this is specific to dom0 virtualization taking advantage of full hardware acceleration VT-d/VT-x, etc, what you say isn’t even necessarily the case.
With modern virtualization tech, the hypervisor mainly sets things up then steps out of the way. It doesn’t have to involve itself at all in the servicing of memory requests (or mapped memory requests) because the cpu does the mapping and knows what accesses are allowed or aren’t. The overhead you’re talking about is basically traversing one extra level in a page table noticeable only on micro benchmarks when filling the tlb or similar - this is a change in performance you might encounter a micro-regression in (without any virtualization to speak of) even when going from one generation of cpu architecture to the next.
Theoretically, the only time you’ll have any overhead is on faults (and even then, not all of them).
Of course I guess you could design a game to fault on every memory request or whatever, but that would be a very intentionally contrived scenario (vs just plain “bad” code).
I don’t understand what you’re trying to imply here.
Are you seriously suggesting that I chose to downgrade the graphics on the XB1 because I felt like it, and that dozens of other AAA game studios did the same thing?
Our engine was Microsoft native, by all rights it should have performed much better than PS4.
If you’re going to argue you’ll have to do a lot better than that since I have many years of lived experience with these platforms.
I didn't take it personally, i just think you're presenting ignorance as fact and it's frustrating.
Especially when seemingly it comes from nowhere and people keep echoing the same thing which I know not to be true.
Look, I know people really love virtualisation (I love it too) but it comes with trade-offs; spreading misinformation only serves to misinform people for.. what, exactly?
I understood the parents perspective, GPU passthrough (IE; VT-d & AMD-Vi) does pass PCI-e lanes from the CPU to the VM at essentially the same performance. My comment was directly stating that graphical fidelity does not solely depend on the GPU, there are other components at play, such as textures being sent to the GPU driver, those textures don't just appear out of thin air, they're taken from disk by the CPU, and passed to the GPU. (there's more to it, but usually I/O involves the CPU on older generations)
The problem with VMs is that normal memory access's take on average a 5% hit, I/O takes the heaviest hit at about 15% for disks access and about 8% for network throughput (ballpark numbers but in-line with publicly available information).
It doesn't even matter what the exact precise numbers are, it should be telling to some degree that PS4 was native and XB1 was virtualised, and the XB1 performed worse with a more optimised and gamedev friendly API (Durango speaks DX11) and with better hardware.
It couldn't be more clear from the outside that the hypervisor was eating some of the performance.
I guess I should clarify that my point was purely in abstract and not specific to the XBox situation.
Of course in reality it depends on the hypervisor and the deployed configuration. Running a database under an ESXi VM with SSDs connected to a passed-through PCIe controller (under x86_64 with hardware-assisted CPU and IO virtualization enabled and correctly activated, interrupts working correctly, etc) gives me performance numbers within the statistical error margin when compared to the same configuration without ESXi in the picture.
I haven’t quantified the GPU performance similarly but others have and the performance hit (again, under different hypervisors) is definitely not what you make it out to be.
My point was that if there’s a specific performance hit, it would be pedantically incorrect to say “virtualizing the GPU is the problem” as compared to saying “the way MS virtualized GPU access caused a noticeable drop in achievable graphics.”
Sorry, I don't think I implied virtualising the GPU is the problem.
I said "the fact that it's a VM has caused performance degradation enough that graphical fidelity was diminished" - this is an important distinction.
To clarify further: the GPU and CPU is a unified package and the request pipeline is also shared, working overtime to send things to RAM will affect GPU bandwidth, so overhead of memory allocations that are non-GPU will still affect the GPU due to that limited bandwidth being used.
I never checked if the GPU bandwidth was constrained by the hypervisor to be fair, because such a thing was not possible to test, the only corrolary is the PS4 which we didn't optimise as much as we did for DX and ran on slightly less performant hardware.
Both the original Xbox One and the Xbox One S have a custom, 1.75GHz AMD 8-core CPU, while the Xbox One X bumps that up to a 2.3GHz 8-core chip. The base PS4 CPU remained clocked at 1.6GHz and contains a similar custom AMD 8-core CPU with x86-based architecture, while the PS4 Pro bumps that clock speed up to 2.13GHz.
The CPU isn't particularly relevant is it (although the CPUs in the PS4/XBone generation were exceptionally terrible compared to what was standard on PCs at the time)? Graphical fidelity is going to depend much more on the GPU (although the CPU is going to bottleneck framerate if it's not powerful enough).
In the current generation the Series X has a more powerful GPU than the PS5, which tends to mean a lot of games run at higher resolutions on the system, although there's some games that run slightly better on PS5 (I think the Call of Duty games might be in that category?). And a lot (most?) are basically the same across both systems - probably because devs aren't bothering to tweak cross platform games separately for the two different consoles.
I also spent time in integrate display controller development and such; it was all very similar.
Nowadays it feels like everything rides on top some ugly and opaque stack.