For running an x64 container, I'd recommend running aarch64 Linux and then running the container using Rosetta 2, that yields significantly better performance than running everything through QEMU.
Coincidentally I set this up last night under UTM (https://docs.getutm.app/advanced/rosetta/) and it worked astonishingly well. Outside of Rosetta, the native virt seemed a bit slower than whatever UTM does normally, but perfectly usable on my M1 MBP.
If you're saying the rest of the aarch64 VM got slower after enabling Rosetta 2, that's unfortunately expected. Under Linux, TSO is enabled for everything, even aarch64 processes when Rosetta is active. On macOS, TSO is selectively enabled only for the x64 processes and leaves everything else alone.
