Can these be used for local inference on large models? I'm assuming the 128G of memory is like system memory, not like GPU VRAM.

It has a unified memory architecture, so the 128G is shared directly between CPU and GPU. Though it's slower than dGPU VRAM.

Yes, but it is substantially cheaper and usually faster to buy a Jetson Orin chip or build an x86 homelab.

Memory bandwidth is less than 300 GB/sec, looking at the data sheet. So it won't really be any faster at local inference than a Mac Pro or a decent x86 box.

It appears to be an embedded DGX Spark, at the end of the day.