John Gruber: WWDC 2024: Apple Intelligence

[A]ccording to well-informed little birdies, Vision Pro is already making significant use of the M2’s Neural Engine to supplement the R1 chip for real-time processing purposes — occlusion and object detection, things like that. With M-series-equipped Macs and iPads, the Neural Engine is basically sitting there, fully available for Apple Intelligence features. With the Vision Pro, it’s already being used.

Gruber is reporting this is why the Vision Pro is the only M-powered chip not to be getting Apple Intelligence features: the M2 and its neural engine are being pushed to their limits already.

Interestingly, I actually replied to a common thing I was hearing in the first month of owning a Vision Pro that the M2 was so powerful already that Apple had plenty of time before any new performance would be needed. I said:

Let’s start with the M2 processor, which is impressive, but is clearly being pushed to its limits. Start with foveated rendering, which renders just the pinhole you’re looking at in full resolution, and everything else far lower res. Some level of this will always be okay because you can’t see the whole screen at once, but it’s so extreme on this version of the hardware to the point that it’s easy to see the blur with your eyes while in the headset. It’s a good reminder that while foveated rendering is a clever solution to a problem, it’s not like it’s ideal for it to be there; it’s only there because the hardware can’t keep up. If you want to see what the Vision Pro runs like without foveated rendering enabled, try recording your screen through a Mac using Reality Composer Pro, which will render the whole view at full clarity, and it runs like absolute butt.

The M2 and R1 do a good job at many things, but it’s so clear that more power would make using the Vision Pro better.

Rumors are that Apple is focusing on getting a lower-cost Vision…something out the door next, and I think that’s great for making this a product more people can even humor buying, but I continue to think that performance is a real bottleneck, and every performance upgrade we can throw at this product is going to make people go, “oh my god, it’s so much better now!”