The iPhone’s next big camera trick could be 3D photos and videos — and here’s why

[ad_1]

Update June 15th: We’ve updated this article with feedback provided to us about Vision Pro and spatial imaging from the Halide camera app.

3D photography never really took off in the same way as the 2D art form, but Apple Vision Pro and the iPhone aim to change that. If they succeed, it could change the way we capture and experience memories.

This lofty goal is far from guaranteed. After all, Apple’s introduction of the Vision Pro (“the first 3D camera”) was simultaneously the most interesting and creepy part of a launch that often felt like a Black Mirror feature.

The long list of Vision Pro launch moments worth noting means it’s easy to forget that Apple announced a device that takes “spatial photos and 3D videos.” Considering the company’s last camera (outside of the iPhone) was Apple QuickTake in 1994, that’s a pretty big deal.

The general reaction to this camera portion of the Apple Vision Pro demo was… “Just no, Apple.” (Image credit: Apple)

On the other hand, the Vision Pro is a camera you have to wear on your face — and Apple’s comical demo led to widespread derision, even from those who helped create it. This is not to say that 3D photos and videos are once again doomed to remain marginal novelties. It just means that the Apple Vision Pro needs a device that can capture those “spatial” memories, while focusing on the playback experience.

These 3D cameras will definitely be the iPhone and iPad. And while that’s likely coming too soon for the iPhone 15, the iPhone 16 — or perhaps next year’s iPad Pro — could add the necessary sensors and software to Apple’s existing Lidar scanner to become the spatial cameras that the Apple Vision Pro desperately needs. If they do, 3D photos and videos could become the next big smartphone camera feature.

short memory

Apple isn’t the only company that’s been exploring the possibilities of capturing memories in 3D. The idea itself dates back to the first stereophonic devices in the 1830s, but we’ve only recently seen the invite-only Wist promise (below) to let you shoot “immersive memories” on your phone that can then be relived on headphones like the Meta Quest 2.

An adjacent trend is the rise of apps like Polycam, which is the most popular 3D scanning app for iOS and Android. Polycam founder Chris Heinrich told us that Apple Vision Pro’s spatial imagery is an exciting moment and that existing phone cameras will need to evolve to capture it.

“Normal images appear very flat in 3D, and don’t quite measure up to the hardware capabilities of 3D or stereoscopic content,” says Heinrich. “But Vision Pro’s 3D shots are also different from full 3D scans.” Spatial photos and videos they made [Apple] Advertised are not proper 3D models, but stereograms that produce a 3D effect, and possibly allow for some small head movements.”

However, while spatial imagery and 3D scanning applications are different beasts, Polycam believes the two will likely coexist in many applications. “We expect Apple to allow third-party developers to import spatial photos and videos from the Photos app just as they do with regular photos and videos,” Heinrich adds. “If this is the case, we will likely support importing, viewing and editing photos and videos with Polycam,” he adds.

Stare into the future

For many, the idea of ​​capturing memories in 3D seems like a gimmick. After all, 3D TVs — of which the Apple Vision Pro is arguably the natural successor — have been one of the tech’s biggest failures. And ten years ago, stereo cameras like the Fujifilm FinePix Real 3D W1 similarly failed to make a cultural imprint.

But even before the Vision Pro arrived, a growing number of experts were hailing 3D capture as the next biggest thing since the arrival of digital photography. Now that Apple is getting in on the game, the question is how fast can it finally get off the ground?

The obvious takeaway from the launch of the Vision Pro was that the iPhone 15, which is due to land in September, will certainly be able to take spatial photos and videos for Apple’s headset. But it may not be that simple, according to Polycam’s Chris Heinrichs.

Some iPhones have had Lidar scanners since the iPhone 12 Pro (above), but future iPhones will need more than that for shooting spatial photos and videos. (Image credit: Apple)

He says that spatial imagery “may require adding another camera to the iPhone that is further apart (about the distance between our eyes) than the other cameras” and that this “would have cost and design trade-offs that probably wouldn’t justify adding it.”

Instead, it seems more likely that 2024 will be the year spatial memories take off. “I think they are [Apple] They’ll release the headset first, and if spatial photos and videos are successful, they’ll consider adding it on iPhone 16 or later,” he adds.

What is not in question is that the iPhone and iPad are more suitable for capturing spatial photos and videos than the Vision Pro itself. “I think it makes sense for Apple to support spatial photo and video capture on the iPhone because, first, taking photos from a handheld device is much more natural than taking them from a headset, and second, it plays to the strength of having a multi-device platform.”

Minority Report

Polycam isn’t the only app excited about 3D photos and videos, now that Vision Pro has arrived. The team behind Halide, which is still widely considered the best camera app, told us they see a lot of potential — even if it means waiting for VisionOS 2.0.

Ben Sandowski, co-founder of Halide’s Lux development studio, told us: “Over the years, we’ve watched Apple release building blocks that culminated in Vision Pro. For example, the iPhone 12 Pro saw the addition of a lidar sensor, open room scanning and 3D imaging. In While these technologies are just a tease of what we expect to see next year, we’ve already used them to improve Halide’s 3D capture capabilities, and we can’t wait for more,” he said.

But there is a stumbling block for third-party spatial imaging applications. “Our only disappointment is that it doesn’t look like the first version of Vision Pro will include capture capabilities for third-party apps,” he noted. He added, “We highly suspect this is a time constraint issue, as Apple has a fairly track record of supporting third-party camera apps. So, assuming these capabilities make it to ‘VisionOS 2.0’, we’re definitely in.”

(Image credit: Apple)

What about the capabilities of future iPhones with 3D cameras? Like Polycam, Sandofsky also noted that current iPhones aren’t up to the task. Lidar is only available on ‘Pro’ iPhones starting with the iPhone 12 Pro, and some iPhones use ML [machine learning] to collect 3D data. “This is more than powerful enough to create effects like ‘portrait mode’, but when it comes to fully realistic 3D reproduction, we suspect the transition from iPhone to Vision Pro will feel like the transition from Standard Definition to HDTVs.”

In a separate tweet, Halide’s team concluded, “Existing iPhones cannot create proper spatial snapshots the way a headset can.” [Apple Vision Pro] Can. We will need significant improvement in terms of processing power, sensor fidelity, and quality to make this work,” they added.

Given the rate of change in phone cameras — and the relative plateau of 2D camera features — there’s a chance the iPhone 16 could remain Apple’s first phone with stereoscopic lenses for 3D photos and videos.

But it’s a long way off, and besides the technical challenge of making it happen, Apple will also need to work hard to get the average person as excited about “spatial” memories as iOS developers are.

As our hands-on Apple Vision Pro review ended after a demo of the feature, “It was both a postcard from Minority Report but instead of prequels, we have a pre-Vision Pro that lets us relive moments like never before.”

This certainly sounds like a fun sci-fi experience — but as the last decade of VR headsets have taught us, there’s a big difference between making sci-fi a reality and turning that into a mainstream success.

[ad_2]