Google's 3D chat booth felt like real life science fiction

Published: October 14, 2022

Project Starline is the real deal. It makes a virtual call feel like a one-on-one conversation. But it's unclear if it's something most people will ever get to experience.

It looked like I could grab the apple. Jason Lawrence, a Google researcher, was sitting across from me, holding the fruit in his hand. I could see it, it was red and shiny, and my brain was telling me it was right there. But Lawrence and the apple were actually in another room, they were just being projected in front of me through Google's Project Starline.

Project Starline is Google's next-generation 3D video chat booth that it first introduced at Google I/O 2021. Slide into a booth, and your image is supposed to be projected to another booth in real time, as if you're actually sitting with somebody else across a table. In a heartwarming video, Google showed family and friends joyfully connecting with each other using Starline, and the virtual recreations looked remarkably lifelike. "That was mindblowing," one person says in the video. "I've seen a lot, but I've never seen this," said another.

But in the year since, Google has largely kept the project under wraps, until now. The company invited me to check out Starline for myself, and I was eager to find out if it would live up to the hype.

"We think we have a breakthrough in communication technology that makes you feel closer and more connected with people that could be anywhere around the world," said Andrew Nartker, director of product management for Starline. Nartker was seated across from me at a table in one of Google's conference rooms for a normal face-to-face meeting. He described the experience of Starline as "a magic window" that lets you feel like "you're connected and together" with other people.

Eventually, we moved on to test Starline. The machine was stuffed into a small conference room, taking up the vast majority of the space. On one side, there was a long wooden bench with a seat cushion right in the middle. On the other, a display showed the empty cushion in the other Starline booth. When I took a seat, more than a dozen cameras and sensors were pointed at me. It was nerve-racking, I could tell my every movement was being tracked. 

But when Nartker slid into frame in his Starline booth, the tech largely faded away, and we were able to immediately pick our conversation back up as if we had moved from one table to another.

Starline does a terrific job making a 3D representation of the person you're talking to. Both Nartker and Lawrence looked just like they did when I shook their hands just a few minutes earlier. Virtual shadows behind both of them helped sell the effect. It was even possible to estimate how far Lawrence's apple was in front of his body.

The whole thing felt much more natural than a Zoom call. There was no discernible latency in their movements or our conversation, so talking just felt like talking. There weren't any weird audio or visual delays. It was easier to believe I was actually with someone because I was making genuine eye contact with full-size virtual humans instead of squinting at a tiny Zoom window. 

Part of the reason Starline is so convincing is that you're not just looking at a screen but a series of lenses in front of a screen, or a lenticular array. The principle is similar to holographic cards that can display a different image or 3D effect when you rock them back and forth, Lawrence said. Starline takes that a step further by tracking your eyes to know where to direct the images it's displaying to you. 

The illusion wasn't perfect. Looking closely, you could tell that the person was being recreated; a head may not be perfectly round, for example, and the top of Lawrence's hair could get quite jittery. If the person you're talking to moves to the bounds of what Starline was actively reconstructing (Nartker estimated it was about one cubic meter of space), they'll get fuzzy, break into glitchy polygons, and eventually disappear entirely.

Starline is impressive, and it definitely made video chatting better. I can see it being useful for one-on-one conversations, especially in places like a doctor's office or a customer service environment. But Starline clearly has some limitations. The Starline booth is huge and loaded with presumably expensive technology. Nartker declined to share a price when I asked, and you'll need two to be able to use Starline at all. 

Still, Google is moving forward with Starline by ever so slightly expanding who will be able to try it out. This week, the company announced it would be installing Starline booths for select enterprise partners. Companies including Salesforce, WeWork, T-Mobile, and Hackensack Meridian Health will be testing the technology as part of an early access program. Google employees have already put in thousands of hours, Nartker said, and the company said it has invited "more than 100 enterprise partners" in industries such as media, healthcare, and retail to try Starline at Google offices. 

There's still a ways to go before Starline is widely available to buy, if it ever is. "It's an early stage technology that we think is very exciting and a breakthrough in the space," Nartker said. "But it's not a product at this point."

Throughout our conversation, Nartker talked about how Starline vastly improved telepresence because it replicated an in-person face-to-face conversation. After experiencing Starline for myself, I'm now on board with the idea that better telepresence can have a meaningful impact on virtual interactions, and I feel as if something like Starline must be in our future, even if that future is years away and hopefully isn't a giant video booth.

The same day, while listening to Meta CEO Mark Zuckerberg's interview with The Verge, I was struck by how he made a similar pitch for a set of vastly different devices. "The magic of VR, for people who have experienced this, is that it basically immediately convinces your mind that you are present in another place and with the people who are there," Zuckerberg said. He discussed how, in the future, the interview could be done with him as a hologram, which is largely what I had just been amazed by with Starline.

I found it striking that two of the biggest tech companies in the world are trying to improve virtual presence. Of course, both would benefit from doing so; Meta is all in on its concept of the metaverse, while Google is pushing hard to take over the enterprise. Chat booths and expensive mixed reality displays are neither practical nor desirable for most people just yet, but perhaps the thing that really takes off will happen somewhere in the middle.

Nartker ended our Starline conversation with a virtual fist bump, something he said he does with every demo. Like with Lawrence's apple, I logically understood that I wouldn't actually feel Nartker's knuckles hit mine. But when we went in for the bump, my brain anticipated my fist would make contact all the same, and my hand felt some sensation anyway. When our hands ghosted into each other, we laughed about the shared moment, even though we were rooms apart.

Source: Re-posted and Summarized from AY PETERS at the verge.