ChatGPT can create virtual worlds

In Star Trek: The Next Generation, Captain Picard and the crew of the U.S.S. Enterprise used the holodeck to generate realistic 3D environments where they could train and also entertain themselves. 

The holodeck is similar to the Construct in The Matrix and any number of other fictional virtual world environments in sci-fi and cyberpunk fiction.

Crucially, these worlds are endlessly customizable and they only require a language input to create an environment. 

For anyone keeping up with the latest AI news that will sound very familiar. ChatGPT is, of course, a large language model (LLM) that can create streams of text and code. All it needs is a simple text prompt.

A team of engineers from the University of Pennsylvania’s Engineering department collaborated with scientists from Stanford, the University of Washington, and the Allen Institute for Artificial Intelligence (AI2) to make its own Holodeck. They used an LLM and a library of video game assets.

Language is a “representation of the entire world”

The team’s real-world Holodeck can generate an endless range of indoor environments by using AI to interpret requests.

“We can use language to control it,” explained Yang, one of Holodeck’s co-creators from the University of Pennsylvania. “You can easily describe whatever environments you want and train the embodied AI agents.”

The team set out to create a system that could build interactive environments for training robots. These types of environments, called “Sim2Real”, are already used in robotics. However, roboticists have traditionally relied on artists to create these environments, meaning they can take a long time to create.

The researchers’ Holodeck uses an LLM to process requests. It then sorts through a massive library of digital videogame assets to create the required scene. The team’s process draws comparisons with people using ChatGPT to recreate classic video games, such as pong, when GPT-4 was first released.

“Language is a very concise representation of the entire world,” Yang explained in a press statement. Ultimately, the team’s Holodeck technology could help train robots with increased efficiency by developing multiple virtual environments at the drop of a hat. It’s also one more reminder that simulation theory exists, and it’s increasingly hard to ignore.