BLOG-img

Real-time audio and video in the metaverse

annam-img
Annam
Dec 20, 2024 • 5 min read
Component 3

Meta's Reality Labs division will be responsible for developing its metaverse product, using Meta Quest, its virtual reality head-mounted displays. Meta's early metaverse environment, Horizon Worlds, rolled out to Quest users in the US and Canada in early December 2021. This drove a growth in its monthly user base by ten times, to 300,000 people.

We spoke to a few developers of browser-based Unity applications, who built real-time audio and video features to help their users connect and interact. We learned they manage game state within the Unity/C# execution environment and write separate JavaScript (JS) code to leverage the browser's WebRTC (libwebrtc) implementation for audio and video.

ai-img

Note that shadowed C# methods don't directly bridge their JS analogue. Instead, we construct a message containing the target method name and associated parameters and pass it through a single bridged function: CallMethod. This technique yields a minimal bridge interface and saves writing lots of JavaScript glue code.

Note that shadowed C# methods don't directly bridge their JS analogue. Instead, we construct a message containing the target method name and associated parameters and pass it through a single bridged function: CallMethod. This technique yields a minimal bridge interface and saves writing lots of JavaScript glue code.

Note that shadowed C# methods don't directly bridge their JS analogue. Instead, we construct a message containing the target method name and associated parameters and pass it through a single bridged function: CallMethod. This technique yields a minimal bridge interface and saves writing lots of JavaScript glue code.

More from VoiceSprint

An open source stack for real-time multimodal..

It feels impossible to keep up with everything happening in AI. There’s seemingly a new breakthrough every week.

aman-img
Aman
Dec 20, 2024 • 5 min read

Using a transformer to improve end of turn detection.

It feels impossible to keep up with everything happening in AI. There’s seemingly a new breakthrough every week.

chris-img
Chris
Dec 20, 2024 • 5 min read

Join the Future of AI

Experience the power of VIAI's AI agents to transform your operations.

up-arrow