In order to demonstrate the capabilities of the Innoactive XR Streaming Platform, we built StreamRoom AI. On behalf of this example, we aim to show how to build custom or adapt existing XR application can benefit from the Innoactive platform and its rich feature set.
What is StreamRoom AI?
StreamRoom AI represents a cutting-edge cloud-native solution that allows users to interact in virtual meetings, visualize large 3D data sets, and even integrate with an AI assistant, all in real-time and without the need for a personal computer. Users can access StreamRoom AI using standalone XR devices or through a web browser, increasing its accessibility and user base.
Some key features of StreamRoom AI include:
Collaborative Environment: StreamRoom AI allows multiple users to meet in a virtual space and share, discuss, and collaborate on 3D data, making it a versatile tool for a wide range of enterprise applications.
3D Data Visualization: Users can upload and visualize complex 3D data in StreamRoom AI, making it useful for industries where detailed 3D models are frequently used, such as architecture, engineering, and game development.
AI Integration: StreamRoom AI includes an AI assistant that can respond to voice commands and assist with tasks within the virtual environment.
Cloud-Based Streaming: Hosted on Innoactive's XR Streaming Platform, StreamRoom AI leverages the power of the cloud for efficient data processing and high-quality graphics.
How to build XR applications like StreamRoom AI?
Setting up the Environment: The Unity game engine serves as the fundamental platform for developing the StreamRoom application. It is one of the most popular tools for developing both 2D and 3D video games, interactive simulations, and other experiences. In this case, it is used as the foundation for creating a VR-enabled 3D environment. Using Unity allows developers to efficiently build and modify the 3D spaces, manage assets, and handle user interactions within these spaces.
Rendering and Streaming: NVIDIA CloudXR is a crucial component. It is used by our XR streaming platform to allow real-time streaming of high-quality, immersive VR experiences. This technology allows the virtual environment to be rendered on powerful cloud servers, reducing the processing load on users' devices. NVIDIA CloudXR technology works with any OpenXR or OpenVR application. It utilizes the power of GPU-powered virtual machines (e.g. AWS EC2 G5) to render VR scenes that are then streamed to the user's device, providing high-quality visuals without the need for high-end hardware at the user's end.
Cloud-based Deployment and Scaling: Innoactive XR Streaming Platform plays a critical role in the scalability and deployment of the StreamRoom AI application. Innoactive provides the infrastructure to host the VR application in the cloud and stream it to users' devices. It is also used for the management of the virtual machines (VMs) which run the application, allowing for smooth scaling and deployment. Together, these components ensure that the StreamRoom application can be quickly deployed, scaled, and maintained in a cloud environment.
Asset Management and Loading: AWS Visual Asset Management System (VAMS) is serving as a central 3D asset library. With its support to set-up asset conversion pipelines and workflows, it can help to connect existing 3D asset sources such as PLM systems like Siemens Teamcenter or SAP Enterprise Product Development (EPD) with XR applications such as StreamRoom AI. In our example, VAMS is responsible for the management and conversion of these 3D assets, making sure they are in the correct format for use within the Unity environment.
To load e.g. JT files into Unity, we recommend the Siemens JT Open Toolkit, which provides efficient, on-demand loading of these assets into the Unity scene during runtime. This combination ensures that high-quality 3D assets are readily available and can be dynamically loaded into the application as needed.
SAP EPD customers can request a Unity integration toolkit (UIT) for the same need. SAP EPD users can download an app to access 3D assets in an AR and 3D experience which is made with the Unity engine and the SAP Business Technology Platform (BTP). The app offers a demo mode and can be accessed on iOS and Android for free.
Collaboration and Interaction: Photon Fusion is utilized for the multiplayer and networking functionalities of StreamRoom. This tool handles the complexities of network programming, allowing multiple users to interact within the same 3D environment in real-time. With Photon Fusion, users can interact with each other and manipulate shared objects within the 3D space, creating a highly collaborative virtual experience. Photon Fusion's robust networking capabilities ensure that these interactions are synchronized across all users, maintaining a consistent state of the virtual environment.
Avatar Creation: ReadyPlayerMe is used to allow users to create personalized avatars in StreamRoom. With this tool, users can create a digital representation of themselves, increasing their sense of presence in the virtual environment. These avatars are customizable, enabling users to express their individuality. The integration of ReadyPlayerMe ensures that users have a unique, personalized virtual identity when they interact within the StreamRoom environment.
ChatGPT and Voice Services: The OpenAI API, Amazon Transcribe, and Amazon Polly power the artificial intelligence and voice services within StreamRoom. Amazon Transcribe is used to convert the spoken language of the user into written text. This text is then sent to the OpenAI's GPT-4, a state-of-the-art language model that generates written responses based on the input. The response from GPT-4 is then transformed back into spoken language by Amazon Polly, a service that turns text into lifelike speech. This sequence allows the AI assistant within StreamRoom to understand user commands.
Global Availability: Native 5G networks are available in more and more areas worldwide. 5G's high bandwidth and low latency enable streamed VR experiences with minimal lag. AWS Local Zones extend AWS services closer to users in locations without an AWS Region, reducing latency. AWS Wavelength embeds AWS services at the edge of telecom networks, further minimizing latency and leveraging 5G's capabilities. Together, these technologies ensure a smooth and immersive VR experience for users worldwide.
Conclusion
StreamRoom AI fuses the power of multiple advanced technologies to offer users an immersive, collaborative, and personalized VR experience. By integrating existing technologies, StreamRoom showcases the potential of virtual reality in revolutionizing the way we interact, learn, and communicate. It demonstrates how truly transformative, scalable, and accessible XR business solutions can be built.
If you're intrigued by the possibilities that StreamRoom offers and would like to experience it for yourself, we would be thrilled to hear from you. We invite you to get in touch with us and explore the potential of this revolutionary platform for your unique needs. Together, we can harness the power of VR and AI to create immersive and interactive experiences like never before. Please contact us today to begin your journey into the future with StreamRoom and Innoactive XR Streaming Platform.