Difference between revisions of "Augmented Reality Roadmap for X3D"

Motivation: Value of X3D Standard for AR

Augmented Reality (AR) and Mixed Reality (MR) operate across many display devices, ranging from small hand-held mobile phones to tablets, laptops, desktops, and full-blown walk-around CAVE spaces. There is always a combination of both 2D imagery and 3D graphics, plus a mix of the real world and virtual worlds. Many people are especially interested in lightweight AR that they can use as part of the GPS-aware mobile devices.

Extensible 3D graphics(X3D) a roylaty free and ISO certified Standard developed by the Web3D Consortium (www.web3d.org) that originated from VRML, now in XML already has capabilities to support AR and MR applications. X3D is a mature standard with many implementations and many converters. It collects numerous proven technologies together in a workable way, providing a simple and coherent way for authors to create 3D models and VR applications. Current X3D efforts include identification of critical lightweight components suitable for mobile devices and AR applications. X3D can save a lot of time for developers looking to design new AR applications.

Because there are multiple implementations and extensions for AR that have already been demonstrated for X3D, it is clear that we can build an AR component for X3D. It will have lightweight capabilities suitable for mobile devices, plus additional compatible capabilities that work with larger 3D scenes and spaces. We will take advantage of X3D extensibility to define component levels and profiles that capture these palettes as easily expanded extensions.

It is sometimes tempting to think of a mobile device as simply a rotatable 2D display situated within the 3D real world. However 2D displays by themselves can be pretty limiting and oversimplified. The X3D group is thinking in terms of the larger 2D + 3D space that many authors and users want. Demonstrations are also showing that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces.

Now is the time for serious AR developers to consider these essential challenges. Our current work is focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors. Collaborative work is happening to extend X3D to make AR/MR capable. Numerous innovative AR activities are ongoing by many researchers, companies and study groups. Web3D is engaged is all these efforts. Coordination and collaboration between numerous developers and standards organizations is necessary for AR to eventually become a seamless part of the Web infrastructure.

X3D Strategy for AR

Multiple AR nodes have been proposed and implemented for X3D by Web3D Consortium members. A comprehensive list of AR nodes will be formally written up as the Augmented Reality (AR) component of the X3D Specification. Establishing common implementations and examples to demonstrate successful interoperability for the spec is also important. Recent work in HTML5/DOM, geospatial GpsSensor, and palettes for interactive scenes has lead to a propostion of a new lightweight (Mobile) Profile for X3D. Heavyweight advanced capabilities for mixed reality, CAVEs etc. will be kept separate from Mobile but available in an AR component.

Continued collaboration and reaching out to the ISO-SC24 Working Group, W3C Augmented Reality Community Group, newly formed OGC ARML Working Group, KHRONOS, and the AR Standards Group is crucial. Our common goal remains maximum interoperability with all AR and Web standards.

X3D Relation to Reality–Virtuality Continuum

The Reality–Virtuality Continuum is often used to describe the range of functionality involved. The goals for X3D capabilities are broad.

          | - - - - - - - - - Mixed Reality (MR) - - - - - - - - |
      
    <<================================================================>>
      
      Real          Augmented                 Augmented         Virtual
      Environment  Reality (AR)           Virtuality (AV)   Environment
      
    <<================================================================>>
    
                   | - - - - - - X3D AR-Related Nodes - - - - - - - - |

Extending X3D for AR/MR application

The following list shows the functions and features to add for supporting AR and MR visualization in X3D:

• Adding real world view
  • Access to live video (Camera on the user's device)
    • Live video stream as a texture in the X3D scene
    • Live video stream as a background of the X3D scene
• Merging real and virtual image correctly
  • Access to calibration information of the camera device providing the video stream
  • Viewpoint nodes using calibration information to set projection parameters
  • Chroma keying of live video stream texture (rendering pixels in key color transparent)
  • Correct occlusion between real and virtual objects
    • Depth masking with virtual representation of physical object
    • Support for depth image and depth imaging devices
• Tracking, registration and real-time interaction
  • Access to motion tracking information
  • Camera for registration between real and virtual spaces
  • Other physical objects for real-time interactivity

X3D's layered "profiles"

The modular architecture of X3D allows for layered "profiles" that can provide increased functionality for immersive environments and enhanced interactivity or focused data interchange formats for vertical market applications within a small downloadable footprint composed of modular blocks of functionality ("Components"), that can be easily understood and implemented by application and content developers. Components can be individually extended or modified through adding new "levels", or new components can be added to introduce new features, such as streaming. Through this mechanism, advancements of the specification can move quickly because development in one area doesn't slow the specification as a whole. Importantly, the conformance requirements for a particular piece of content are unambiguously defined by indicating the profiles, components and levels required by that content.

• Interchange is the basic profile for communicating between applications. It support geometry, texturing, basic lighting, and animation. There is no run time model for rendering, making it very easy to use and integrate into any application.
• Interactive enables basic interaction with a 3D environment by adding various sensor nodes for user navigation and interaction (e.g., PlaneSensor, TouchSensor, etc.), enhanced timing, and additional lighting (Spotlight, PointLight).
• Immersive enables full 3D graphics and interaction, including audio support, collision, fog, and scripting.
• Full includes all defined nodes including NURBS, H-Anim and GeoSpatial components.

Multiple implementations and extensions for AR have already been demonstrated in X3D. X3D’s extensibility provides the advantage to define component levels and profiles as extensions, allowing applications to not have to implement the entire specification. It has also been demonstrated that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces.

Current Web3D Consortium AR working group activities are focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors. This Working Group is thinking in terms of the larger 2D + 3D space that authors and users want. We will harmonize lightweight AR components within an easily implementable X3D Mobile Profile. Heavyweight requirements for MR and CAVEs can be compatibly defined.

X3D's layered Components and Profiles

                Eight Profiles                for common use cases
               
            35 X3D Components                 for modular design
      
       Two Hundred Eighteen X3D Nodes         for every little thing!

For comparison purposes, 68 nodes are in the X3D Interactive profile which is lightweight already runs on mobile devices. Approximately 10 new nodes are being considered for AR, of which only a portion are suitable for mobile.

X3D Lightweight/Mobile profile

Recent work in HTML5/DOM, geospatial GpsSensor, and palettes for interactive scenes has lead to a propostion of a new lightweight (Mobile) Profile for X3D. A lightweight profile is valuable for author creation of simple but dramatic content. This approach allows a small-footprint and efficient operation of X3D on mobile devices, easy export from other formats, simple vocabulary for new authors and improve X3D value to important new areas:

• Mobile applications
• Lightweight HTML5 web pages, various devices
• Augmented reality (AR)

• Heavyweight advanced capabilities for mixed reality, CAVEs etc. kept separate from lightweight/Mobile profile but available in an AR Profile

Integrating X3D into HTML(X3dom)

The traditional way to embed a X3D augmented reality scene into a webpage uses a plug-in that needs to be installed by the user. The plug-in approach has always been an issue for web developers and users. Several efforts have been initiated by Web3D on how to integrate X3D with HTML. Our goal is to bring 3D to Web developers by fully integrating 3D content into HTML documents. Today Web pages exist as large collections of hierarchical elements that are rendered to the page; all content appears in the Document Object Model (DOM). Several tools are available to allow programmers to manipulate the DOM, in particular tools and frameworks based on JavaScript. Placing 3D graphics in the DOM, allows these JavaScript tools to access and manipulate 3D graphics. Web3D and Fraunhofer IGD‘s recent efforts – X3DOM (www.x3dom.org), integrating X3D with HTML and the DOM, have proven this capability. X3DOM uses three basic approaches to display an X3D scene inside an HTML page without a plug-in.

• External reference - HTML page includes an object element tag that refers to an .x3d scene, implemented via an X3D plug-in. Data might be passed within the page using DOM events.
• X3D as XML in HTML - HTML page directly includes X3D source, likely with an XML namespace prefix, presumably implemented via an X3D plug-in or the browser itself. Again data might be exchanged within the page using DOM events.
• API access - HTML page includes some form of canvas (or maybe Canvas3D) element that allows programmatic access to the page, so that X3D Scene Access Interface (SAI) might draw a bitmap.

The recently formed “Declarative 3D” W3C Community Group's efforts to define requirements to integrate 3D with HTML and the DOM, is using X3DOM as one of its test models. This effort has strategic importance for Web3D and indeed for all 3D graphics, since X3D is an interchange format for a wide variety of models. We intend to establish a solid foundation for X3D to properly support 3D graphics for the native Web page enhancing AR capabilities and experience.

X3D Lightweight/Mobile profile

A lightweight profile is valuable for author creation of simple but dramatic content. This approach allows a small-footprint and efficient operation of X3D on mobile devices, easy export from other formats, simple vocabulary for new authors and improve X3D value to important new areas:

• Mobile applications
• Lightweight HTML5 web pages, various devices
• Augmented reality (AR)

• Heavyweight advanced capabilities for mixed reality, CAVEs etc. kept separate from lightweight/Mobile profile but available in an AR Profile

Why X3D is a viable and proven standard platform for implementing AR/MR

The Web3D consortium is a standardization group that has been producing and maintaining it's standard X3D for ISO for more than 10 years. Today, the Web3D Consortium is utilizing its broad-based industry support to develop the X3D specification, for communicating 3D on the web, between applications and across distributed networks and web services. Through the well-coordinated efforts with the ISO, OGC and W3C, the Web3D Consortium is maintaining and extending its standardization activities.

X3D is a mature standard with many examples, many players and many converters. Multiple implementations and extensions for AR have already been demonstrated in X3D. These efforts have been successful because X3D collects numerous proven technologies together in a workable way, providing a simple and coherent path for authors to create 3D models and VR applications. X3D’s extensibility provides the advantage to define component levels and profiles as extensions, allowing applications to not have to implement the entire specification. It has also been demonstrated that mobile X3D can fit well with HTML5 and DOM improvements. This all "plays well" when moving from simple mobile applications to bigger models and spaces. Current Web3D Consortium AR working group activities are focused on harmonizing proven capabilities into best practices for AR and 3D graphics, implementable by multiple X3D viewers and usable by content authors. This Working Group is thinking in terms of the larger 2D + 3D space that authors and users want. We will harmonize lightweight AR components within an easily implementable X3D Mobile Profile. Heavyweight requirements for MR and CAVEs can be compatibly defined. X3D is becoming a viable and reliable standard for AR developers looking to save time when designing new AR applications.

Features already present in X3D

• XML Integrated: the key to integration with:
  • Web Services
  • Distributed Networks
  • Cross-platform, inter-application file and data transfer
• Componentized: allows lightweight core 3D runtime delivery engine
• Extensible: allows components to be added to extend functionality for vertical market applications and services
• Profiled: standardized sets of extensions to meet specific application needs
• Evolutionary: easy to update and preserve VRML97 content as X3D
• Broadcast/Embedded Application Ready: from mobile phones to supercomputers
• Real-Time: graphics are high quality, real-time, interactive, and include audio and video as well as 3D data.
• Well-Specified: makes it easier to build conformant, consistent and bug-free implementations
• 3D graphics and programmable shaders - Polygonal geometry, parametric geometry, hierarchical transformations, lighting, materials, multi-pass/multi-stage texture mapping, pixel and vertex shaders, hardware acceleration
• 2D graphics - Spatialized text; 2D vector graphics; 2D/3D compositing
• CAD data - Translation of CAD data to an open format for publishing and interactive media
• Animation - Timers and interpolators to drive continous animations; humanoid animation and morphing
• Spatialized audio and video - Audio-visual sources mapped onto geometry in the scene
• User interaction - Mouse-based picking and dragging; keyboard input
• Navigation - Cameras; user movement within the 3D scene; collision, proximity and visibility detection
• User-defined objects - Ability to extend built-in browser functionality by creating user-defined data types
• Scripting - Ability to dynamically change the scene via programming and scripting languages
• Networking - Ability to compose a single X3D scene out of assets located on a network; hyperlinking of objects to other scenes or assets located on the World Wide Web
• Physical simulation and real-time communication - Humanoid animation; geospatial datasets; integration with Distributed Interactive Simulation (DIS) protocols

Also, the modular architecture of X3D allows for layered "profiles" to provide increased functionality for vertical market applications. X3D applications are used world wide for visualization in CAD, GIS, Games, Stereoscopic Viewing. With the addition of Augmented Reality reality functionality.

MYTHS: Standard is old and not efficient for today's Web, lack of Adoption, Spec too large, needs plug-in, ...etc

• We need supporting material? ( Johannes and Peter wating for numbers on IR and BS Contact)

Interoperability with other AR and Web Standards

Several discussions at the Web3D Conference, SIGGRAPH and ISO-SC24 meeting about the recent AR Standards developments continue to improve and refine our strategy on interoperability with other standards. Establishing common implementations and examples is important to demonstrate successful interoperable capabilities for the spec. Continued collaboration and reaching out to the ISO SC24, W3C Augmented Reality Community Group, newly formed OGC ARML WG, AR Standards group, KHRONOS and others is essential. The ISO Standards Committee - SC24, has established a new Working Group for Augmented and Mixed Reality. The group proposes a AR content model as an extension of a virtual world with provisions for representing the physically-sensed objects.

Current technical work within the Web3D AR WG includes extending X3D for AR while harmonizing the following proposals for best fitting AR capabilities into X3D scenes.

• HTML and X3D - X3dom (Declarative 3D) for abstract content components for 2D and 3D Augmentation.
• OGC's ARML proposal for describing POIs and sensed physical objects.
• W3C ---

This work will be formally written up as the Augmented Reality (AR) Component for the X3D Specification. Our common goal remains maximum interoperability with all Web standards.

How to come to Fruition

TODO and Next Steps

• X3D's AR component (X3D/AR Profile) - AR WG Please review technical details and expand
• screen-capture pictures showing key capabilities already demonstrated
• Position Paper to be Review by the AR WG - Further discussion, details and comments to finish this document in September 23rd, 2011