For the last couple of weeks , I have been working on the concept of rendering 3D graphics on WebRTC media stream using different JavaScript libraries as part of a Virtual Reality project .
What is Augmented Reality ?
Augmented reality (AR) is viewing a real-world environment with elements that are supplemented by computer-generated sensory inputs such as sound, video, graphics , location etc.
How is it diff. from Virtual Reality ?
Virtual Reality – replaces the real world with simulated one , user is isolated from real life , Examples – Oculus Rift & Kinect
Augmented Reality – blending of virtual reality and real life , user interacts with real world through digital overlays , Examples – Google glass & Holo Lens
Methods for rendering augmented Reality
- Computer Vision
- Object Recognition
- Eye Tracking
- Face Detection and substitution
- Emotion and gesture picker
- Edge Detection
web based Augmented Reality solution
Components for a Web base end to end AR solution
Web :
WebRTC getusermedia
Web Speech API
WebGL
css
svg
HTML5 canvas
sensor API
H/w :
Graphics driver
microphone and camera
sensors
3D :
Geometry and Math Utilities
3D Model Loaders and models
Lights, Materials,Shaders, Particles,
Animation
WebRTC
- Web based Real Time communications
- Definition for browser’s media stream and data
- Awaiting more standardization , on a API level at the W3C and at the protocol level at the IETF.
- Enable browser to browser applications for voice calling, video chat and P2P file sharing without plugins.
- Enables web browsers with Real-Time Communications (RTC) capabilities
- MIT : Free, open project
Code snippet for WebRTC API
1.To begin with WebRTC we first need to validate that the browser has permission to access the webcam.
<video id="webcam" autoplay width="640" height="480"></video>
- Find out if the user’s browser can use the getUserMedia API.
function hasGetUserMedia() { return !!(navigator.webkitGetUserMedia); }
- Get the stream from the user’s webcam.
var video = $('#webcam')[0]; if (navigator.webkitGetUserMedia) { navigator.webkitGetUserMedia( {audio:true, video:true}, function(stream) { video.src = window.webkitURL.createObjectURL(stream); }, function(e) { alert('Webcam error!', e); } ); }
Screenshot AppRTC
WebGL
- Web Graphics Library
- JavaScript API for rendering interactive 2D and 3D computer graphics in browser
- no plugins
- uses GPU ( Graphics Processing Unit ) acceleration
- can mix with other HTML elements
- uses the HTML5 canvas element and is accessed using Document Object Model interfaces
- cross platform , works on all major Desktop and mobile browsers
WebGL Development
To get started you should know about :
- GLSL, the shading language used by OpenGL and WebGL
- Matrix computation to set up transformations
- Vertex buffers to hold data about vertex positions, normals, colors, and textures
- matrix math to animate shapes
Cleary WebGL is bit tough given the amount of careful coding , mapping and shading it requires .
Proceeding to some JS libraries that can make 3D easy for us .
CCV
- website : http://libccv.org/
- SourceCode : https://github.com/liuliu/ccv
Three.js
- website : http://threejs.org/
- SourceCode : https://github.com/mrdoob/three.js/
- Demo: http://www.davidscottlyons.com/threejs/
Awe.js
- Website : https://buildar.com/awe/tutorials/intro_to_awe.js/index.html#
- SourceCode : https://github.com/buildar/awe.js
ArcuCO
- SourceCode: https://github.com/jcmellado/js-aruco
Potree
- website: http://potree.org/wp/
- SourceCode: https://github.com/potree
- Demo: http://potree.org/wp/demo/
Karenpeng
- emotion & gesture-based arpeggiator and synthesizer
- website:
- SourceCode : https://github.com/karenpeng/motionEmotion
- Demo: http://motionemotion.herokuapp.com/
Three.JS
- MIT license
- javascript 3D engine ie ( WebGL + more)
- started a year ago
- under active development
- no dependencies or installation
3D space with webcam input as texture
Display the video as a plane which can be viewed from various angles in a given background landscape. Credits for code : https://stemkoski.github.io/Three.js/
1.Use code from slide 10 to get user’s webcam input through getUserMedia
- Make a Screen , camera and renderer as previously described
-
Give orbital CONTROLS for viewing the media plane from all angles
controls = new THREE.OrbitControls( camera, renderer.domElement );
- Add point LIGHT to scene
-
Make the FLOOR with an image texture
var floorTexture = new THREE.ImageUtils.loadTexture( 'imageURL.jpg' ); floorTexture.wrapS = floorTexture.wrapT = THREE.RepeatWrapping; floorTexture.repeat.set( 10, 10 ); var floorMaterial = new THREE.MeshBasicMaterial({map: floorTexture, side: THREE.DoubleSide}); var floorGeometry = new THREE.PlaneGeometry(1000, 1000, 10, 10); var floor = new THREE.Mesh(floorGeometry, floorMaterial); floor.position.y = -0.5; floor.rotation.x = Math.PI / 2; scene.add(floor);</pre> <pre>
6. Add Fog
scene.fog = new THREE.FogExp2( 0x9999ff, 0.00025 );
7.Add video Image Context and Texture.
video = document.getElementById( 'monitor' ); videoImage = document.getElementById( 'videoImage' ); videoImageContext = videoImage.getContext( '2d' ); videoImageContext.fillStyle = '#000000'; videoImageContext.fillRect( 0, 0, videoImage.width, videoImage.height ); videoTexture = new THREE.Texture( videoImage ); videoTexture.minFilter = THREE.LinearFilter; videoTexture.magFilter = THREE.LinearFilter; var movieMaterial=new THREE.MeshBasicMaterial({map:videoTexture,overdraw:true,side:THREE.DoubleSide}); var movieGeometry = new THREE.PlaneGeometry( 100, 100, 1, 1 ); var movieScreen = new THREE.Mesh( movieGeometry, movieMaterial ); movieScreen.position.set(0,50,0); scene.add(movieScreen);
- Set camera position
camera.position.set(0,150,300); camera.lookAt(movieScreen.position);
- Define the render function
videoImageContext.drawImage( video, 0, 0, videoImage.width, videoImage.height ); renderer.render( scene, camera );
- Animation
requestAnimationFrame( animate ); render();