By Geoff Arnold
Download
Puppet in Motion Case Study [PDF 1.16MB]
Sixense Puts Puppets in Perceptual Motion
Danny Woodall (creative director) and Ali Diaz (CTO) of Sixense Studios have been through their share of tough user acceptance testing. After all, for years they’ve been blazing a trail in the market for motion tracking technologies, where ideas about what’s possible too often come from the movies rather than the real world. (For example, think of the gesture interface Tom Cruise used in the 2002 film “Minority Report.”)
Woodall and Diaz led the Sixense Entertainment team that developed the grand-prize-winning app in the seven-week Intel® Ultimate Coder Challenge: Going Perceptual contest, which wrapped up in April. Sixense’s virtual puppeteering app Puppet In Motion bested entries from six other developer teams in the challenge’s main criteria: to build groundbreaking apps that would make the most of the perceptual computing capabilities enabled by the latest Ultrabook™ device platforms. But before they could submit the app to the Intel judges, they had to satisfy a more finicky set of critics—their kids.
“My daughter is five,” said Woodall, the creative director at Sixense Studios, the business unit within Sixense Studios that is focused on software development. “She sees me with my hand raised in the air and I’m driving these little wolf and pig puppets on the screen and she gets ecstatic; she jumps in my lap and wants to do it too.”
Indeed, if the work of Sixense and the other entrants is any indication, children growing up today will increasingly expect perceptual computing to be the norm rather than a novel way to interact with computing technology.
The virtual puppets in the winning app come from the iconic Three Little Pigs fairy tale.
Figure 1.Sixense created scenes at surfer pig’s beachfront grass hut, lumberjack pig’s log house, and suburban pig’s practical bungalow.
Train Your Users Well
Woodall and Diaz say the key to delivering a good perceptual computing experience is to focus on the user and not on the capabilities of the various new and evolving sensor technologies available in the hardware. Applying this approach to their contest app meant spending time on features that weren’t immediately part of gameplay but nonetheless were essential to the overall experience. Perhaps the best example of this is the brief setup and calibration that users walk through upon starting the app.
When you Launch Puppet In Motion and before you start acting out the scene between the surfing pig and the hungry wolf, a darkened stage appears. The red curtains rise and there are the pig and wolf, each standing in a pool of light.
You raise your hand and a virtual image of it appears, along with on-screen text that instructs you how to place your hand in the correct position to operate the virtual puppets. This calibration is a key element and contributes to a positive first impression, which so often sets the tone for a user’s overall impression of an app.
Figure 2.Hand calibration at launch of the Puppet In Motion app.
The calibration step was critical in teaching users how to interact with the gesture recognition in a way that establishes a set of conventions for the game. When told they’re about try out a virtual puppet, most users first tip their hand forward toward the screen, much as they would if they had their hands in actual sock puppets. The problem is that with the hand in this position, tracking where fingers are relative to each other and to the palm is extraordinarily difficult. In contrast, the camera has no problem “seeing” the data to bring the puppets to life if the user’s hand is more or less flat to the camera. To operate the puppets’ mouths, the user simply moves his thumbs: up (adjacent to the palm, pointing in the same direction as the four fingers) for closed; down (away from the palm, at a right angle to the four fingers) for open.
Figure 3.Hand position to close the puppet’s mouth.
At first Diaz didn’t readily accept this sort of limiting hand position. After many days of coding and testing he actually was able to get the camera to recognize a standard “pinching” hand position to move a puppet’s mouth. (Again, think sock puppets.)
He experimented with an algorithm that essentially assumed that the closest points to the camera were the fingertips. Then, after tinkering with the 2D depth buffer, the algorithm looked for separation between these points to estimate the top and bottom of the hand. Ultimately, Diaz got the code to the point where users could control the puppets' mouths with their hands in a “natural” position. The C# Unity code in its entirety can be found in the Ultimate Coder Challenge: Going Perceptual week 4 contest blog post.
However, all the compensation and extrapolation of the incoming camera data spoiled much of the rest of the puppets' motion, including moving about in three dimensions in the on-screen environment or changing the direction the puppet is looking. (Some of the drama obviously drains away if the wolf is looking in another direction as it’s threatening the pig.)
Ultimately, requiring that users hold their hands upright, palms flat to the camera, was the right technical solution. The initial mandatory calibration helped ensure this approach was right for users—and was among the more innovative features of the app. The evolution of perceptual computing will depend in part on the establishment of conventions for controlling a computer and navigating apps. Gesture and facial recognition will eventually need to be as well understood as using a mouse to control a cursor and click and drag objects, or pinching and swiping on a touch screen. Indeed, the calibration won raves according to the comments from the contest judges excerpted on the Intel Ultimate Coder Challenge: Going Perceptual contest page. One judge exclaimed, “Keep it simple and educate users about perceptual computing in the first 2 seconds. This is the perceptual computing poster boy team!” Another judge wrote, “Best feedback, best gesture recognition...+1 for the awesome UI feedback on hand position + calibration step.”
public void UpdateRotation ()
{
if ( PuppetJoint == null ) return;
Quaternion baseQuat = new Quaternion();
Vector3 forwards = _playerController.ControllerObject.transform.forward;
Vector3 up = _playerController.ControllerObject.transform.up;
Vector3 right = _playerController.ControllerObject.transform.right;
float roll = 0.0f, pitch = 0.0f, yaw = 0.0f;
if ( ( InputBind == PXCInputType.ORIENTATION ) && _playerController.ControllerSettings.UseHandOpenness )
{
Vector3 lookDir = new Vector3( _playerController.SmoothHandNormal.x,
_playerController.SmoothHandNormal.z,
-_playerController.SmoothHandNormal.y);
Quaternion lookQuat = Quaternion.LookRotation( lookDir );
Vector3 eulerAngles = lookQuat.eulerAngles;
roll = ClampAngles( eulerAngles.z, Constraints.MinRoll, Constraints.MaxRoll );
pitch = ClampAngles( eulerAngles.x, Constraints.MinPitch, Constraints.MaxPitch );
yaw = ClampAngles( eulerAngles.y, Constraints.MinYaw, Constraints.MaxYaw );
}
else if ( InputBind == PXCInputType.OPENNESS )
{
// handle openness press
float handOpenness = _playerController.ControllerSettings.UseHandOpenness ?
_playerController.UseAdjustedThumbOpenness ?
_playerController.SmoothThumbOpenness :
_playerController.SmoothHandOpenness :
_playerController.SmoothPinchOpenness;
float inputVal = ( handOpenness / 100.0f ) * RotationMultiplier;
roll = Mathf.Clamp(inputVal, Constraints.MinRoll, Constraints.MaxRoll);
pitch = Mathf.Clamp(inputVal, Constraints.MinPitch, Constraints.MaxPitch);
yaw = Mathf.Clamp(inputVal, Constraints.MinYaw, Constraints.MaxYaw);
}
Figure 4.Screen shot of the PXCPuppetJointController code sample, which demonstrates how Sixense was able to control orientation and openness to a puppet joint (bone).
Use the SDK Plug-in to Speed Learning of the Unity Game Engine
One of the key decisions Woodall and Diaz faced was which game engine to use. Both had ample experience in AAA game development from their previous positions at Sega Studios.
However, after considering a few different engines, they decided to use Unity*, an engine they had never worked with before, in part because the Unity plug-in is included with the Intel Perceptual Computing SDK.
Figure 5.The basic components of the Puppet In Motion app.
"When we first fired up Unity, it was sort of eerie how familiar the interface and editor felt. It was just like the previous engine we worked with when we were at Sega. I mean, so much so that we were joking around, saying 'Wow, did somebody leave Sega and go over there and recreate this?'
“It's amazing to see what people have accomplished within a night's work within Unity. Given the time constraints of the contest, it was pretty clear to us that this was the way to go,” he continued. “It’s such a great engine to get your ideas prototyped and running quickly. Everything is so compartmentalized. You can attach scripts to game objects, and they’re so contained that with little effort, you’re up and running.”
The Intel plug-in gave Diaz a head start in using the Unity engine by exposing lots of Unity’s functionality in an easy-to-use interface. But getting up to speed with a new engine and using the plug-in presented challenges.
Diaz experienced occasional system crashes when using the Unity editor in the Intel Perceptual Computing SDK. And he found that at least some of the features in Unity’s C++ API weren’t accessible through the Intel plug-in.
using UnityEngine;
using System.Collections;
public class PXCManager : MonoBehaviour
{
public static PXCUPipeline PXCUPipelineInstance = new PXCUPipeline();
public static Texture2D PXCImage = null;
private static bool m_bPXCUPipelineInitialized = false;
private static int[] m_PXCImageSize = new int[2];
void Start()
{
Debug.Log( g_bPrimaryIsRight ? "Primary is Right" : "Primary is Left" );
if ( !m_bPXCUPipelineInitialized )
{
if ( !PXCUPipelineInstance.Init( m_PXCUPipelineMode ) )
{
print( "Unable to initialize the PXCUPipelinen" );
return;
}
else
{
m_bPXCUPipelineInitialized = true;
if ( !PXCUPipelineInstance.SetVoiceCommands( m_PXCVoiceCmds ) )
{
print( "Failed adding voice commands" );
}
}
}
if ( !PXCImage && PXCUPipelineInstance.QueryLabelMapSize( m_PXCImageSize ) )
{
PXCImage = new Texture2D( m_PXCImageSize[0], m_PXCImageSize[1], TextureFormat.RGB24, false );
}
}
void OnApplicationQuit()
{
PXCUPipelineInstance.Close();
PXCUPipelineInstance = null;
PXCImage = null;
}
Figure 6.A screenshot of access to the Unity* engine via the Intel® Perceptual Computing SDK plug-in.
Diaz and his colleagues fed all this information back to Intel, which set up the contest in the first place to encourage innovation around perceptual computing—among thought leaders and industry early adopters and within Intel itself. The contest, which was launched when the Intel Perceptual ComputingSDK was still in beta, was marked by an open-source-like level of collaboration and sharing; contestants blogged weekly about their experiences with the Intel Perceptual ComputingSDK, including issues that arose and solutions that worked to surmount them. Most of the contestants had the opportunity to meet face-to-face at the Game Developers Conference 2013 and borrowed freely from each other to collectively improve the resulting apps that the judges eventually considered.
The likely end result of all this sharing: improvement of the entire Intel Perceptual Computing SDK ecosystem over the long term, an outcome made all the more likely since Intel is using contest feedback such as Diaz’s to improve the Intel Perceptual Computing SDK technology going forward.
For Rapid Prototyping, Separate Artists and Coders ASAP
Diaz and Woodall were joined by three other Sixense colleagues in building the app. Designer chip Sbrogna set up the various scenes and oversaw the overall story arc. Art Director Dan Paullus and artist AJ Sondossi fleshed out the puppets—note surfer pig’s floppy ears and expressive eyes—and the rest of the app’s visual assets. Together this team represented more than a quarter of Sixense’s software development staff, though at most they worked only half-time on the contest app because of other projects and deadlines.
Figure 7.From left to right: Alejandro Diaz, Dan Paulus, Danny Woodall, and chip Sbrogna.
While Diaz and Woodall worked on the puppet controller system, the artists got started creating their models and the associated rigs, textures, and shaders. There was much communication upfront as the developers figured out precisely what parts of a virtual puppet could be animated by their puppet controller. Swiveling the head and moving the mouth to speak, yes; crouching or running, no.
However, once the main decisions about the puppet controller were made, the artists were free to work more or less independently. Woodall says the goal was to set up a production flow so that early on, the artists could start bagging all the bones they were interested in attaching to the controller. Bringing a puppet to life was simply a matter of importing it into the code base, dragging it into a scene, and attaching a script to it.
Figure 8.Wolf puppet character rig.
In addition to the puppet controller, the calibration system, and a fairly simple UI, Diaz created a director system that gave the designer and the artists the ability to set up the individual scenes and the locations of the imaginary camera capturing the action. The system also helped in transitions from one scene to the next, ensuring that each puppet was always where the user’s hand was relative to the camera as the pig and wolf moved from beach to forest to suburbia.
Diaz also incorporated AVPro Movie Capture*, a Unity plug-in available on the Unity Asset Store, to make it easy to record gameplay directly as AVI files. This is a common-sense feature in a world where the ability to post and share video from gameplay is expected.
using UnityEngine;
using System.Collections;
public class PuppetShowDirector : MonoBehaviour
{
public PuppetSet[] Sets = new PuppetSet[1];
public int ActiveSet = 1;
public bool EnableDebug = false;
private int m_currentSet = 0;
void Start ()
{
if ( ActiveSet <= Sets.Length )
{
m_currentSet = ActiveSet - 1;
}
Sets[m_currentSet].Init(this);
}
void Update ()
{
if ( m_currentSet != ActiveSet - 1 )
{
if ( ActiveSet <= Sets.Length && ActiveSet > 0 )
{
if ( Sets[m_currentSet].SetGameObject )
{
// disable and hide the current set
Sets[m_currentSet].SetGameObject.SetActive( false );
}
m_currentSet = ActiveSet - 1;
if ( Sets[m_currentSet].SetGameObject )
{
// enable and show the new active set
Sets[m_currentSet].SetGameObject.SetActive( true );
}
}
else
{
ActiveSet = m_currentSet + 1;
}
// initialize the new set
Sets[m_currentSet].Init(this);
}
if ( m_currentSet <= Sets.Length )
{
// update the current set
Sets[m_currentSet].Update();
}
}
Figure 9.A screenshot of the code that handles character transitions between each set.
Prepare for Perceptual Computing, the Future of Human-Computer Interaction
With high expectations for Sixense’s future, both in regard to Puppet In Motion and the rest of its activities, Woodall and Diaz have several ideas for improving the winning app and perhaps even releasing a version to the public, with help from Intel.
There’s a laundry list of features to add. At the top of the list: make it easier for a single user to put on a show with two puppets. For now, for the wolf to make good on his threat to a pig’s house or even to transition between scenes, a user has to either tap the touch screen or keyboard.
Woodall say the app’s basic technology could be used in several ways. One is creating content for children, including educational material. (What is Sesame Street if not teaching with puppets?) Then there are the unexpected use cases. For example, Woodall says that occasionally the Sixense team convenes on Skype*, sharing their desktops and interacting with each other as puppets.
“It’s interesting what happens in this sort of ‘sand-boxy’ environment,” he said. “You drop your guard and don't really care about being silly and self-conscious anymore; you become much more creative.”
All of which points to the fact that the world of perceptual computing, the one Intel is helping to create with the Intel Perceptual Computing SDK and a host of other activities, is fast approaching. Indeed, the leading edge is already here.
Developers looking for guidance might do well to follow the example of the three little pigs. As you’ll recall, the first two pigs, who built flimsy houses of straw and sticks, didn’t fare well. Only the third pig, with his stout house of bricks, prevailed against the wolf’s bluster.
So go on and build an app, one that’s sturdy enough to stand up to users who blunder about as they learn to move beyond the confines of the desktop GUI. Rely on industry-leading tools and hardware, especially from Intel. And most of all, take seriously the idea that new ways of controlling computers beyond keyboards and mice and even touch screens are fast moving from movies to niche applications to the mainstream.
Resources
- Read Woodall’s blog posts from the contest.
- See the videos of the calibration step and a beach scene starring surfer pig and the big bad wolf.
- See Intel’s interview with Sixense Entertainment at GDC 2013.
- See the C# Unity code Diaz wrote experimenting with pinch gesture mouth control; note that ultimately he opted for a different mouth control gesture (palm flat to the camera, thumb levered up and down to move the puppet's mouth).
- Download the Intel Perceptual Computing SDK (and order the Creative Interactive Gesture Camera Developer Kit).
Intel does not make any representations or warranties whatsoever regarding quality, reliability, functionality, or compatibility of third-party vendors and their devices. For optimization information, see software.Intel.com/en-us/articles/optimization-notice/. All products, dates, and plans are based on current expectations and subject to change without notice. Intel, the Intel logo, and Ultrabook are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Copyright © 2013. Intel Corporation. All rights reserved.