‘Mario in Real Time’ shows off new software toolkit

If you went to the Consumer Electronics Show in January, you may have wondered how on earth Nintendo got a 3D, on-screen version of Mario to chat with the crowd at its booth. Then again, it looked so natural you may not have even thought twice about it.

What you were seeing, rather surprisingly, was not a Nintendo product at all — it was a real-time character animation system by SimGraphics Engineering Corp. of Pasadena, CA.

Though it is hardly ever given credit for doing so, SimGraphics is the company that actually wrote the Unix software drivers for VPL Research’s Data Glove that launched the “virtual reality” phenomenon. It also invented and sells the 3D “Flying Mouse” that’s part of the real-time animation system and is used to navigate through other 3D software applications.

NEW TECHNOLOGY FOR THEME PARKS

SimGraphics’s latest foray into real-time animation may have an equally revolutionary effect on a new but potentially explosive market for theme parks and other location-based entertainment attractions — and maybe even for film and TV broadcast as well. (There are many projects under way along these lines that are really quite remarkable in scope. We hope to report on them soon.)

In fact, the system was so interesting that SimGraphics vice president Steve Glenn says Nintendo’s public relations execs asked SimGraphics not to give out film clips about the system at the show — they were afraid it would get more attention than Mario.

Called PAS, for Performance Animation System, Glenn says the system is a software toolkit for creating real-time character animation applications. PAS supports what Glenn calls “real-time morphing,” first developed by electronic puppeteers deGraf/Wahrman. SimGraphics, however, is first to drive face animation using an operator’s actual face movement, and to provide a toolkit for it.

MORPHING FACIAL EXPRESSIONS

Morphing has recently entered the vernacular outside the computer graphics community because of Michael Jackson’s recent video which “morphs” between various dancers, giving the illusion that their faces are actually melting and changing into one another.

But at the summer CES show, the morphing was between different expressions on one character’s face, not between different faces. And was done live, on the spot, not in a computer graphics studio. At the next CES show, an improved version of the MIRT (Mario in Real Time) software will allow Mario to morph into various Nintendo products.

Here are the components of the system SimGraphics used for Nintendo’s pavilion at CES:

• A “Waldo”(the trademarked term for this particular kind of facial armature) consisting of eight digital encoders, linked to plastic cups and then to the operator’s face: two on the forehead, two on jaws, two on the upper lip, one on the chin and one on the lower lip (the system can support 12 sensors); two joysticks, a 3D mouse (for head motion tracking) and a foot pedal for eye motion.

• Silicon Graphics 420 VGX workstation for data interpolation and image generation, using MIRT designed by SimGraphics using PAS.

• Graphics databases in Wavefront format developed by Rhythm & Hues using line art from Nintendo. The databases include 15 different morph targets — i.e., neutral, angry, happy, sad, surprised — including mouth phonemes.

The man behind the curtain. An actor wore the Waldo (created by animatronics expert Rick Lazzarini of the Character Shop in Burbank, his first computer animation job) that tracked his eyebrow, cheek, head, chin and lip movements, and allowed him to control Mario’s corresponding features.

The Waldo’s sensors tracked the actor’s facial movements and mouth phonemes, generating data that was collected and interpreted by SimGraphics’ software. That data was then projected in full 3D color onto a large screen in the Nintendo pavilion.

“Audience interaction stations” in the Nintendo booth were equipped with monitors and microphones so the actor could observe crowd reaction, and hear and ask questions. The result? Super Mario “himself,” yukking it up with the audience and looking as “real” as a character can look on screen. The closest analogy is chatting up Mickey Mouse while walking around at Disneyland, but the giant Mickeys can’t move their faces. PAS’s effects are actually more realistic.

REALISM — AND RETAKES

Even Pixar’s fabulous animated baby in Tin Toy looks computer-generated and decidedly unrealistic, and Glenn says there are many reasons why. One, he says, is that frame-by-frame animation either creates each frame separately, or uses key frames –starting a character’s movement here, marking that you want it to end there, and letting the computer draw and link the frames in-between — or creates rules and algorithms that create movement.

In any case, it’s very time consuming and expensive — and the result is movement that doesn’t look very realistic. You can’t “direct” it, in the auteur sense, either — ask Pixar how hard and expensive it is to keep programming a certain scene over and over again for emotional impact. This electronic version of the “retake” adds an enormous amount of time and expense to a process that is already grueling.

PAS allows all of these things — realistic movement (since it is actually real movement), direction and cost-effective retakes. You don’t like how it looks? Adjust a sensor, maybe, and have the actor do it again.

A ‘FIRST’ WORTH NOTING

Despite how easy PAS makes computer animation look, this is obviously not yet a desktop technology. The SGI workstation alone, required for the high-powered graphics crunching, is a cool $120,000 (though Glenn says SimGraphics is in the process of porting the software to a specially optimized version of SGI’s Indigo Elan).

Using PAS, SimGraphics will custom-develop runtime character animation applications. Developing an application (pre-production) costs anywhere from $80,000 to $100,000, including database development and configuration costs. Production costs include buying or leasing the SGI hardware, specialized hardware required by PAS (like Waldo) and a/v equipment.

NOT JUST THEME PARKS

Because it is both more cost effective and provides palpably improved realism, Glenn believes that PAS has great potential for animating characters in interactive multimedia and video games, as well as for film and TV.

TV production companies, trade show producers, rock band managers and theme parks have all contacted the company. “We’ve even had bids on a couple of projects that would involve permanent installations at a site like a stadium or a very large store,” says Glenn.

And despite its high cost today, it’s clear that products like PAS will only get cheaper as time goes on. The only thing that’s real about virtual reality today is that many, many companies — including U.S. movie studios and oodles of Japanese firms — are investing serious amounts of money in the technology for location-based entertainment.

As these systems congeal over time, just like everything else they will get cheaper. And eventually, maybe, we’ll have VR stations at home just as we have component stereos set up today. In any case, the animation system that SimGraphics has invented today is bound to go down in the history discs as one of those “firsts” worth noting.

Denise Caruso