This article first appeared in the March 2002 issue of DV magazine

By Chris Manners

Virtual characters certainly aren't a new phenomenon. Max Headroom, arguably the most famous, made his debut over 15 years ago. However, they remain fairly uncommon. Although mo-cap (motion capture) technology continues to advance, it's still mainly used to create prerendered sequences. But one cable television channel, TechTV, is pushing the boundaries of realtime virtual TV hosts.

Like many broadcast cable channels, TechTV (www.techtv.com) fills a specific niche-in its case, technology-based news programming. For our purposes, what's unique about TechTV is its Virtual Media Department's (VMD) two virtual characters. The VMD integrates a full mo-cap studio within a broadcast TV operation. I'm a part-time member of the department, and I've been impressed with what I've seen.

The VMD was started in 1996 by Nancy Juliber for TechTV's precursor, Ziff-Davis Television (ZDTV). ZDTV developed a single character, Dev Null, for a show called The Site. Dev Null made his MSNBC debut in July 1996 delivering anecdotal Web information while acting as a foil for the show's host, Soledad O'Brien.

At the same time, Dev Null provided tech how-to information, industry gossip, and answered viewer mail. This initial virtual character project was run on Alive software from Protozoa (www.protozoa.com) in a system the company provided as a single unit. Dev Null was driven by a single person who worked as both voice talent and body performer. The character was a small presence on the weekly hour-long show, which ran until August 1997, but the experience let VMD further develop a mo-cap environment appropriate for broadcast television. For a list of companies working on and with virtual character technology, see the online version of this story on DV.com.

When Ziff-Davis launched its full 24-hour ZDTV channel in March 1998, Dev Null was replaced with a pair of virtual hosts-the female Tilde and the male Dash. The hosts were supposed to represent the network and, because the Web was still new to many people, provide information about the Internet. Tilde provided programming guides and was tied into ZDTV's branding campaign. Dash had a number of roles, initially focusing on the Internet and acting as a Web surfer. As these characters developed, the VMD built a full-blown mo-cap set with a virtual character stage.

Over time, Tilde became a strong personality who represented the TechTV network. Dash, on the other hand, morphed into the network's animation expert, providing behind-the-scenes information about 2D and 3D technologies. Today Tilde is the ambassador of the prime time block and hosts some international shows, as does Dash.

For domestic prime time, Tilde fronts bumpers, lineup spots, and 30-second and 60-second spots. As a character, Tilde is pretty opinionated about the state of technology. Dash hosts Extended Play International, a weekly one-hour show focusing on video and computer games. Dash may also host a new show called Dash's Animation House. As of press time, this half-hour weekly program showcasing outstanding animation from around the world is scheduled to debut domestically in January 2002. The show will consider including work from amateur animators. You can find submission information on TechTV's Web site.

TechTV's Slater Penney performs all body movement for the channel's two virtual characters on a magnetic stage running Ascension Technology's MotionStar motion-capture program.

Model citizens

When it was developing the two characters, VMD chose to move away from Protozoa's Alive software mainly because MediaLab provided new options for realism and movement. The Alive system used a single person in a mo-cap suit with facial animation controls residing in the hands. This limited the number of gestures the character could perform, and the hands were often off-screen. Clovis, on the other hand, uses two performers who work together-a body performer on a motion capture stage and a voice artist who speaks in a sound booth and also performs the facial puppeteering. While the body performer acts out the character's movements, the voice talent both reads the script and animates the facial expressions with a set of mo-cap gloves.

TechTV saw other advantages to using the Clovis system. Data capture is faster, as are the frame rates the system can achieve. The Alive system ran at around 24fps, but Clovis can run 40fps to 50fps. Clovis can support shadows and has fewer latency issues. And TechTV considers the Clovis interface easier to use, despite its French-language interface. Both systems, however, generate a realtime matte for compositing a moving character over a video background.

With Clovis, the body actor performs the character's movements while standing on a 25x25-foot magnetic stage wearing a full-body mo-cap suit with 18 tracking sensors placed at key body points: the feet, ankles, knees, thighs, hips, chest, shoulders, elbows, and head. The suit includes a pair of mo-cap gloves to animate the character's hands. The stage itself is part of the MotionStar motion-capture system developed by Ascension Technology Corporation (www.ascensiontech.com). The system's sensors, named Flock of Birds, are each calibrated separately through a Windows-based system. The system's data capture box moves the motion data over to the SGI Onyx 2, where Clovis applies the data to the prebuilt character models.

Each virtual character's face has 10 control points, all driven by the voice actor's mo-cap gloves. Each glove has three sets of controls that operate the facial animation. These controls include standard lip shapes that represent the common phonemes, such as "Oooh" and "Aaah." The gloves also let the voice actor open and close the character's mouth, blink, and raise its eyebrows.

Each character can make a number of facial expressions, such as furrowing its brows to look angry and turning its mouth down at the corners, an statement that looks particularly French.

The voice performers also operate a number of foot controls that resemble guitar volume pedals. These pedals perform some less frequently used features, including raising a single eyebrow and locking or unlocking the character's eyes to the camera.

Laura LeBleu, the voice of TechTV's virtual character Tilde, wears gloves that she uses to animate the character's facial features. Tilde's reflection in the window is from the monitor that gives LeBleu realtime feedback on her performance.

Performance setup

Once the sensor positions have been set and tested, the team selects the characters' virtual clothing. Although both Tilde and Dash have a default outfit, the Clovis system makes it easy to redress each character with a variety of sartorial options. By default, Dash wears black pants and a black shirt, but his virtual closet also holds a safari suit, a sport coat with white shirt, and a turtleneck. Tilde has a wider wardrobe, including pants, shirts, dresses, skirts, and a variety of footwear. For accessories, Dash has a removable goatee, eyeglasses, and sunglasses. Tilde has earrings, bracelets, a cell phone, tattoo, and some other accessories. The clothing options are essentially texturing changes, which are already built into the models. But for Tilde there are some geometry variations because pants and dresses are different shapes. In addition, Tilde also has hair that moves when she walks.

Next the VMD sets the lighting and camera options. By default, the characters are lighted fairly evenly from five sources. However, usually only three lights are used because more lights can slow down the frame rate. The lights can be moved in all planes in 3D, and lighting schemes can be saved as a preset. The camera also moves in 3D; however, most shots use a few set angles with specific backgrounds.

The performance itself is an impressive piece of choreographed collaboration. Performing the voice-overs isn't easy, but the puppeteering is considerably more specialized. To that end, TechTV collaborated with Jim Henson's Creature Shop to train the voice artists and the body performers.

The voice artist reads the script while twitching and fluttering the gloved hands. The body performer simultaneously walks around the stage, gesturing in sync. It's certainly a strange experience to see the disassociation between the body and the voice, but it works well in realtime. Four monitors surround the mo-cap stage at roughly eye level, so the body performer can see what's being captured to tape from any angle. The voice-over talent also has a monitor directly in front of the sound booth's window. Typically each block requires three or four takes before the producer is happy.

There are, however, a number of limitations with TechTV's current system. The body performers must take care that the characters' arms don't penetrate the characters' torsos. The system isn't exact, so it's very difficult for the characters to do things like clasp their hands together or rest a hand on one hip.

Nonetheless, together the two performers usually create a seamless character. Once the body performer has become accustomed to the voice talent's way of speaking and inflections, he can anticipate certain moves. And because the Clovis system has a sufficiently fast frame rate, it can update both body and the face on-screen simultaneously.

During production, the performances are captured live to videotape. All of the character environments are recorded onto digital tapes and are looped and played back in realtime as the on-camera performance takes place.

The recording setup includes a switcher that separates the realtime matte from the characters' motion and several Panasonic AJ-D950 decks recording a DVCPRO50 signal to tape. One deck plays back the background environment, a second records only the realtime mask generated in Clovis, and a third deck records the composited RGB character data and the video background.

TechTV's Vitual Media Department uses Panasonic AJ-D950 DVCPRO50 VTRs to play and record each shot's video background, realtime character matte, and final composited video.

Environments and post

One great advantage of TechTV's mo-cap facility environment is that production is fast. Tilde's schedule involves creating a 30-minute show four times a week. Dash hosts one 60-minute show a week, and preproduction has already begun for the season premiere of his new animation program. With the performance occurring in realtime, sometimes with a real background, production schedules and style get closer to those of standard television, even including interviews with real people. Working with this setup doesn't require a camera operator or lighting technician on the set, thus keeping costs down.

Prior to the shoot and performance, a production assistant goes through a full day's worth of TechTV content and logs the timecode for appropriate spots. From this log, scripts are generated. After the performances have been captured, they are edited the same day on one of TechTV's Avid Media Composer 8000 systems. Tapes with the full composite performance are digitized and host wraps are placed into templates. With this speedy production arrangement, the body performer and voice talent don't see the script until the day of the shoot.

TechTV's virtual characters are shot with a bluescreen background that is then composited onto a virtual set, as shown in this sequence from the <I>Zip File</I> show, hosted by the virtual character Tilde.

Recent developments

The VMD consists of only a handful of staff. Technical Producer Karsten Bondy runs the Clovis system and VTRs. He also develops some video background sets. Senior Producer Mark Devito produces content for the characters, directs in-house shoots, and generates art environments. Devito has headed the group for about two years and currently devotes most of his time to Dash's Animation House.

The department includes a production assistant, two voice performers, and the body artist. Producer Laura LeBleu provides Tilde's voice, writes Tilde's scripts, and assists with the development of Dash's Animation House. Slater Penney performs the body movements for both characters. As a part-time staff member, I provide the voice for Dash. Beyond each person's defined job, everyone in the department adds their input about the show's content and scripts.

(far right)The Virtual Media Department's control console holds a monitor for the Ascension data-capture system, (right) a large monitor for the SGI-hosted Clovis virtual character system, (left and far left) and two monitors displaying the composited live video.

While the VMD is working on the upcoming Dash's Animation House, its main focus is to extend the characters' international perspective. Tilde will be involved with new projects for the company's Asian markets, while Dash will be included in additional short-form projects that accompany his main show.

The long-term goal is to have Dash's Animation House provide solid and entertaining programming for TechTV's Tech Prime block. This new show will be written, produced, and edited exclusively by the VMD. Although this is a new challenge, the clips already developed for the season premiere suggest that this is definitely one challenge the VMD is ready to face.

TechTV frequently composites one of its virtual characters, in this case Dash, into a live real-world scene shot with a locked-down camera.

Chris Manners is the director of design at LimeVoodoo Productions, a San Francisco–based digital design firm, and he sincerely apologizes to his wife for the countless hours of soccer babble.