Tag: disney research

Posted on

Flexible expressions could lift 3D-generated faces out of the uncanny valley

3D-rendered faces are a big part of any major movie or game now, but the task of capturing and animated them in a natural way can be a tough one. Disney Research is working on ways to smooth out this process, among them a machine learning tool that makes it much easier to generate and manipulate 3D faces without dipping into the uncanny valley.

Of course this technology has come a long way from the wooden expressions and limited details of earlier days. High resolution, convincing 3D faces can be animated quickly and well, but the subtleties of human expression are not just limitless in variety, they’re very easy to get wrong.

Think of how someone’s entire face changes when they smile — it’s different for everyone, but there are enough similarities that we fancy we can tell when someone is “really” smiling or just faking it. How can you achieve that level of detail in an artificial face?

Existing “linear” models simplify the subtlety of expression, making “happiness” or “anger” minutely adjustable, but at the cost of accuracy — they can’t express every possible face, but can easily result in impossible faces. Newer neural models learn complexity from watching the interconnectedness of expressions, but like other such models their workings are obscure and difficult to control, and perhaps not generalizable beyond the faces they learned from. They don’t enable the level of control an artist working on a movie or game needs, or result in faces that (humans are remarkably good at detecting this) are just off somehow.

A team at Disney Research proposes a new model with the best of both worlds — what it calls a “semantic deep face model.” Without getting into the exact technical execution, the basic improvement is that it’s a neural model that learns how a facial expression affects the whole face, but is not specific to a single face — and moreover is nonlinear, allowing flexibility in how expressions interact with a face’s geometry and each other.

Think of it this way: A linear model lets you take an expression (a smile, or kiss, say) from 0-100 on any 3D face, but the results may be unrealistic. A neural model lets you take a learned expression from 0-100 realistically, but only on the face it learned it from. This model can take an expression from 0-100 smoothly on any 3D face. That’s something of an over-simplification, but you get the idea.

Computer generated faces all assume similar expressions in a row.

Image Credits: Disney Research

The results are powerful: You could generate a thousand faces with different shapes and tones, and then animate all of them with the same expressions without any extra work. Think how that could result in diverse CG crowds you can summon with a couple clicks, or characters in games that have realistic facial expressions regardless of whether they were hand-crafted or not.

It’s not a silver bullet, and it’s only part of a huge set of improvements artists and engineers are making in the various industries where this technology is employed — markerless face tracking, better skin deformation, realistic eye movements, and dozens more areas of interest are also important parts of this process.

The Disney Research paper was presented at the International Conference on 3D Vision; you can read the full thing here.

Disney Research neural face-swapping technique can provide photorealistic, high-resolution video

Posted on

This autonomous spray-painting drone is a 21st-century tagger’s dream

Whenever I see an overpass or billboard that’s been tagged, I worry about the tagger and the danger they exposed themselves to in order to get that cherry spot. Perhaps this spray paint-toting drone developed by ETH Zurich and Disney Research will take some of the danger out of the hobby. It also could be used for murals and stuff, I guess.

Although it seems an obvious application in retrospect, there just isn’t a lot of drone-based painting being done out there. Consider: A company could shorten or skip the whole scaffolding phase of painting a building or advertisement, leaving the bulk of painting to a drone. Why not?

There just isn’t a lot of research into it yet, and like so many domain-specific applications, the problem is deceptively complex. This paper only establishes the rudiments of a system, but the potential is clearly there.

The drone used by the researchers is a DJI Matrice 1002, customized to have a sensing rig mounted on one side and a spraying assembly on the other, counterbalancing each other. The sprayer, notably, is not just a nozzle but a pan-and-tilt mechanism that allows details to be painted that the drone can’t be relied on to make itself. To be clear, we’re still talking broad strokes here, but accurate to an inch rather than three or four.

It’s also been modified to use wired power and a constant supply of paint, which simplifies the physics and also reduces limits on the size of the surface to be painted. A drone lugging its own paint can wouldn’t be able to fly far, and its thrust would have to be constantly adjusted to account for the lost weight of sprayed paint. See? Complex.

The first step is to 3D scan the surface to be painted; this can be done manually or via drone. The mesh is then compared to the design to be painted and a system creates a proposed path for the drone.

Lastly the drone is set free to do its thing. It doesn’t go super fast in this prototype form, nor should it, since even the best drones can’t stop on a dime, and tend to swing about when they reduce speed or change direction. Slow and steady is the word, following a general path to put the nozzle in range of where it needs to shoot. All the while it is checking its location against the known 3D map of the surface so it doesn’t get off track.

In case you’re struggling to see the “bear,” it’s standing up with its paws on a tree. That took me a long time to see, so I thought I’d spare you the trouble.

Let’s be honest: This thing isn’t going to do anything much more complicated than some line work or a fill. But for a lot of jobs that’s exactly what’s needed — and it’s often the type of work that’s the least suited to skilled humans, who would rather be doing stuff only they can do. A drone could fill in all the easy parts on a building and then the workers can do the painstaking work around the windows or add embellishments and details.

For now this is strictly foundational work — no one is going to hire this drone to draw a Matterhorn on their house — but there’s a lot of potential here if the engineering and control methods can be set down with confidence.

Posted on

Disney tech smooths out bad CG hair days

Disney is unequivocally the world’s leader in 3D simulations of hair — something of a niche talent in a way, but useful if you make movies like Tangled, where hair is basically the main character. A new bit of research from the company makes it easier for animators to have hair follow their artistic intent while also moving realistically.

The problem Disney Research aimed to solve was a compromise that animators have had to make when making the hair on characters do what the scene requires. While the hair will ultimately be rendered in glorious high definition and with detailed physics, it’s too computationally expensive to do that while composing the scene.

Should a young warrior in her tent be wearing her hair up or down? Should it fly out when she turns her head quickly to draw attention to the movement, or stay weighed down so the audience isn’t distracted? Trying various combinations of these things can eat up hours of rendering time. So, like any smart artist, they rough it out first:

“Artists typically resort to lower-resolution simulations, where iterations are faster and manual edits possible,” reads the paper describing the new system. “But unfortunately, the parameter values determined in this way can only serve as an initial guess for the full-resolution simulation, which often behaves very different from its coarse counterpart when the same parameters are used.”

The solution proposed by the researchers is basically to use that “initial guess” to inform a high-resolution simulation of just a handful of hairs. These “guide” hairs act as feedback for the original simulation, bringing a much better idea of how the rest will act when fully rendered.

The guide hairs will cause hair to clump as in the upper right, while faded affinities or an outline-based guide (below, left and right) would allow for more natural motion if desired.

And because there are only a couple of them, their finer simulated characteristics can be tweaked and re-tweaked with minimal time. So an artist can fine-tune a flick of the ponytail or a puff of air on the bangs to create the desired effect, and not have to trust to chance that it’ll look like that in the final product.

This isn’t a trivial thing to engineer, of course, and much of the paper describes the schemes the team created to make sure that no weirdness occurs because of the interactions of the high-def and low-def hair systems.

It’s still very early: it isn’t meant to simulate more complex hair motions like twisting, and they want to add better ways of spreading out the affinity of the bulk hair with the special guide hairs (as seen at right). But no doubt there are animators out there who can’t wait to get their hands on this once it gets where it’s going.

Posted on

Stickman is Disney’s new headless acrobatic robot

The team at Disney Research never fails to deliver fascinating (if not always particularly useful) experiments. Take Stickman. The robot is essentially one long limb, capable of some cool acrobatic maneuvers.

The system, detailed in a new paper from DR titled “Towards a Human Scale Acrobatic Robotic,” has two degrees of freedom and a pendulum it uses to launch itself in the air after swinging on a rope. The relatively simple robot tucks and folds, somersaulting in the air before landing on the padding below.

Those aerials are executed courtesy of a built-in laser range finder and six axis inertial measurement unit (a combination gyroscope/accelerometer), which calculate its position in-flight and adjust its positioning accordingly.

“Stickman emulates the behavior of human performers using a very limited set of sensing and actuation capabilities,” the team writes in the paper. “It is able to successfully perform several different somersaulting stunts by changing initial orientation and the timing of tuck, release, and untuck.”

The team says it’s going to continue experimenting with the robot, in an attempt to create more complex stunts down the road. No word on future plans beyond that, but for this headless acrobat of a robot, the sky, it seems, is the limit.

Posted on

This soft robotic arm is straight out of Big Hero 6 (it’s even from Disney)

The charming robot at the heart of Disney’s Big Hero 6, Baymax, isn’t exactly realistic, but its puffy bod is an (admittedly aspirational) example of the growing field of soft robotics. And now Disney itself has produced a soft robot arm that seems like it could be a prototype from the movie.

Created by Disney Research roboticists, the arm seems clearly inspired by Baymax, from the overstuffed style and delicate sausage fingers to the internal projector that can show status or information to nearby people.

“Where physical human-robot interaction is expected, robots should be compliant and reactive to avoid human injury and hardware damage,” the researchers write in the paper describing the system. “Our goal is the realization of a robot arm and hand system which can physically interact with humans and gently manipulate objects.”

The mechanical parts of the arm are ordinary enough — it has an elbow and wrist and can move around the way many other robot arms do, using the same servos and such.

But around the joints are what look like big pillows, which the researchers call “force sensing modules.” They’re filled with air and can detect pressure on them. This has the dual effect of protecting the servos from humans and vice versa, while also allowing natural tactile interactions.

“Distributing individual modules over the various links of a robot provides contact force sensing over a large area of the robot and allows for the implementation of spatially aware, engaging physical human-robot interactions,” they write. “The independent sensing areas also allow a human to communicate with the robot or guide its motions through touch.”

Like hugging, as one of the researchers demonstrates:

Presumably in this case the robot (also presuming the rest of the robot) would understand that it is being hugged, and reciprocate or otherwise respond.

The fingers are also soft and filled with air; they’re created in a 3D printer that can lay down both rigid and flexible materials. Pressure sensors within each inflatable finger let the robot know whether, for example, one fingertip is pressing too hard or bearing all the weight, signaling it to adjust its grip.

This is still very much a prototype; the sensors can’t detect the direction of a force yet, and the materials and construction aren’t airtight by design, meaning they have to be continuously pumped full. But it still shows what they want it to show: that a traditional “hard” robot can be retrofitted into a soft one with a bit of ingenuity. We’re still a long way from Baymax, but it’s a more science than fiction now.

Posted on

Robot posture and movement style affects how humans interact with them

It seems obvious that the way a robot moves would affect how people interact with it, and whether they consider it easy or safe to be near. But what poses and movement types specifically are reassuring or alarming? Disney Research looked into a few of the possibilities of how a robot might approach a simple interaction with a nearby human.

The study had people picking up a baton with a magnet at one end and passing it to a robotic arm, which would automatically move to collect the baton with its own magnet.

But the researchers threw variations into the mix to see how they affected the forces involved, how people moved and what they felt about the interaction. The robot had two types each of three phases: movement into position, grasping the object and removing it from the person’s hand.

For movement, it either started hanging down inertly and sprung up to move into position, or it began already partly raised. The latter condition was found to make people accommodate the robot more, putting the baton into a more natural position for it to grab. Makes sense — when you pass something to a friend, it helps if they already have their hand out.

Grasping was done either quickly or more deliberately. In the first condition the robot’s arm attaches the magnet as soon as it’s in position; in the second, it pushes up against the baton and repositions it for a more natural way to pull out. There wasn’t a big emotional difference here, but opposing forces were much less in the second grasp type, perhaps meaning it was easier.

Once attached, the robot retracted the baton either slowly or more quickly. Humans preferred the former, saying that the latter felt as if the object was being yanked out of their hands.

The results won’t blow anyone’s mind, but they’re an important contribution to the fast-growing field of human-robot interaction. Soon there ought to be best practices for this kind of thing for when we’re interacting with robots that, say, clear the table at a restaurant or hand workers items in a factory. That way they’ll be operating with the knowledge that they won’t be producing any unnecessary anxiety in nearby humans.

A side effect of all this was that the people in the experiment gradually seemed to learn to predict the robot’s movements and accommodate them — as you might expect. But it’s a good sign that even over a handful of interactions a person can start building a rapport with a machine they’ve never worked with before.