Facebook machine learning aims to modify faces, hands and... outfits




 

The latest research out of Facebook sets machine learning models to tasks that, to us, seem rather ordinary - but for a computer are still monstrously difficult. These projects aim to anonymize faces, improvise hand movements and - perhaps hardest of all - give credible fashion advice.

The research here was presented recently at the International Conference on Computer Vision, among a few dozen other papers from the company, which has invested heavily in AI research, computer vision in particular.

Modifying faces in motion is something we've all come to associate with "deepfakes" and other nefarious applications. But the Facebook team felt there was actually a potentially humanitarian application of the technology.

Deepfakes use a carefully cultivated understanding of the face's features and landmarks to map one person's expressions and movements onto a completely different face. The Facebook team used the same features and landmarks, but instead uses them to tweak the face just enough that it's no longer recognizable to facial recognition engines.

This could allow someone who, for whatever reason, wants to appear on video but not be recognized publicly to do so without something as clunky as a mask or completely fabricated face. Instead, they'd look a bit like themselves, but with slightly wider-set eyes, a thinner mouth, higher forehead and so on.

The system they created appears to work well, but would of course require some optimization before it can be deployed as a product. But one can imagine how useful such a thing might be, either for those at risk of retribution from political oppressors or more garden variety privacy preferences.

In virtual spaces it can be difficult to recognize someone at all - partly because of the lack of nonverbal cues we perceive constantly in real life. This next piece of research attempts to capture, catalog and reproduce these movements, or at least the ones we make with our hands.

It's a little funny to think about, but really there's not a lot of data on how exactly people move their hands when they talk. So the researchers recorded 50 full hours of pairs of people having ordinary conversations - or as ordinary as they could while suited up in high-end motion capture gear.

These (relatively) natural conversations, and the body and hand motions that went with them, were then ingested by the machine learning model; it learned to associate, for example, that when people said "back then" they'd point behind them, or when they said "all over the place," they'd make a sweeping gesture.

What might this be used for? More natural-seeming conversations in virtual environments, perhaps, but maybe also by animators who'd like to base the motions of their characters in real life without doing motion capture of their own. It turns out that the database Facebook put together is really like nothing else out there in scale or detail, which is valuable in and of itself.

Similarly unique, but arguably more frivolous, is this system meant to help you improve your outfit. If we're going to have smart mirrors, they ought to be able to make suggestions, right?


Fashion++ is a system that, having ingested a large library of images labeled with both the pieces worn (e.g. hat, scarf, skirt) and overall fashionability (obviously a subjective measure), can then look at a given outfit and suggest changes. Nothing major - it isn't that sophisticated - but rather small things like removing a layer or tucking in a shirt.

It's far from a digital fashion assistant, but the paper documents early success in making suggestions for outfits that real people found credible and perhaps even a good idea. That's pretty impressive, given how complex this problem proves to be when you really consider it, and how ill-defined "fashionable" really is.

Facebook's ICCV research shows that the company and its researchers are looking fairly broadly at the question of what computer vision can accomplish. It's always nice to detect faces in a photo faster or more accurately, or infer location from the objects in a room, but clearly there are many more obscure or surprising aspects of digital life that could be improved with a little visual intelligence. You can check out the rest of the papers here.

COMMENTS

More Related News

Facebook to launch officially licensed music videos in the US next month
Facebook to launch officially licensed music videos in the US next month

Facebook is preparing to launch officially licensed music videos on its social network in the U.S. next month, in a direct challenge to YouTube. In materials reviewed by TechCrunch, Facebook informed Page owners linked to artists they'll need to toggle on a new setting to add their music videos to their page ahead of an August 1st deadline, at which point Facebook will automatically create a page of their videos if no action had been taken. Artists will not have to manually upload their videos or even provide links, Facebook told the artist Page admins.

Fact check: Viral photo is from 2016 incident in Chile
Fact check: Viral photo is from 2016 incident in Chile

A photo claims to depict an Israeli soldier who is killing a Palestinian child. But the photo is from protests in Chile in 2016.

"Empire" actor Bryshere Gray arrested on domestic violence charges
"Empire" actor Bryshere Gray arrested on domestic violence charges
  • US
  • 2020-07-14 01:41:56Z

At approximately 10:15 p.m., the Goodyear Police Department responded to a call saying an adult female had been assaulted by her husband, the department said on Facebook.

Bedraggled bald eagle spent 24 hours stuck in a manure pit, Pennsylvania rescuers say
Bedraggled bald eagle spent 24 hours stuck in a manure pit, Pennsylvania rescuers say

"He needs to rest, get more hydration, and he should be able to be back in the skies soon."

Families of Italy
Families of Italy's virus dead seek answers, solace, justice
  • US
  • 2020-07-13 07:51:50Z

It started out as way for grief-struck families to mourn their coronavirus dead online: a Facebook group where relatives who were denied a funeral because of Italy's stringent lockdowns could share photos, memories and sorrow that their loved ones had died all alone. Lawyers for the Noi Denunceremo (We Will Denounce) Facebook group and an affiliated non-profit committee are filing 100 new cases Monday with Bergamo prosecutors investigating the outbreak, on top of 50 complaints lodged last month. The case files and Facebook posts paint a visceral portrait of the people swept up in Italy's devastating coronavirus outbreak, the first in the West: of mothers and fathers taken away by...

Leave a Comment

Your email address will not be published. Required fields are marked with *

Cancel reply

Comments

Top News: Economy