Home / News & Analysis / Facebook’s AI researchers task ‘tourist bots’ with finding their way in NYC

Facebook’s AI researchers task ‘tourist bots’ with finding their way in NYC

Facebook is getting guide bots to help tourist bots explore Hell’s Kitchen in a virtual New York City. It’s not just for kicks either, a new research paper published today by FAIR is looking to examine how AI systems can orient themselves and communicate observed data better than humans can.

The setup for Facebook’s “Walk the Talk” research experiment involves throwing a “tourist” bot onto a random street corner of NYC and getting a “guide” bot to direct them to a spot on a 2D map. This involved Facebook capturing 360 photos of a bunch of different street corners in random spots in NY and feeding them to the AI tourist bot who then had to peer around at the behest of the guide agent who would gain a sense of where the tourist was based and try to direct them through a text conversation.

It’s indeed quite the novel experiment, which plays out like this in practice.

Guide:    Hello, what are you near?
Tourist:  Hello, in front of me is a Brooks Brothers
Guide:    Is that a shop or restaurant?
Tourist:  It is a clothing shop.
Guide:    You need to go to the intersection in the northwest corner of the map
Tourist:  There appears to be a bank behind me.
Guide:    Ok, turn left then go straight up that road

Facebook isn’t doing all of this to give you a virtual guide in some unannounced mapping product, this is Facebook AI Research as opposed to their applied machine learning arm so this stuff really resides in the long-term, less product centric sphere. What this experiment is helping Facebook’s AI researchers approach is a concept called “Embodied AI.”

Talk the Walk: AI "tourist" and "guide" demonstration

Posted by Facebook Engineering on Tuesday, July 10, 2018

Embodied AI basically entails giving AI models the ability to learn while on-the-go gathering data that is present around them that can help them make sense of what they already do know. In “Talk the Walk,” the guide AI bot had all of this 2D map data and the tourist bot had all of this rich 360 visual data but it was only through communication with each other that they were able to carry out their directives.

The real goal was to work on the two agents gathering information through natural language, but the researchers found that the bots did a better job of completing the task when they used “synthetic language” which relied more on them using more simplistic symbols to convey information and location. This less natural way of communicating data not only outperformed a more human-like chat, it also enable the bots to find their way more concisely than humans would in a natural language chat.

What made this environment particularly difficult was the fact that it was the real world. The 360 snapshots were, of course, much more cluttered than what would appear in a simulated model that a lot of these experiments would typically run in. Putting this into words is hard enough when two humans are already vaguely familiar with a location, for two bots that have access to different data, this can be awfully difficult to communicate efficiently.

To tackle this, Facebook built a mechanism called MASC (Masked Attention for Spatial Convolution) that basically enables these language models the agents are running to quickly parse what the keywords are in responses that are probably the most critical to getting a sense of what’s being conveyed. Facebook said that utilizing the process doubled the accuracy of results that were being tested.

For Facebook’s part, this is foundational research in that it seems to raise far more questions than it seeks to answer about best practices, but even grasping at those is an important milestone and a good direction to point the broader community in taking on hard problems that need to be tackled the company’s researchers say.

“If you really want to solve all of AI, then you probably want to have these different modules or components that solve different subproblems,” Facebook AI research scientist Douwe Kiela told me. “In that sense this is really a challenge to the community asking people how they would solve this and inviting them to work with us in this exciting new research direction.”

Read more

Check Also

Twitter is holding off on fixing verification policy to focus on election integrity

Twitter is pausing its work on overhauling its verification process, which provides a blue checkmark to public figures, in favor of election integrity, Twitter product lead Kayvon Beykpour tweeted today. That’s because, as we approach another election season, “updating our verification program isn’t a top priority for us right now (election integrity is),” he wrote on Twitter this afternoon. Last November, Twitter paused its account verifications as it tried to figure out a way to address confusion around what it means to be verified. That decision came shortly after people criticized Twitter for having verified the account of Jason Keller, the person who organized the deadly white supremacist rally in Charlottesville, Virginia. Fast forward to today, and Twitter still verifies accounts “ad hoc when we think it serves the public conversation & is in line with our policy,” Beykpour wrote. “But this has led to frustration b/c our process remains opaque & inconsistent with our intented [sic] pause.” While Twitter recognizes its job isn’t done, the company is not prioritizing the work at this time — at least for the next few weeks, he said. In an email addressed to Twitter’s health leadership team last week, Beykpour said his team simply doesn’t have the bandwidth to focus on verification “without coming at the cost of other priorities and distracting the team.” The highest priority, Beykpour said, is election integrity. Specifically, Twitter’s team will be looking at the product “with a specific lens towards the upcoming elections and some of the ‘election integrity’ workstreams we’ve discussed.” Once that’s done “after ~4 weeks,” he said, the product team will be in a better place to address verification. We've heard some questions recently about the status of Verification on Twitter, so wanted to address directly. Updating our verification program isn’t a top priority for us right now (election integrity is). Here’s some history & context, and how we plan to put it on our roadmap — Kayvon Beykpour (@kayvz) July 17, 2018

Leave a Reply

Your email address will not be published. Required fields are marked *

Disclaimer: Trading in bitcoins or other digital currencies carries a high level of risk and can result in the total loss of the invested capital. theonlinetech.org does not provide investment advice, but only reflects its own opinion. Please ensure that if you trade or invest in bitcoins or other digital currencies (for example, investing in cloud mining services) you fully understand the risks involved! Please also note that some external links are affiliate links.