Cortana gets the power of sight in Microsoft’s future vision of conference rooms

Cortana can see you. Maybe. A mysterious, cone-shaped device, shown in a demo Monday at Microsoft’s Build event in Seattle, hinted at a big leap in capability for the company’s digital assistant: the ability to see and identify people.

In the ruthless war among Amazon’s Alexa, Apple’s Siri, and Google’s Assistant, this is a feature we haven’t yet seen from the competition. What we don’t know is what this prototype is, and whether it will ever come out.

Microsoft’s official, overshadowed demonstration showed us the conference room of the future. Microsoft incorporated Cortana, the Surface Hub, and Skype’s translation and transcription features, together with PowerBI and Microsoft Teams, into a cohesive working environment. 

In a presentation led by Raanah Amjadi, a marketing manager for Microsoft Teams, Microsoft showed off how simply scheduling the meeting room could be facilitated by Cortana, who was asked to set up a meeting with the “smart building team” to discuss issues with a hypothetical product.

The prototype device resembled the Harman/Kardon Invoke that was shown off in previous demos. According to Amjadi, however, this device could both see and hear. “It picks up both audio and video that you can pair with AI services and Microsoft 365 to help with key tasks,” Amjadi said, “like identification, voice transcription, and even real-time translation.”

build demo transcrpt and notes Mark Hachman / IDG

Microsoft’s cognitive services recorded and transcribed the conversation, taking notes as appropriate.

In the demo, as each attendee walked in, the device identified each person and greeted them by first name. When the leader, Amjadi, entered, the device also asked her whether she wanted to start the meeting. As Amjadi did so, a nearby Surface Hub began transcribing the conversation, separately taking meeting notes of key tasks. (Apparently, one of the trigger words was “follow up,” acting on Cortana’s existing capability to track emails and identify commitments or promises that you make to others.)

Above, a panoramic window showed all of the participants, apparently from the perspective of the prototype device. Microsoft’s AI services also provided a translation of the meeting for a colleague in China.

build demo translation Mark Hachman / IDG

Microsoft even provided more than one translation at a time.

Software, services connect employees

Microsoft’s Teams app stood in as a sort of back-channel communication between the participants, providing links to shared services such as GitHub or Trello, but also facilitating follow-ups that the team members wanted to set. In one example, Cortana stepped in to volunteer her services to find a convenient meeting time among the participants. Likewise, Teams was smart enough to know which “Q2 sales forecast” the participants wanted, tapping into a feature within Microsoft Delve for surfacing relevant documents.