Conversations with Cortana

I wrote a little in this Link Removed about the basics of integrating Cortana into a Windows Phone 8.1 application with my specific example at the time being one of using the Bing synonym API service to bring back synonyms for a word that the user had spoken.

Ultimately, this boiled down to being able to hold down the search button on the phone to wake up Cortana and then being able to say something like;

“SYNONYMS FOR {WORD}”

where {WORD} is the placeholder for whatever word the user wanted to get a list of synonyms for and the app then goes off and hits the Bing synonym API to get that list and display it in front of the user.

That’s all good and the platform does 99% of the work for me in getting that done but there’s a scenario where I might want to continue the conversation.

As an example, I might want to have an app that took speech like;

“PICTURES OF {NOUN}” ( phrase 1 )

where {NOUN} is the subject of the pictures that the user wants to search for – for example, “cars” - but they may want to carry on this interaction with something like;

“EXCLUDE {FILTER}” ( phrase 2 )

where {FILTER} is the set of results that they want to take out of what’s already been brought back – for example, “red cars”.

It’s something that I’ve been wanting to experiment with but it has a number of implications including;

1. How long the app continues to listen after ( phrase 1 ) above for a potential follow-on ( phrase 2 ) given that there’s more than likely a cost to listening in terms of battery usage and also given that if the user puts their phone down on the desk for 5 minutes then they are probably not expecting that the app is still listening to whatever they are doing.
2. The UX – if the app is to continue listening then it probably needs some kind of visual cue to the user that it is still listening and also perhaps some way of cancelling and retrying that listening.
3. The platform – i.e. I don’t believe that the pieces for interaction with Cortana currently handle this type of follow-on scenario. Cortana can be used to launch the app with voice-provided parameters but she doesn’t then provide a way of continuing a conversation once that initial interaction has taken place. Any follow-on dialog is going to have to work a different way.

In order to experiment with this, I put together a little app using my standard practice of searching for photos on the flickR online service. I thought the easiest way to explain the app would be to show a little screen capture of it running so here it is below;

While it’s quite “rough and ready”, it has just enough functionality to represent this idea of functionality that might be initiated by speech and then followed up with speech, i.e.

• “FLICKR SHOW ME PICTURES OF {TOPIC}”

to launch the application but then I might want to follow that on with;

• “SHOW ME {SUBTOPIC}”

as a simple example of constraining the results that have been brought back but (as far as I know) Cortana doesn’t give me that functionality so I need to take different steps.

One Thing to Flag…


...

Continuing the Conversation


When the app lands on the page that displays a flip view control of search results, I’m going to need to do some manual speech recognition in order to accept filtering commands from the user....

Side-Bar: Trying to Mix Grammar + Dictation


At this point I’ll admit that I dropped into a 4-5 hour “black hole” around trying to get what I wanted to work with the SpeechRecognizer. ....

Getting Speech Recognized


The SpeechRecognizer is pre-configured to do natural language dictation recognition so, strictly speaking, I don’t need to constrain it in order to pick up a phrase such as “SHOW ME {SUBTOPIC}”. Instead, I can just tell it to recognize and then have a look at what has been heard after the event and see if I can unpick the string myself. It’s less elegant but less complex.....

Trying it Out


With that in place, I can try out this idea of carrying on the conversation that’s initiated by Cortana and bringing it into my app...


Link Removed.
​

Table of Contents

• Scope
• Introduction
• Integrating Cortana
  • The VCD file
  • Create the method for install
  • The OnActivated method
• Using Cortana
• See Also

Scope


The purpose of this article is to show how to integrate Cortana in the Menu App.

Introduction


One of the interesting features in Windows Phone 8.1 is Cortana. Cortana is an intelligent personal assistant, that will help users in common basic tasks, like calling a friend, scheduling an appointment ( and others ).

Cortana is not available in all languages, for this reason some non-English users have changed their devices to support it. For see more about how to have Cortana in our Windows Phone 8.1 devices see the following article.

Integrating Cortana


Cortana will use Voice Commands for interacting with the apps, these voice commands are pre-installed in the apps so Cortana knows how to launch each app.

The first step to integrate Cortana, in the Menu App, is to define the Voice Command Definition (VCD) file, that represents an XML file with the commands that Cortana will recognize and will match with the app.

For the Menu App we will define "Menu" as the name for talking with the Menu App and define the following two commands:

• Show Command: that will allow you to choose which menu we want to see: Beverages, Starters, Mains, Desserts and Special Offers.
• Natural Language Command : that will allow to recognize an expression like "I am hungry", "I want to eat" and "I want to drink".

In this sample, the VCD is defined for English (en-US) but more languages could be added.
The VCD file


Here is the VCD file:

...

Using Cortana


Before we talk with Cortana, we need to run the Menu App in the device, because we need to install the VCD. After it, we can start Cortana.

...

Code

See the code for this sample in:

Link Removed

See Also

• Integrating Store Apps with Cortana in Windows Phone 8.1
• Link Removed
• Cortana Integration and Speech Recognition: New Code Samples

This article describes how to register and use voice commands to start your Universal Windows Phone app, and how to continue the conversation within your app. The attached sample app comes with the following features:

• It registers a set of voice commands with Cortana,
• it recognizes a simple ‘open the door’ command,
• it discovers whether it was started by voice or by text,
• it recognizes a natural ‘take over‘ command, whit lots of optional terms,
• it recognizes a complex ‘close a colored something’, where ‘something’ and ‘color’ come from a predefined list,
• it modifies one of these lists programmatically,
• it requests the missing color when you ask it to ‘close something’, and
• it comes with an improved version of the Cortana-like SpeechInputBox control.

Here are some screenshots of this app. It introduces my new personal assistant, named Kwebbel (or in English ‘Kwebble’):

Link Removed

Kwebble is written as a Universal app, I dropped the Windows app project because Cortana is not yet available on that platform. I you prefer to stick to Silverlight, check the Link Removed.

Registering the Voice Command Definitions ...

A simple command


Let’s take a look at the individual commands. Each command comes with ...

How the app reacts


A Universal app can determine if it is activated by Cortana in the OnActivated event of its root App class. If ...

Optional words in command phrases


The ListenFor elements in the command phrases may contain optional words. ....

Detecting the command mode


If the app was started through a spoken Cortana command, it may start talking. If ...

Natural command phrases


Do not assume that the user will try to launch your app by just saying the call sign and a command name: “computer, start simulation” is so eighties. Modern speech ....

Continuing the conversation inside the app


Cortana’s responsibilities stop when it started up your app via a spoken or typed command. If ....

Natural phrases in the conversation


I already mentioned that the Voice Command Definitions for Cortana activation are quite capable of dealing with natural language....

That’s all folks!

Here’s the full code of the sample app: the XAML, the C#, and last but not least the different powerful XML files. The solution was created with Visual Studio 2013 Update 3....
​