Core Assistant: Cannot disambiguate similar intents in the same LUIS recognizer

[darrenj]

Describe the bug

Basic Assistant template has a duplicated intents recognized capability which I'm unable to trigger. Some of this is related to microsoft/BotFramework-Composer#7404 but after inspecting trace it would appear that IncludeAllIntents isn't being set on the call to LUIS meaning your only going to get the topIntent back not the other intents and scores meaning this feature cannot work when LUIS is being used.

{
  "activity": {
    "type": "trace",
    "timestamp": "2021-04-27T14:12:01.477Z",
    "serviceUrl": "http://localhost:5000",
    "channelId": "emulator",
    "from": {
      "id": "ec66a842-b822-4078-92c4-d2bf4890942d",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "04b1056c-dc37-44ad-8764-ea2ff77add8d|livechat"
    },
    "recipient": {
      "id": "91b9c1ca-4273-4517-87a4-6cf4ffe6a8ab",
      "role": "user"
    },
    "locale": "en-us",
    "replyToId": "1d2ffde0-5652-4b7a-a5d9-3c7f8702e5d0",
    "label": "LuisV3 Trace",
    "valueType": "https://www.luis.ai/schemas/trace",
    "value": {
      "recognizerResult": {
        "text": "book",
        "alteredText": null,
        "intents": {
          "None": {
            "score": 0.79196495
          }
        },
        "entities": {}
      },
      "luisModel": {
        "ModelID": "44388ba7-de06-46e6-9c98-090b6219f5e9"
      },
      "luisOptions": {
        "IncludeAllIntents": false,
        "IncludeInstanceData": false,
        "IncludeAPIResults": false,
        "Log": true,
        "DynamicLists": null,
        "ExternalEntities": null,
        "PreferExternalEntities": true,
        "DateTimeReference": null,
        "Slot": "production",
        "Version": null
      },
      "luisResult": {
        "query": "book",
        "prediction": {
          "topIntent": "None",
          "intents": {
            "None": {
              "score": 0.79196495
            }
          },
          "entities": {}
        }
      }
    },
    "name": "LuisRecognizer",
    "id": "bfc80184-d739-470e-9d93-dbd10d6cabfa",
    "localTimestamp": "2021-04-27T15:12:01+01:00"
  },
  "id": "0f59b4bf-598d-4d26-9621-11fac3acbffc",
  "timestamp": 1619532721477
}

Version

2.0.0-nightly-239644

Browser

• Electron distribution
• Chrome
• Safari
• Firefox
• Edge

OS

• macOS
• Windows
• Ubuntu

To Reproduce

Steps to reproduce the behavior:

1. Create Basic Assistant
2. Create two dialogs with overlapping intents (be aware of Modelling two intents in LU with overlapping utterances in intents/utterances being dropped silently BotFramework-Composer#7404)
3. Observe LUIS trace in Composer and see that only topIntent is being returned

Expected behavior

Disambiguation to work!

[darrenj]

** UPDATE **

Did some digging as per this:

https://github.com/microsoft/BotFramework-Composer/blob/ef424c5583d9c2813f4b7255201d912ccc770696/Composer/packages/server/schemas/sdk.en-US.schema

By hand I updated my recognizer configuration in recognizers folder to include following (not clear if Composer surfaces this anywhere)

  "predictionOptions" : 
	{
		"includeAllIntents": true
	}

This results with multiple intents coming back but not able to prove disambiguation works due to microsoft/BotFramework-Composer#7404. If it does we'll need to figure out if we can have this set in the template?

[cwhitten]

@darrenj Composer does not (yet) surface the the entire interface for a specific recognizer.

[lauren-mills]

This will impact EA as well so @ryanlengel and I can tag team it

[taicchoumsft]

@darrenj . even if we set includeAllIntents to true, and even if we solved the lu de-dupe problem in Composer#7404, the disambig template will not work with LUIS.

The new template logic takes into account child recognizers that fire their own ChooseIntent intents, like Orchestrator (CrossTrainedRecognizerSet, the parent recognizer in Composer, already does this).

i.e. To see this template disambiguate on intents, the intent-based child recognizers must themselves be able to fire the "ChooseIntent" intent, and put what it thinks are duplicates into the candidate property. When the trigger is hit, the template takes all these candidates from all child recognizers of CrossTrainedRecognizerSet and flattens them into one list, that's all it does.

LUIS does not fire this intent, If you set the includeAllIntents property to true, it'll simply populate the intents property, not the candidates property.

To do intent disambiguation for LUIS (more than disambiguating between LUIS and QNA like the old template), we also need to change the template logic, and set the includeAllIntents flag for LUIS in Composer.

[ryanisgrig]

I've validated that we can include the includeAllIntents property in the Basic Assistant's recognizer and LUIS does in fact return all intents after having configured it. While the Enterprise Assistant is configured with the Orchestrator recognizer, the Basic Assistant uses the LUIS recognizer (since there are no prebuilt skills included).

Based on what @taicchoumsft described, the disambiguation dialog used by both Composer and these templates is configured only for Orchestrator and will not work in the same way for LUIS. That being said I'd like to hold on assistant template changes until the end to end scenario is fully understood and agreed upon, the changes needed to make work here will have to bubble upstream. Will wait for @darrenj's reply.