diff --git a/CallAutomation_AzOpenAI_Voice/README.md b/CallAutomation_AzOpenAI_Voice/README.md new file mode 100644 index 00000000..56dce583 --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/README.md @@ -0,0 +1,54 @@ +|page_type| languages |products +|---|---------------------------------------|---| +|sample|
Java
|
azureazure-communication-services
| + +# Call Automation - Quick Start Sample + +This sample application shows how the Azure Communication Services - Call Automation SDK can be used with Azure OpenAI Service to enable intelligent conversational agents. +It answers an inbound call, does a speech recognition with the recognize API and Cognitive Services, uses OpenAi Services with the input speech and responds to the caller through Cognitive Services' Text to Speech. +This sample application configured for accepting input speech until the caller terminates the call or a long silence is detected. +This sample application is also capable of making multiple concurrent inbound calls. The application is a web-based application built on Java's Spring framework. + + +## Prerequisites + +- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F). +- A deployed Communication Services resource. [Create a Communication Services resource](https://docs.microsoft.com/azure/communication-services/quickstarts/create-communication-resource). +- A [phone number](https://learn.microsoft.com/en-us/azure/communication-services/quickstarts/telephony/get-phone-number) in your Azure Communication Services resource that can make outbound calls. NB: phone numbers are not available in free subscriptions. +- [Java Development Kit (JDK) Microsoft.OpenJDK.17](https://learn.microsoft.com/en-us/java/openjdk/download) +- [Apache Maven](https://maven.apache.org/download.cgi) +- Create and host a Azure Dev Tunnel. Instructions [here](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started) +- Create an Azure Cognitive Services resource. For details, see Create an Azure Cognitive Services Resource. +- An Azure OpenAI Resource and Deployed Model. See https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/create-resource?pivots=web-portal. + +## Before running the sample for the first time + +- Open the application.yml file in the resources folder to configure the following settings + + - `connectionstring`: Azure Communication Service resource's connection string. + - `basecallbackuri`: Base url of the app. For local development use dev tunnel url. + - `cognitiveServicesUrl`: The Cognitive Services endpoint + - `azureOpenAiServiceKey`: Open AI's Service Key + - `azureOpenAiServiceEndpoint`: Open AI's Service Endpoint + - `openAiModelName`: Open AI's Model name + - `agentPhoneNumber`: Agent Phone Number + + +### Setup and host your Azure DevTunnel + +[Azure DevTunnels](https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/overview) is an Azure service that enables you to share local web services hosted on the internet. Use the commands below to connect your local development environment to the public internet. This creates a tunnel with a persistent endpoint URL and which allows anonymous access. We will then use this endpoint to notify your application of calling events from the ACS Call Automation service. + +```bash +devtunnel create --allow-anonymous +devtunnel port create -p 8080 +devtunnel host +``` + +### Run the application + +- Navigate to the directory containing the pom.xml file and use the following mvn commands: + - Compile the application: mvn compile + - Build the package: mvn package + - Execute the app: mvn exec:java +- Access the Swagger UI at http://localhost:8080/swagger-ui.html + - Try the GET /outboundCall to run the Sample Application \ No newline at end of file diff --git a/CallAutomation_AzOpenAI_Voice/pom.xml b/CallAutomation_AzOpenAI_Voice/pom.xml new file mode 100644 index 00000000..61335353 --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/pom.xml @@ -0,0 +1,172 @@ + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 3.0.6 + + + + com.communication.callautomation + CallAutomation_AzOpenAI_Voice + 1.0-SNAPSHOT + + CallAutomation_AzOpenAI_Voice + CallAutomation Sample application for OpenAI usage + + + 17 + 17 + UTF-8 + 1.18.26 + + + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-test + test + + + com.vaadin.external.google + android-json + + + + + junit + junit + 4.13.2 + test + + + com.azure + azure-core + 1.42.0 + + + com.azure + azure-identity + 1.10.0 + + + com.azure + azure-communication-identity + 1.4.8 + + + com.azure + azure-communication-common + + + + + com.azure + azure-communication-callautomation + 1.3.0-beta.1 + + + com.azure + azure-messaging-eventgrid + 4.17.1 + + + com.azure + azure-communication-common + 2.0.0-beta.1 + + + org.projectlombok + lombok + provided + ${lombok.version} + + + org.springframework.boot + spring-boot-configuration-processor + true + + + org.springframework.boot + spring-boot-autoconfigure-processor + true + + + org.springdoc + springdoc-openapi-starter-webmvc-ui + 2.0.0 + + + org.json + json + 20230618 + + + com.azure + azure-ai-openai + 1.0.0-beta.3 + + + + + + + + maven-clean-plugin + 3.2.0 + + + maven-resources-plugin + 3.3.1 + + + maven-compiler-plugin + 3.11.0 + + + maven-surefire-plugin + 3.1.0 + + + maven-jar-plugin + 3.3.0 + + + maven-deploy-plugin + 3.1.1 + + + maven-site-plugin + 3.12.1 + + + maven-project-info-reports-plugin + 3.4.3 + + + org.codehaus.mojo + exec-maven-plugin + 3.1.0 + + + + java + + + + + com.communication.callautomation.Main + + + + + + \ No newline at end of file diff --git a/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/AppConfig.java b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/AppConfig.java new file mode 100644 index 00000000..330ee2f3 --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/AppConfig.java @@ -0,0 +1,54 @@ +package com.communication.callautomation; + +import lombok.Getter; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.boot.context.properties.bind.ConstructorBinding; + +@ConfigurationProperties(prefix = "acs") +@Getter +public class AppConfig { + private final String connectionString; + private final String basecallbackuri; + private final String cognitiveServicesUrl; + private final String azureOpenAiServiceKey; + private final String azureOpenAiServiceEndpoint; + private final String openAiModelName; + private final String agentPhoneNumber; + + @ConstructorBinding + AppConfig(final String connectionString, + final String basecallbackuri, + final String cognitiveServicesUrl, + final String azureOpenAiServiceKey, + final String azureOpenAiServiceEndpoint, + final String openAiModelName, + final String agentPhoneNumber) { + this.connectionString = connectionString; + this.basecallbackuri = basecallbackuri; + this.cognitiveServicesUrl = cognitiveServicesUrl; + this.azureOpenAiServiceKey = azureOpenAiServiceKey; + this.azureOpenAiServiceEndpoint = azureOpenAiServiceEndpoint; + this.openAiModelName = openAiModelName; + this.agentPhoneNumber = agentPhoneNumber; + } + + public String getCallBackUri() { + return basecallbackuri + "/api/callback"; + } + + public String getCognitiveServicesUrl() { + return cognitiveServicesUrl; + } + + public String getConnectionString() { + return connectionString; + } + + public String getAzureOpenAiServiceKey() { + return azureOpenAiServiceKey; + } + + public String getAzureOpenAiServiceEndpoint() { + return azureOpenAiServiceEndpoint; + } +} diff --git a/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/Main.java b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/Main.java new file mode 100644 index 00000000..d1ad1c14 --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/Main.java @@ -0,0 +1,13 @@ +package com.communication.callautomation; + +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.context.properties.EnableConfigurationProperties; + +@SpringBootApplication +@EnableConfigurationProperties(value = AppConfig.class) +public class Main { + public static void main(String[] args) { + SpringApplication.run(Main.class, args); + } +} \ No newline at end of file diff --git a/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/ProgramSample.java b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/ProgramSample.java new file mode 100644 index 00000000..3160fb8e --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/src/main/java/com/communication/callautomation/ProgramSample.java @@ -0,0 +1,172 @@ +package com.communication.callautomation; +import com.azure.ai.openai.OpenAIAsyncClient; +import com.azure.ai.openai.OpenAIClientBuilder; +import com.azure.communication.callautomation.CallAutomationAsyncClient; +import com.azure.communication.callautomation.CallAutomationClientBuilder; +import com.azure.communication.callautomation.CallAutomationEventParser; +import com.azure.communication.callautomation.models.*; +import com.azure.communication.callautomation.models.events.*; +import com.azure.core.credential.AzureKeyCredential; +import com.azure.core.http.rest.Response; +import com.azure.core.util.BinaryData; +import com.azure.messaging.eventgrid.EventGridEvent; +import com.azure.messaging.eventgrid.SystemEventNames; +import com.azure.messaging.eventgrid.systemevents.SubscriptionValidationEventData; +import com.azure.messaging.eventgrid.systemevents.SubscriptionValidationResponse; +import lombok.extern.slf4j.Slf4j; +import reactor.core.publisher.Mono; +import org.json.JSONObject; +import org.springframework.http.ResponseEntity; +import org.springframework.web.bind.annotation.*; + +import java.net.URI; +import java.util.*; + +@RestController +@Slf4j +public class ProgramSample { + private final AppConfig appConfig; + private final CallAutomationAsyncClient asyncClient; + private final OpenAIAsyncClient aiClient; + private static final String INCOMING_CALL_CONTEXT = "incomingCallContext"; + + public ProgramSample(final AppConfig appConfig) { + this.appConfig = appConfig; + asyncClient = initAsyncClient(); + aiClient = initOpenAIClient(); + } + + @GetMapping(path = "/") + public ResponseEntity hello() { + return ResponseEntity.ok().body("Hello! ACS CallAutomation OpenAI Sample!"); + } + + @PostMapping(path = "/api/incomingCall") + public ResponseEntity recordinApiEventGridEvents( + @RequestBody final String reqBody) { + List events = EventGridEvent.fromString(reqBody); + for (EventGridEvent eventGridEvent : events) { + if (eventGridEvent.getEventType().equals(SystemEventNames.EVENT_GRID_SUBSCRIPTION_VALIDATION)) { + return handleSubscriptionValidation(eventGridEvent.getData()); + } else if (eventGridEvent.getEventType().equals(SystemEventNames.COMMUNICATION_INCOMING_CALL)) { + handleIncomingCall(eventGridEvent.getData()); + } else { + log.debug("Unhandled event."); + } + } + return ResponseEntity.ok().body(null); + } + + @PostMapping(path = "/api/callback/{contextId}") + public ResponseEntity callbackEvents(@RequestBody final String reqBody, + @PathVariable final String contextId, + @RequestParam final String callerId) { + List events = CallAutomationEventParser.parseEvents(reqBody); + for (CallAutomationEventBase event : events) { + String callConnectionId = event.getCallConnectionId(); + if (event instanceof CallConnected) { + log.info("Call connected performing recognize for Call Connection ID: {}", callConnectionId); + + } else if (event instanceof CallDisconnected) { + log.info("Received Call Disconnected event for Call Connection ID: {}", callConnectionId); + } + } + return ResponseEntity.ok().body(""); + } + + private void handleIncomingCall(final BinaryData eventData) { + JSONObject data = new JSONObject(eventData.toString()); + String callbackUri; + AnswerCallOptions options; + String cognitiveServicesUrl; + + try { + MediaStreamingOptions mediaStreamingOptions = new MediaStreamingOptions( + "", + MediaStreamingTransport.WEBSOCKET, + MediaStreamingContent.AUDIO, + MediaStreamingAudioChannel.MIXED, + false).setEnableBidirectional(true).setAudioFormat(AudioFormat.Pcm24KMono); + callbackUri = String.format("%s/%s?callerId=%s", + appConfig.getCallBackUri(), + UUID.randomUUID(), + data.getJSONObject("from").getString("rawId")); + cognitiveServicesUrl = new URI(appConfig.getCognitiveServicesUrl()).toString(); + CallIntelligenceOptions callIntelligenceOptions = new CallIntelligenceOptions() + .setCognitiveServicesEndpoint(appConfig.getCognitiveServicesUrl()); + options = new AnswerCallOptions(data.getString(INCOMING_CALL_CONTEXT), + callbackUri).setCallIntelligenceOptions(callIntelligenceOptions) + .setMediaStreamingOptions(mediaStreamingOptions); + Mono> answerCallResponse = asyncClient.answerCallWithResponse(options); + answerCallResponse.subscribe(response -> { + log.info("Incoming call answered. Cognitive Services Url: {}\nCallbackUri: {}\nCallConnectionId: {}", + cognitiveServicesUrl, + callbackUri, + response.getValue().getCallConnectionProperties().getCallConnectionId()); + }); + } catch (Exception e) { + log.error("Error getting recording location info {} {}", + e.getMessage(), + e.getCause()); + } + } + + private ResponseEntity handleSubscriptionValidation(final BinaryData eventData) { + try { + log.info("Received Subscription Validation Event from Incoming Call API endpoint"); + SubscriptionValidationEventData subscriptioneventData = eventData + .toObject(SubscriptionValidationEventData.class); + SubscriptionValidationResponse responseData = new SubscriptionValidationResponse(); + responseData.setValidationResponse(subscriptioneventData.getValidationCode()); + return ResponseEntity.ok().body(responseData); + } catch (Exception e) { + log.error("Error at subscription validation event {} {}", + e.getMessage(), + e.getCause()); + return ResponseEntity.internalServerError().body(null); + } + } + + private OpenAIAsyncClient initOpenAIClient() { + OpenAIAsyncClient aiClient; + String key; + String endpoint; + try { + key = appConfig.getAzureOpenAiServiceKey(); + endpoint = appConfig.getAzureOpenAiServiceEndpoint(); + + aiClient = new OpenAIClientBuilder() + .credential(new AzureKeyCredential(key)) + .endpoint(endpoint) + .buildAsyncClient(); + return aiClient; + + } catch (NullPointerException e) { + log.error("Please verify if Application config is properly set up"); + return null; + } catch (Exception e) { + log.error("Error occurred when initializing open ai Async Client: {} {}", + e.getMessage(), + e.getCause()); + return null; + } + } + + private CallAutomationAsyncClient initAsyncClient() { + CallAutomationAsyncClient client; + try { + client = new CallAutomationClientBuilder() + .connectionString(appConfig.getConnectionString()) + .buildAsyncClient(); + return client; + } catch (NullPointerException e) { + log.error("Please verify if Application config is properly set up"); + return null; + } catch (Exception e) { + log.error("Error occurred when initializing Call Automation Async Client: {} {}", + e.getMessage(), + e.getCause()); + return null; + } + } +} diff --git a/CallAutomation_AzOpenAI_Voice/src/main/resources/application.yml b/CallAutomation_AzOpenAI_Voice/src/main/resources/application.yml new file mode 100644 index 00000000..608380dc --- /dev/null +++ b/CallAutomation_AzOpenAI_Voice/src/main/resources/application.yml @@ -0,0 +1,15 @@ +spring: + application: + name: CallAutomation_OutboundCalling + +server: + port: 8080 + +acs: + connectionstring: + basecallbackuri: + cognitiveServicesUrl: + azureOpenAiServiceKey: + azureOpenAiServiceEndpoint: + openAiModelName: + agentPhoneNumber: diff --git a/CallAutomation_AzOpenAI_Voice/static/Confirmed.wav b/CallAutomation_AzOpenAI_Voice/static/Confirmed.wav new file mode 100644 index 00000000..bbe4a0bc Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/Confirmed.wav differ diff --git a/CallAutomation_AzOpenAI_Voice/static/Goodbye.wav b/CallAutomation_AzOpenAI_Voice/static/Goodbye.wav new file mode 100644 index 00000000..09e632dc Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/Goodbye.wav differ diff --git a/CallAutomation_AzOpenAI_Voice/static/Invalid.wav b/CallAutomation_AzOpenAI_Voice/static/Invalid.wav new file mode 100644 index 00000000..7fe29a2e Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/Invalid.wav differ diff --git a/CallAutomation_AzOpenAI_Voice/static/MainMenu.wav b/CallAutomation_AzOpenAI_Voice/static/MainMenu.wav new file mode 100644 index 00000000..dfb6de01 Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/MainMenu.wav differ diff --git a/CallAutomation_AzOpenAI_Voice/static/OutboundCallDesign.png b/CallAutomation_AzOpenAI_Voice/static/OutboundCallDesign.png new file mode 100644 index 00000000..c750ab72 Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/OutboundCallDesign.png differ diff --git a/CallAutomation_AzOpenAI_Voice/static/Timeout.wav b/CallAutomation_AzOpenAI_Voice/static/Timeout.wav new file mode 100644 index 00000000..99a3385b Binary files /dev/null and b/CallAutomation_AzOpenAI_Voice/static/Timeout.wav differ