Application Insights Java Agent: ConnectionStringOverrides Feature Fails to Route Telemetry as Expected

[lobregliolu]

Expected behavior

I expected requests to be tracked in the respective Application Insights resources based on the httpPathPrefix defined in the configuration. Specifically:

• Requests to /OsaEcommerceServerBE/ should be tracked in THE_ONE_BE.
• Requests to /OsaEcommerceServerNL/ should be tracked in THE_ONE_NL.

---

Actual behavior

The behavior observed differs depending on the configuration JSON:

Case 1

Configuration:

{
  "role": {
    "name": "testrolename",
    "instance": "testroleinstance"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerBE",
        "connectionString": "THE_ONE_BE"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerNL",
        "connectionString": "THE_ONE_NL"
      }
    ]
  },
  "connectionString": "THE_ONE_BE"
}

Observed Behavior:

• /OsaEcommerceServerBE/ is tracked in THE_ONE_BE.
• /OsaEcommerceServerNL/ is not tracked anywhere.

---

Case 2

Configuration:

{
  "role": {
    "name": "testrolename",
    "instance": "testroleinstance"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerNL",
        "connectionString": "THE_ONE_BE"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerNL",
        "connectionString": "THE_ONE_NL"
      }
    ]
  },
  "connectionString": "THE_ONE_BE"
}

Observed Behavior:

• Both /OsaEcommerceServerBE/ and /OsaEcommerceServerNL/ are tracked in THE_ONE_BE.

---

Case 3

Configuration:

{
  "role": {
    "name": "testrolename",
    "instance": "testroleinstance"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerBE",
        "connectionString": "THE_ONE_BE"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerBE",
        "connectionString": "THE_ONE_NL"
      }
    ]
  },
  "connectionString": "THE_ONE_BE"
}

Observed Behavior:

• /OsaEcommerceServerBE/ is tracked in THE_ONE_BE.
• /OsaEcommerceServerNL/ is also tracked in THE_ONE_BE.

---

Case 4

Configuration:

{
  "role": {
    "name": "testrolename",
    "instance": "testroleinstance"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerNL",
        "connectionString": "THE_ONE_BE"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerBE",
        "connectionString": "THE_ONE_NL"
      }
    ]
  },
  "connectionString": "THE_ONE_BE"
}

Observed Behavior:

• /OsaEcommerceServerBE/ is not tracked anywhere.
• /OsaEcommerceServerNL/ is tracked in THE_ONE_BE.

---

Case 5

Configuration:

{
  "role": {
    "name": "testrolename",
    "instance": "testroleinstance"
  },
  "sampling": {
    "percentage": 100
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerNL",
        "connectionString": "THE_ONE_BE"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerBE",
        "connectionString": "THE_ONE_NL"
      }
    ]
  }
}

Observed Behavior:

• No requests are tracked at all.

---

Logs

When the connectionString field is not specified in the configuration, the Application Insights Java agent failed to start, citing that no connection string was provided. This indicates that the connectionStringOverrides feature does not work as expected without a main connectionString specified.

---

To Reproduce

1. Set up a Tomcat server with two applications running on distinct context paths (/OsaEcommerceServerBE and /OsaEcommerceServerNL).
2. Configure the Application Insights agent using the provided JSON configurations for each case.
3. Make requests to http://localhost:8081/OsaEcommerceServerBE/ and http://localhost:8081/OsaEcommerceServerNL/.
4. Observe the Application Insights telemetry data.

---

System information

• SDK Version: 21
• OS type and version: Windows 10
• Application Server type and version: Apache Tomcat 10.1.33
• Using spring-boot? Yes, version 3.1.5
• Additional relevant libraries: None applicable

---

[trask]

hi @lobregliolu, is there any info in the applicationinsights.log file?

[lobregliolu]

hi @lobregliolu, is there any info in the applicationinsights.log file?

Hi @trask thanks for replying!

When the "main" connectionString is provided in the json (that's not what I want but I tried just to try figruing out), I just receive
2025-01-23 17:11:48.468+01:00 INFO c.m.applicationinsights.agent - Application Insights Java Agent 3.6.2 started successfully (PID 5520, JVM running for 5.319 s) 2025-01-23 17:11:48.469+01:00 INFO c.m.applicationinsights.agent - Java version: 21.0.5, vendor: Oracle Corporation, home: C:\Program Files\Java\jdk-21

When I start the agent with the "Case 5" json, so with the connectionStringOverrides object without the main connectionString
I see in the logs that the connectionString has not been provided

`Application Insights Java Agent 3.6.2 startup failed
*************************

Description:
No connection string provided

Action:
Please provide connection string.`

[trask]

When I start the agent with the "Case 5" json, so with the connectionStringOverrides object without the main connectionString
I see in the logs that the connectionString has not been provided

this is expected

let's focus on Case 1 which looks correct to me (and we have smoke tests validating a similar configuration)

can you enable self-diagnostic logging at debug level and post an examples of the "exporting span" diagnostic for both /OsaEcommerceServerBE and /OsaEcommerceServerNL (and the associated json just past that which includes "Request")?

[lobregliolu]

In the server.xml I have these two service elements

<!-- Service for nl-qa -->
  <Service name="nl-qa-Service">
    <Connector port="8082" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
			   maxParameterCount="1000"
               enableLookups="false"
               maxKeepAliveRequests="1" />
    <Engine name="Catalina-nl-qa" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase" />
      </Realm>
      <Host name="localhost" appBase="webapps_nl-qa"
            unpackWARs="true" autoDeploy="true">
      </Host>
    </Engine>
  </Service>
  
   <!-- Service for be-qa -->
  <Service name="be-qa-Service">
    <Connector port="8081" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
			   maxParameterCount="1000"
               enableLookups="false"
               maxKeepAliveRequests="1" />
    <Engine name="Catalina-be-qa" defaultHost="localhost">
      <Realm className="org.apache.catalina.realm.LockOutRealm">
        <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
               resourceName="UserDatabase" />
      </Realm>
      <Host name="localhost" appBase="webapps_be-qa"
            unpackWARs="true" autoDeploy="true">
      </Host>
    </Engine>
  </Service>

In the setenv.bat I have

rem Set the Java Home path for your Java installation
set JAVA_HOME=C:\Program Files\Java\jdk-21

rem Set the Tomcat Home directory path
set CATALINA_HOME=C:\Program Files\apache-tomcat-10.1.33

rem Ensure the classpath is correctly set
set CLASSPATH=%CATALINA_HOME%\bin;%CATALINA_HOME%\lib\*

rem Add your custom properties files
set CLASSPATH=%CLASSPATH%;C:\Program Files\apache-tomcat-10.1.33\conf\osa

set JAVA_OPTS=%JAVA_OPTS% -Dspring.config.location="file:C:/Program Files/apache-tomcat-10.1.33/conf/osa/" -DbuildNumber=1

rem Load Application Insights agent for NL application
set CATALINA_OPTS=%CATALINA_OPTS% -javaagent:"C:\Program Files\apache-tomcat-10.1.33\lib\applicationinsights-agent-3.7.0.jar"

set CATALINA_OPTS=%CATALINA_OPTS% -Dapplicationinsights.debug=true`

So I have two application running and reachable at (due to war names):
http://localhost:8082/OsaEcommerceServerNL/
and
http://localhost:8081/OsaEcommerceServerBE/

This is my applicationinsights.json file the agent read because is in the tomcat lib folder

`{
  "role": {
    "name": "OsaEcommerceServer"
  },
  "preview": {
    "connectionStringOverrides": [
      {
        "httpPathPrefix": "/OsaEcommerceServerNL/BE",
        "connectionString": "OsaEcommerceServerBE_ConnectionString"
      },
      {
        "httpPathPrefix": "/OsaEcommerceServerNL/NL",
        "connectionString": OsaEcommerceServerNL_ConnectionString"
      }
    ]
  },
  "connectionString": "OsaEcommerceServerBE_ConnectionString",
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "INFO",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  },
  "sampling": {
  "percentage": 100
	}
}

In the applicationinsights.log I just see

2025-01-30 13:01:44.883+01:00 WARN  c.m.applicationinsights.agent - The applicationinsights-agent JAR is in the class path. You should remove it because it could lead to unexpected results. You should configure the Java agent with -javaagent. You can also use the runtime attachment with Spring Boot applications.
2025-01-30 13:01:48.949+01:00 INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.7.0 started successfully (PID 31240, JVM running for 5.553 s)
2025-01-30 13:01:48.961+01:00 INFO  c.m.applicationinsights.agent - Java version: 21.0.5, vendor: Oracle Corporation, home: C:\Program Files\Java\jdk-21
2025-01-30 13:02:46.231+01:00 ERROR c.a.c.h.netty.NettyAsyncHttpClient - io.netty.channel.AbstractChannel$AnnotatedSocketException: Network is unreachable: getsockopt: /169.254.169.254:80

This line
2025-01-30 13:01:44.883+01:00 WARN c.m.applicationinsights.agent - The applicationinsights-agent JAR is in the class path. You should remove it because it could lead to unexpected results. You should configure the Java agent with -javaagent. You can also use the runtime attachment with Spring Boot applications.
I don't know why, because the agent is not present anywhere except the lib folder of tomcat (also cleaning the temp folder), it's not between the dependencies of the project, so it is not in the wars.

[trask]

I think all jar files in the tomcat lib are automatically added to the classpath, can you try moving the agent jar to another location?

[lobregliolu]

Yes, thank you. Moving it in another folder changing the -javaagent option solved that line.
Meanwhile, I have updates on my issue. It seems the problem is in the Live Metrics section.
I tried requests intentionally wrong (like 404), and I can see these correctly tracked in the Failures section of the related AppInsights resources. Looking at the Live Metrics section, I see, only in the OsaEcommerceServerBE AppInsights resource, logs related to the tomcat instance starting both applications, and then I can see only exceptions coming from the OsaEcommerceServerBE app. So the LiveMetrics section of the OsaEcommerceServerNL AppInsights resource doesn't show anything, even if exception are there in the Failures section

[trask]

ah, this is a good point, I don't believe that live metrics is respecting the connection overrides, we will need to investigate