[BUG] Elastic.Apm.Model.Transaction+DroppedSpanStatsKey Duplicate Key

[bliles]

APM Agent version

1.25.2

Environment

Operating system and version:
Docker mcr.microsoft.com/dotnet/aspnet:8.0
Host: Amazon Linux 2 ARM64

.NET Framework/Core name and version (e.g. .NET 4.6.2, NET Core 3.1.100) :
.NET 8.0

Application Target Framework(s) (e.g. net462, netcoreapp3.1):
net8.0

Describe the bug

We see the following exception in our logs 1-2 times a day since the upgrade to .NET 8:

Elasticsearch.Net.UnexpectedElasticsearchClientException: An item with the same key has already been added. Key: Elastic.Apm.Model.Transaction+DroppedSpanStatsKey
 ---> System.ArgumentException: An item with the same key has already been added. Key: Elastic.Apm.Model.Transaction+DroppedSpanStatsKey
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Elastic.Apm.Model.Transaction.UpdateDroppedSpanStats(String serviceTargetType, String serviceTargetName, String destinationServiceResource, Outcome outcome, Double duration)
   at Elastic.Apm.Model.Span.End()
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.UpdateSpan(Activity activity, Span span)
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.<get_ActivityStopped>b__21_0(Activity activity)
   at System.Diagnostics.SynchronizedList`1.EnumWithAction(Action`2 action, Object arg)
   at System.Diagnostics.Activity.Stop()
   at System.Diagnostics.DiagnosticSource.StopActivity(Activity activity, Object args)
   at Elasticsearch.Net.Diagnostics.Diagnostic`2.Dispose()
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)
   --- End of inner exception stack trace ---
[Redacted to remove references to our code, our code is making requests to an Elasticsearch 8.9 cluster using the Elasticsearch.Net library]
--- End of stack trace from previous location ---
[Redacted to remove references to our code]
System.ArgumentException: An item with the same key has already been added. Key: Elastic.Apm.Model.Transaction+DroppedSpanStatsKey
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Elastic.Apm.Model.Transaction.UpdateDroppedSpanStats(String serviceTargetType, String serviceTargetName, String destinationServiceResource, Outcome outcome, Double duration)
   at Elastic.Apm.Model.Span.End()
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.UpdateSpan(Activity activity, Span span)
   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.<get_ActivityStopped>b__21_0(Activity activity)
   at System.Diagnostics.SynchronizedList`1.EnumWithAction(Action`2 action, Object arg)
   at System.Diagnostics.Activity.Stop()
   at System.Diagnostics.DiagnosticSource.StopActivity(Activity activity, Object args)
   at Elasticsearch.Net.Diagnostics.Diagnostic`2.Dispose()
   at Elasticsearch.Net.HttpConnection.RequestAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearchAsync[TResponse](RequestData requestData, CancellationToken cancellationToken)
   at Elasticsearch.Net.Transport`1.RequestAsync[TResponse](HttpMethod method, String path, CancellationToken cancellationToken, PostData data, IRequestParameters requestParameters)

To Reproduce

I am not sure how to reproduce, but I assume that this is happening due to concurrency/race condition.

Expected behavior

APM should not have this error if the key already exists.

Actual behavior

APM encounters this error.

[tomap]

Hello,

I have the exact same issue. It works fine on v1.23 but breaks on newer versions, including the version released this morning: 1.25.3

I am using is in conjunction with Couchbase .Net client: https://github.com/couchbase/couchbase-net-client (And I assumed it was the cause)

System.ArgumentException: An item with the same key has already been added. Key: Elastic.Apm.Model.Transaction\u002BDroppedSpanStatsKey
   at System.Collections.Generic.Dictionary\u00602.TryInsert(TKey key, TValue value, InsertionBehavior behavior)

   at Elastic.Apm.Model.Transaction.UpdateDroppedSpanStats(String serviceTargetType, String serviceTargetName, String destinationServiceResource, Outcome outcome, Double duration)

   at Elastic.Apm.Model.Span.End()

   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.UpdateSpan(Activity activity, Span span)

   at Elastic.Apm.OpenTelemetry.ElasticActivityListener.\u003Cget_ActivityStopped\u003Eb__21_0(Activity activity)

   at System.Diagnostics.SynchronizedList\u00601.EnumWithAction(Action\u00602 action, Object arg)

   at System.Diagnostics.Activity.Stop()

   at Couchbase.Core.Diagnostics.Tracing.RequestSpan.End()

   at Couchbase.KeyValue.CouchbaseCollection.UpsertAsync[T](String id, T content, UpsertOptions options)

[bliles]

apm-agent-dotnet/src/Elastic.Apm/Model/Transaction.cs

Line 555 in 4459636

internal void UpdateDroppedSpanStats(string serviceTargetType, string serviceTargetName, string destinationServiceResource, Outcome outcome,

Looks like a Dictionary is being used when a ConcurrentDictionary is likely needed.

[Mpdreamz]

Thank you for bringing this to our attention everyone.

I've opened #2283 to address this.