Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type when writting to NLog

[fsimonovski]

NEST/Elasticsearch.Net version: 7.13.2

Elasticsearch version: Elastic cluster is v7.10

.NET runtime version: 8.0

Operating system version: Windows

Description of the problem including expected versus actual behavior:
When using NLog together with elasticsearch, I am getting Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type errors. One thing to note is that they are not immediate, one or 2 calls will go through and after that, it starts failing.

Steps to reproduce:
1.
2.
3.

Expected behavior
Being able to log on elasticsearch, this was not an issue on net6.0 framework with the same config file.

Provide ConnectionSettings (if relevant):

<target xsi:type="ElasticSearch" index="${configsetting:item=Nlog.Log_Index}-${date:format=yyyy.MM.dd}" uri="${configsetting:item=Nlog.Log_Url}" layout ="API:SpecificName |${longdate}|${event-properties:item=EventId_Id}|${uppercase:${level}}|${logger}|${message} ${exception:format=tostring}|url: ${aspnet-request-url}|action: ${aspnet-mvc-action}"documentType="logevent"includeAllProperties="true">

This is the nlog.config file, I am reading configsetting items from appsettings.json, which is as follows:
{
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft.AspNetCore": "Warning"
}
},
"NLog": {
"Log_Url": "{someurl}",
"Log_MinLevel": "Info",
"Log_Index": "{someindex}"
},

"AllowedHosts": "*"
}

These are installed package versions:

Provide DebugInformation (if relevant):
2024-04-16 17:32:46.3235 Error ElasticSearch: Error while sending log messages Exception: Elasticsearch.Net.UnexpectedElasticsearchClientException: Failed to serialize anonymous type: Microsoft.AspNetCore.Routing.RouteEndpoint.
The type defines the following properties:
'Order' of type System.Int32
'RoutePattern' of type Microsoft.AspNetCore.Routing.Patterns.RoutePattern
'DisplayName' of type System.String
'Metadata' of type Microsoft.AspNetCore.Http.EndpointMetadataCollection
'RequestDelegate' of type Microsoft.AspNetCore.Http.RequestDelegate
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: Microsoft.AspNetCore.Routing.RouteEndpoint.
The type defines the following properties:
'Order' of type System.Int32
'RoutePattern' of type Microsoft.AspNetCore.Routing.Patterns.RoutePattern
'DisplayName' of type System.String
'Metadata' of type Microsoft.AspNetCore.Http.EndpointMetadataCollection
'RequestDelegate' of type Microsoft.AspNetCore.Http.RequestDelegate
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: Microsoft.AspNetCore.Http.EndpointMetadataCollection.
The type defines the following properties:
'Item' of type System.Object
'Count' of type System.Int32
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: System.Runtime.CompilerServices.NullableContextAttribute.
The type defines the following properties:
'TypeId' of type System.Object
---> Elasticsearch.Net.Utf8Json.Resolvers.AnonymousTypeSerializationException: Failed to serialize anonymous type: System.RuntimeType.
The type defines the following properties:
'IsCollectible' of type System.Boolean
'DeclaringMethod' of type System.Reflection.MethodBase
'FullName' of type System.String
'AssemblyQualifiedName' of type System.String
'Namespace' of type System.String
'GUID' of type System.Guid
'IsEnum' of type System.Boolean
'IsConstructedGenericType' of type System.Boolean
'IsGenericType' of type System.Boolean
'IsGenericTypeDefinition' of type System.Boolean
'GenericParameterAttributes' of type System.Reflection.GenericParameterAttributes
'IsSZArray' of type System.Boolean
'GenericParameterPosition' of type System.Int32
'ContainsGenericParameters' of type System.Boolean
'StructLayoutAttribute' of type System.Runtime.InteropServices.StructLayoutAttribute
'IsFunctionPointer' of type System.Boolean
'IsUnmanagedFunctionPointer' of type System.Boolean
'Name' of type System.String
'DeclaringType' of type System.Type
'Assembly' of type System.Reflection.Assembly
'BaseType' of type System.Type
'IsByRefLike' of type System.Boolean
'IsGenericParameter' of type System.Boolean
'IsTypeDefinition' of type System.Boolean
'IsSecurityCritical' of type System.Boolean
'IsSecuritySafeCritical' of type System.Boolean
'IsSecurityTransparent' of type System.Boolean
'MemberType' of type System.Reflection.MemberTypes
'MetadataToken' of type System.Int32
'Module' of type System.Reflection.Module
'ReflectedType' of type System.Type
'TypeHandle' of type System.RuntimeTypeHandle
'UnderlyingSystemType' of type System.Type
'GenericTypeParameters' of type System.Type[]
'DeclaredConstructors' of type System.Collections.Generic.IEnumerable1[System.Reflection.ConstructorInfo] 'DeclaredEvents' of type System.Collections.Generic.IEnumerable1[System.Reflection.EventInfo]
'DeclaredFields' of type System.Collections.Generic.IEnumerable1[System.Reflection.FieldInfo] 'DeclaredMembers' of type System.Collections.Generic.IEnumerable1[System.Reflection.MemberInfo]
'DeclaredMethods' of type System.Collections.Generic.IEnumerable1[System.Reflection.MethodInfo] 'DeclaredNestedTypes' of type System.Collections.Generic.IEnumerable1[System.Reflection.TypeInfo]
'DeclaredProperties' of type System.Collections.Generic.IEnumerable1[System.Reflection.PropertyInfo] 'ImplementedInterfaces' of type System.Collections.Generic.IEnumerable1[System.Type]
'IsInterface' of type System.Boolean
'IsNested' of type System.Boolean
'IsArray' of type System.Boolean
'IsByRef' of type System.Boolean
'IsPointer' of type System.Boolean
'IsGenericTypeParameter' of type System.Boolean
'IsGenericMethodParameter' of type System.Boolean
'IsVariableBoundArray' of type System.Boolean
'HasElementType' of type System.Boolean
'GenericTypeArguments' of type System.Type[]
'Attributes' of type System.Reflection.TypeAttributes
'IsAbstract' of type System.Boolean
'IsImport' of type System.Boolean
'IsSealed' of type System.Boolean
'IsSpecialName' of type System.Boolean
'IsClass' of type System.Boolean
'IsNestedAssembly' of type System.Boolean
'IsNestedFamANDAssem' of type System.Boolean
'IsNestedFamily' of type System.Boolean
'IsNestedFamORAssem' of type System.Boolean
'IsNestedPrivate' of type System.Boolean
'IsNestedPublic' of type System.Boolean
'IsNotPublic' of type System.Boolean
'IsPublic' of type System.Boolean
'IsAutoLayout' of type System.Boolean
'IsExplicitLayout' of type System.Boolean
'IsLayoutSequential' of type System.Boolean
'IsAnsiClass' of type System.Boolean
'IsAutoClass' of type System.Boolean
'IsUnicodeClass' of type System.Boolean
'IsCOMObject' of type System.Boolean
'IsContextful' of type System.Boolean
'IsMarshalByRef' of type System.Boolean
'IsPrimitive' of type System.Boolean
'IsValueType' of type System.Boolean
'IsSignatureType' of type System.Boolean
'TypeInitializer' of type System.Reflection.ConstructorInfo
'IsSerializable' of type System.Boolean
'IsVisible' of type System.Boolean
'CustomAttributes' of type System.Collections.Generic.IEnumerable1[System.Reflection.CustomAttributeData] ---> System.InvalidOperationException: Method may only be called on a Type for which Type.IsGenericParameter is true. at System.RuntimeType.get_DeclaringMethod() at Serialize(Byte[][], Object[], JsonWriter&, RuntimeType, IJsonFormatterResolver) at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Serialize(Byte[][], Object[], JsonWriter&, NullableContextAttribute, IJsonFormatterResolver) at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
--- End of inner exception stack trace ---
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.CollectionFormatterBase4.Serialize(JsonWriter& writer, TCollection value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) --- End of inner exception stack trace --- at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
at Serialize(Byte[][], Object[], JsonWriter&, RouteEndpoint, IJsonFormatterResolver)
at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver) --- End of inner exception stack trace --- at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter1.Serialize(JsonWriter& writer, T value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver)
at Elasticsearch.Net.Utf8Json.Formatters.DictionaryFormatterBase5.Serialize(JsonWriter& writer, TDictionary value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.Formatters.DynamicObjectTypeFallbackFormatter.Serialize(JsonWriter& writer, Object value, IJsonFormatterResolver formatterResolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.SerializeUnsafe[T](T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.Utf8Json.JsonSerializer.Serialize[T](Stream stream, T value, IJsonFormatterResolver resolver) at Elasticsearch.Net.LowLevelRequestResponseSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) at Elasticsearch.Net.DiagnosticsSerializerProxy.Serialize[T](T data, Stream stream, SerializationFormatting formatting) at Elasticsearch.Net.PostData1.Write(Stream writableStream, IConnectionConfigurationValues settings)
at Elasticsearch.Net.HttpConnection.SetContent(HttpRequestMessage message, RequestData requestData)
at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters) --- End of inner exception stack trace --- at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticLowLevelClient.Bulk[TResponse](PostData body, BulkRequestParameters requestParameters)
at NLog.Targets.ElasticSearch.ElasticSearchTarget.SendBatch(ICollection`1 logEvents)
2024-04-16 17:32:46.4161 Trace AsyncWrapper_ElasticSearchTarget(Name=elasticsearch): Writing 1 events (Timer)
2024-04-16 17:36:38.0247 Debug Message Template Auto Format enabled
2024-04-16 17:36:38.0579 Info Loading assembly name: NLog.Targets.ElasticSearch
2024-04-16 17:36:38.0782 Info NLog.Targets.ElasticSearch, Version=1.0.0.0, Culture=neutral, PublicKeyToken=3eb62cd298662c6a. File version: 1.0.0.0. Product version: 1.0.0.

[snakefoot]

The bug is in NEST / Elastic-Client and not in the NLog-target. You should report the bug to Elastic/NEST-support.

Alternative you can try and protect yourself from the very fragile Json-Serializer in NEST / Elastic-Client by specifying MaxRecursionLimit="1" for the NLog ElasticTarget.

See also: https://github.com/markmcdowell/NLog.Targets.ElasticSearch/wiki

[fsimonovski]

Hi,

This is a duplicate of elastic/elasticsearch-net#8132

For this issue, when setting NLog to log to file target only, I have no problems. Only when trying to log onto elastic cluster v7.10.2, on NET8.0. That is why I am trying to resolve this issue here if possible.
Here is a link to a sample repo, which has the same problem, when we take a look at the internallog file which is generated after running the webapi project.
It only needs a running elastic cluster to see the problem
https://github.com/fsimonovski/NLog-ElasticSearch-Net8

Thanks for the help.

[snakefoot]

Can only repeat that you can use work around by setting MaxRecursionLimit="1" The bug is in NEST / Elastic-Client, and this is not support forum for NEST / Elastic-Client

[fhimaj]

Replying as was suggested to try for a solution here. Currently experiencing a similar exception:NLog/NLog#5616 .
Setting the 'MaxRecursionLimit=1' does not solve the issue. Did you find a solution @fsimonovski ?