Azure Kinect bodies deserialization exception

[AuMilliat]

Hello,

I'm a struggling to debug an issue with kinect bodies. In our setup we use a application (and computer) dedicated to broadcast Kinect streams throught RendezVous Process with RemoteExporter/Importer (in TCP). And a server application to record.

For bodies with have an InvalidCastException on the reception side:

System.InvalidCastException
  HResult=0x80004002
  Message=Impossible d'effectuer un cast d'un objet de type 'Microsoft.Psi.AzureKinect.AzureKinectBody' en type 'System.Double[]'.
  Source=Microsoft.Psi
  StackTrace:
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.RefHandler`1.UntypedDeserialize(BufferReader reader, Object& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 162
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 83
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.ArraySerializer`1.Deserialize(BufferReader reader, T[]& target, SerializationContext context) in Microsoft.Psi.Serialization\ArraySerializer.cs:line 41
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.DictionarySerializer`2.DictionarySerializerImpl.Deserialize(BufferReader reader, Dictionary`2& target, SerializationContext context) in Microsoft.Psi.Serialization\DictionarySerializer.cs:line 57
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.ArraySerializer`1.Deserialize(BufferReader reader, T[]& target, SerializationContext context) in Microsoft.Psi.Serialization\ArraySerializer.cs:line 45
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Serialization.RefHandler`1.Deserialize(BufferReader reader, T& target, SerializationContext context) in Microsoft.Psi.Serialization\RefHandler.cs:line 92
   at Microsoft.Psi.Data.PsiStoreStreamReader.Deserialize[T](SerializationHandler`1 handler, BufferReader br, Envelope env, Boolean isDynamic, Boolean isRaw, T objectToReuse, String typeName, IDictionary`2 schemas) in Microsoft.Psi.Data\PsiStoreStreamReader.cs:line 286
   at Microsoft.Psi.Data.PsiStoreStreamReader.<>c__DisplayClass42_0`1.<OpenStream>b__1(BufferReader br, Envelope e) in Microsoft.Psi.Data\PsiStoreStreamReader.cs:line 242
   at Microsoft.Psi.Data.PsiStoreStreamReader.MoveNext(Envelope& envelope) in Microsoft.Psi.Data\PsiStoreStreamReader.cs:line 79
   at Microsoft.Psi.Data.Importer.StreamImporter.Next(Boolean moreDataPromised, Envelope env) in Microsoft.Psi.Data\Importer.cs:line 152
   at Microsoft.Psi.Pipeline.<>c__DisplayClass107_0`1.<CreateReceiver>b__0(Message`1 m) in Microsoft.Psi\Pipeline.cs:line 210
   at Microsoft.Psi.Executive.PipelineElement.<>c__DisplayClass43_1`1.<TrackStateObjectOnContext>b__1() in Microsoft.Psi.Executive\PipelineElement.cs:line 103
   at Microsoft.Psi.Receiver`1.<>c__DisplayClass13_0.<.ctor>b__0(Message`1 m) in Microsoft.Psi\Receiver.cs:line 89
   at Microsoft.Psi.Receiver`1.DeliverNext() in Microsoft.Psi\Receiver.cs:line 236
   at Microsoft.Psi.Scheduling.Scheduler.ExecuteAndRelease(SynchronizationLock synchronizationObject, Action action, SchedulerContext context) in Microsoft.Psi.Scheduling\Scheduler.cs:line 393
   at Microsoft.Psi.Scheduling.Scheduler.Run(Object workItem) in Microsoft.Psi.Scheduling\Scheduler.cs:line 350
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

I'm not sure it's \psi... sometimes it works perfectly and some other not, it's not the network as we have the same issue with the two applications running on a single computer. We have tested with differents applications (console, wpf...).

Any idea to where to check ?
Thanks!

[sandrist]

Can you show us your code on the receiver side where this exception is being raised? Wherever you are setting up the receiver side stream of AzureKinectBody. It looks like the system is trying to cast it to a System.Double[] array, but I'm not sure why that would be.

[AuMilliat]

From what I understand it could be the CoordinateSystem that trigger the exception as it's a DenseMatrix.

if (endpoint is Rendezvous.RemoteExporterEndpoint remoteExporterEndpoint)
{
     var remoteImporter = remoteExporterEndpoint.ToRemoteImporter(pipeline);
     foreach (var stream in remoteExporterEndpoint.Streams)
     {
         if (stream.StreamName.Contains("Bodies"))
         {
             OutBodies = Connection<List<AzureKinectBody>>(stream.StreamName, remoteImporter);
             break;
         }
     }
}

......

  protected virtual Emitter<T>? Connection<T>(string name, RemoteImporter remoteImporter)
  {
      if (remoteImporter.Connected.WaitOne() == false)
      {
          logStatus(Configuration.RendezVousApplicationName + " failed to connect stream " + name);
          return null;
      }
      logStatus(Configuration.RendezVousApplicationName + " stream " + name + " connected.");
      var stream = remoteImporter.Importer.OpenStream<T>(name).Out;
      if (Configuration.Debug)
          stream.Do((m, e) => { logStatus($"Message recieved on {name} @{e.OriginatingTime} : {m}"); });
      return stream;
  }

On the other side:

     RemoteExporter skeletonExporter = new RemoteExporter(ParentPipeline, portCount++, Configuration.ConnectionType);
     skeletonExporter.Exporter.Write(Sensor.Bodies, $"{Configuration.RendezVousApplicationName}_Bodies");
     exporters.Add(skeletonExporter.ToRendezvousEndpoint(Configuration.IpToUse));