Advice on improving performance

[PabloRuiz98]

Hello,

I'm a master's degree student, and for my final project, we are working on a mouse model to quantify various sexual and social interaction variables using SLEAP.

We have already recorded videos, and since they were initially intended for creating an ethogram, their conditions and quality may not be the best.

The general quality of the videos and the skeleton are as follows:

I'm using these parameters (bottom-up):

Anyway, we are trying to make it work. I have already labeled around 700 frames (evenly distributed across all videos), but I have found some mixed results:

FIRST PROBLEM: MERGED SKELETONS

This is one of the biggest problems I'm facing. In a considerable number of social interaction frames (usually the sexual ones), the skeletons of the two animals are merging.

I'm trying to fix this by labeling more frames during interactions (as mentioned in the troubleshooting guide). I think this is helping a lot, but I still have these errors in many frames.

SECOND PROBLEM: MOTION BLUR

In some frames, motion blur makes it impossible to know what is exactly happening. My doubst are about how I should label these cases:

1-In some videos, I can infer what is happening in the frame, so I usually mark nodes as visible, even when they are not, based on their probable real location.

2-In other videos, it is almost impossible to know what is happening.

How do I need to proceed with both cases?

Some things that I have tried:

• Activating online mining: I think I am getting better results with this.
• Lowering inputs scaling: I read in a post that the receptive field should be around the height of the animal, but when I lower the input scaling to 0,5-0,6, I get poor results.

Can I get some insight or advice on how I can achieve better results? I don't mind labeling tons of new frames, but I'm worried that the issue is more related to a parameter configuration thing.

Thank you and I really apreciate your help :)

[roomrys]

Hi @PabloRuiz98,

Thanks for reaching out and providing detailed information!

Merged Skeletons

I see you are using a bottom-up pipeline, in which case it is true that a receptive field (RF) about the size of the animal generally should help - however, I expected to see truncation of the animal pose when using a smaller RF which does not seem to appear in your example images. I attribute this to using shorter edges in your skeleton. If you wanted to play with increasing the RF, you could increase the Max Stride (but this will increase memory costs and training time) instead of decreasing the input scaling.

The size of the RF is important in bottom-up models because the model needs to be able to see source/destination nodes together (in the RF) to be able to reason which source/destination nodes should be connected to each other (i.e. which nose node connects to which head node).

Merged skeletons happen when we can see some nodes from one animal and some (missing) nodes from another animal and they get grouped together. So, we can try to solve this in two ways. We could:

1. Get our model to predict the missing (but visible) nodes [do try]
   The idea here is that if all nodes are predicted, then our model might have a chance at grouping the correct nodes together in the same instance. The model may already be predicting some of these missing nodes, but just with extremely low confidence, hence they get filtered out before SLEAP even turns them into predictions.

   If you are training in the GUI, the heat maps that pop up during training could be helpful to guide your intuition of the model.

   During training, we could increase the sigma value (between 3 and 5) to make key points easier to learn (although their predictions will be less precise). When doing this, we want to be careful not to make sigma too large as we might accidentally mask the detection of multiple key-points as a single probability distribution.

   Alternatively, during inference, we could use the CLI sleap-track command with the --peak_threshold option followed by a float between (0, 0.2):

     --peak_threshold PEAK_THRESHOLD
                       Minimum confidence map value to consider a peak as valid.
   

   where 0.2 is the default value.

2. Crop which nodes are even considered for grouping [do NOT try]
   The idea here is that if we only consider nodes within n-by-n pixels of a "centroid" node then we might be able to filter some of these merged skeletons.

   This would require switching to a top-down model and selecting an anchor/centroid node which we can then use on model to find and take a crop about. Each crop represents an instance of an animal. A second model then uses these crops to find all key-points. The grouping of the key-points is implicit to the crop - the highest confidence key-point of a node in the crop will be used for that specific node in the prediction of the instance.

   While this is an option, I don't really see it solving our problem because:

   • If we opt for a central "centroid" node, then instead of merged skeletons, we'll see more missing predictions (due to an occluded centroid node - especially during mounting behavior).
   • And if we opt for a non-central "centroid" node, we need a larger crop size to fit the entire instance which defeats the purpose of our crop in the first place.

Motion Blur

Marking key-points as visible in cases where you can infer where they are isn't a bad strategy. It could lead to some variance in the location of the predicted key-points from frame to frame. In my experience, I prefer to leverage what the model thinks it knows to help me annotate difficult frames and even converge towards where I should be labeling key-points.

On frames where you have no clue what is happening, you could scrub left/right through adjacent frames to try and make sense of the pose. If you are still unsure about where to annotate even with temporal clues from adjacent frames, then I would advise skipping that frame for now. Perhaps you will get a prediction later on that looks correct and can help guide your annotation. Otherwise, leave it unlabeled.

Thanks,
Liezl