How to interpret stumpy.motifs?

[SaVoAMP]

Hey, me again!

I have trouble understanding stumpy.motifs.
I had understood the API to find a desired number of motifs (max_motifs) after calling stumpy.motifs as well as their nearest neighbors (max_matches). Or can I just search different motifs in stumpy.motifs and need stumpy.macth to get their nearest neighbors as well?

I have written the following lines of code:

mp_pre = stumpy.gpu_stump(recording['pre_mag_magnitude'], sub_len)
motifs = stumpy.motifs(T=recording['pre_mag_magnitude'], P=mp_pre[:, 0], max_matches=10, max_motifs=3)

As a result I get a tuple with two ndarrays of size (1,2) - i.e. the motif_distances and motif_indices.
How can I interpret this now? Why do the arrays only have shape (1,2) and not (3,2) when I am looking for 3 different motifs? Concretely I receive motifs[0][0, 0] = 0 (self match) and motifs[0][0, 1] = 8.06108 (probably the distance to the nearest neighbor?) with the corresponding indices motifs[1][0, 0] = 9823 and motifs[1][0, 1] = 9119. Wouldn't I have to have two more rows each for the other two motif pairs here?
And where is my max_matches information? I thought that by setting max_matches=10 I would also automatically get the 10 nearest neighbors for each of the subjects as with stumpy.match?

[seanlaw]

@SaVoAMP Good question. It sounds like we need to add some more details to the documentation to make these things clearer. So, in an attempt to conserve memory/space, we opted to only return essential information to the user that satisfies several key criterion that are defined by the input parameters. When you set max_matches=10, this is the upper limit as to how many matches are allowed to be returned for each "candidate motif". As you can imagine, for each "candidate motif", it is possible for it to have many, many matches (i.e., beyond 10) OR it can have no matches (i.e., when the distance between the candidate and the matches is greater than max_distance). Similarly, when you set max_motifs=3, this is the upper limit as to how many motifs are allowed to be returned. Note that a "candidate motif" must have at least one nearest neighbor that is within a distance that is equal to or smaller than cutoff.

So, in theory, if everything works out nicely and you have great candidate motifs (that aren't disqualified) that then have an abundance of matches (that aren't disqualified), then you should expect to get a motif_distances of shape (3, 10). However, since you are getting something of size (1, 2) implies that only one candidate motif was found (within some cutoff criteria) and it had two matches (that had a distance below max_distance). All other candidate motifs had a distance to a nearest neighbor that was larger than the default cutoff distance AND the other matches have a distance to the candidate motif that is larger than max_distance.

Of course, you can set the cutoff to something large like np.inf and also set the max_distance to something large like np.inf and then you'll guarantee that the arrays will come back with shape (3, 10) (or, in general, shape (max_motifs, max_matches)). However, the key here is that not all matches are created equal (i.e., the quality of the matches will differ by their distance to the candidate motif) and not all motifs are created equal (i.e., you can have motifs that have a one nearest neighbor that is quite far away). So, the default cutoff and max_distance apply some reasonable controls to ensure that the identified motifs and their matches are significant (namely, less than 2 standard deviations from the mean distance).

Now that I think about it, perhaps it would make more sense to always make sure that the returned arrays have shape (max_motifs, max_matches) but then "blank out" the parts where the cutoff/max_distance constraints were not satisfied.

Let me know if that makes sense. I'd be curious to get your feedback so that we can improve this.

[SaVoAMP]

I tried to execute stumpy.match with the parameters cutoff = np.inf and ´max_distance = np.inf´ but it did not work.
There was no error message but the program seems to get stuck in a loop somewhere.

Strangely setting ´cutoff = np.inf´ only, gets me the right number of motifs but using ´max_distance = np.inf´ gets the program stuck.

I'm only able to get the desired number of matches using ´stumpy.match()´ and setting the same parameter ´max_distance = np.inf´ seems to work fine in this case.

[seanlaw]

I tried to execute stumpy.match with the parameters cutoff = np.inf and ´max_distance = np.inf´ but it did not work.

There was no error message but the program seems to get stuck in a loop somewhere.

Strangely setting ´cutoff = np.inf´ only, gets me the right number of motifs but using ´max_distance = np.inf´ gets the program stuck.

I'm only able to get the desired number of matches using ´stumpy.match()´ and setting the same parameter ´max_distance = np.inf´ seems to work fine in this case.

There is a small bug that was just fixed when max_distance=np.inf (see Issue #501). Can you please try setting it to an absurdly large finite number like max_distance=1e20 instead?

[SaVoAMP]

Ahh, thank you!
I updated stumpy to the latest version and it works now.

[seanlaw]

Glad to hear that. Happy holidays, @SaVoAMP!

[SaVoAMP]

Thank you, I wish you happy holidays too!