How to apply Multidimensional Motif Discovery for multiple data points?

[luomein]

Thanks for the interesting library. I read in the Multidimensional Motif Discovery sample code, it says, "we restrict our use of 'dimensions' to refer only to the number of time series and not to the number of data points". But what if I need to apply the algorithm for multiple data points, how could I do it?

For example, if I want to find same failure pattern(motif) across all failed devices, how could I apply stumpy for this kind of task? Thanks for your advice.

[seanlaw]

@luomein Welcome to the STUMPY community and thank you for your question. The first thing that I recommend you do is to go over both the "Multidimensional Motif Discovery Tutorial" (which I believe you had already referenced) and then understand the basic concepts in "Fast Pattern Matching". Then, I recommend going through this other discussion that goes into some detail about finding other matching patterns for a given motif. Then, feel free to ask clarifying questions and provide code examples of what you've tried and we'll do our best to assist.

[seanlaw]

So, "multi-dimensional" is an overloaded term when it comes to time series. So, for a simple time series like:

T = np.random.rand(100)

T contains 100 random data points. When we say "we restrict our use of 'dimensions' to refer only to the number of time series and not to the number of data points", this means that the number of "dimensions" in T is 1 and not 100. Note that both are technically correct but, for STUMPY, we refrain from ever referring to the number of data points as the number of "dimensions". Thus, something like:

T5 = np.random.rand(100).reshape(5, 20) 

has 5 "dimensions". Hopefully, this makes sense.

if there are more than 2 time series and each time series is multidimensional, how to find motif across different time series?

Thank you for clarifying. This helps! I don't know if this is correct terminology but I try to think of this as having two "sets" of multi-dimensional time series (assuming that each set contains the same number of related dimensions but may have different number of data points each)

What I could think of is to concat all the time series together to make single time series and solve it by the stumpy.mstump

Yes, that would be the solution. However, I recommend adding a np.nan between each time series and this will help avoid finding patterns/matches that straddle any two sets of time series.

T1 = np.random.rand(6).reshape(3, 2)
T2 = np.random.rand(9).reshape(3, 3)
T3 = np.random.rand(12).reshape(3, 4)

nan = np.full((T1.shape[0], 1), np.nan)

np.hstack([T1, nan, T2, nan, T3])

array([[0.49273097, 0.76261672,        nan, 0.08465868, 0.38402637,
        0.30669966,        nan, 0.62337789, 0.95539535, 0.85730367,
        0.04387865],
       [0.94183309, 0.23561322,        nan, 0.41927956, 0.02422469,
        0.59748816,        nan, 0.18186638, 0.21765156, 0.84131697,
        0.51594425],
       [0.63079202, 0.37158291,        nan, 0.33191575, 0.32868551,
        0.15556673,        nan, 0.28161051, 0.53560547, 0.52509972,
        0.05899535]])

This is here for demonstration purposes and may not be performant. Then, you should be able to use stumpy.mstump accordingly.

[luomein]

Great explanation, thank you. It is clear for me now. I will do some experiments and see how it goes.

[seanlaw]

Awesome! Out of curiosity, are you able to share any details of what you are thinking about using STUMPY for?

[luomein]

I am thinking of applying motif finding for device failure analysis.

For example, if I collect multiple sensors data from multiple failed devices, can I find some similar "failure path" (motif)?

I did not find any research goes in this way yet.

My assumption is, there may be two kinds of failure detection algorithm
(1) if every failed device has its own unique motif, then we could use outlier detection to find possible failed device.
(2) if most failed devices share some common motif (path to failure), then we could find out which motif is common in the failed devices.

[seanlaw]

That's wonderful and I think matrix profile is well suited for those applications. Thank you for sharing!