Improving Performance When Finding Patterns in Long Time Series Data

[zeydabadi]

Hi,
I wonder how I can use STUMPY to perform signal matching for two time-series when they are very long? (over a million time-steps)
I imagine this would be something similar to the example of finding the motif in two time-series; however, I have no domain knowledge for the time-series in hand, so I can't define an appropriate m. Also, I think the length of the time-series will be a problem.
Below is a plot of two time-series, and I want to know at what time step a pattern in Channel1 happens in Channel2 if it exists at all.

[seanlaw]

@zeydabadi Welcome and thank you for your question. So there are a few independent questions that you are asking so let's unpack things a little:

I wonder how I can use STUMPY to perform signal matching for two time-series when they are very long? (over a million time-steps)

The computational time depends on the hardware that you have available to you and the size of your input data. You might be able to get pretty far if you have a large server with 16 CPU cores but most users are running on a laptop with 2-4 cores. To get a general sense of the computational time, I recommend checking out this performance plot/table. Note that the performance info is outdated now and STUMPY should be faster than displayed but it should give you a rough idea. As you can see, the computational time is exponential (O(n^2)) as you increase the number of data points and, beyond 50K+ data points, we usually recommend the following strategies:

1. Computing an approximate matrix profile. This is pretty inexpensive and allows you to test out different window sizes too
2. Downsampling your data to, say, 10K data points. Combining this with approximate matrix profiles is a nice combination that can often get you most of the way there.
3. Increase the number of cores/servers. STUMPY can leverage Dask but it is up to you to set up your own Dask cluster.
4. Switch to using GPUs. Here's an example Google Colab notebook demonstrating how you can use their GPUs for 12-hours/day for free with STUMPY

I imagine this would be something similar to the example of finding the motif in two time-series; however, I have no domain knowledge for the time-series in hand, so I can't define an appropriate m.

The lack of domain knowledge is not unique to STUMPY and is a general problem to all time series analysis. However, this only means that you'll have to iterate and test out different values of m. We strongly recommend starting with some reasonable value of m and then doubling that value every time and re-computing the matrix profile. If you combine this with the downsampling and approximate matrix profile approach described above then you should be able to iterate quickly for finding a reasonable m.

Also, I think the length of the time-series will be a problem.

Please see the suggestions above

Below is a plot of two time-series, and I want to know at what time step a pattern in Channel1 happens in Channel2 if it exists at all.

Yes, this is exactly the point of computing a matrix profile. The minimum values tell you where there is conserved behavior in your data.