Replicating/recalculating regimes

[rnjv]

Hi,
Thank you for this library and to main authors.

I am wondering how would one recalculate the regimes (perhaps for a larger sequence) like being output by stumpy.snippets given a snippet subsequence found by the algorithm?

Should one just use MASS for the search, how would you deal with finding the right k to reflect regimes found by the snippets module, if so?

Update: Tried MASS search with count of regimes found as k. Top k MASS matches are not the same indexes as the regimes.

I still have not found the answer to getting additional regimes location independent of snippet module and how it differs to the distance profile from MASS.

[seanlaw]

@rnjv Are you able to provide a minimum-reproducible-example (with sample data and code used)? This will help make your question a bit more concrete. Otherwise, we are only able to guess at what you are trying to accomplish.

@NimaSarajpoor Given your knowledge and work on snippets, do you have any suggestions for @rnjv?

[NimaSarajpoor]

@rnjv
As @seanlaw pointed out, it is often better to have some data / some code.

Let's go over your post piece by piece...

I am wondering how would one recalculate the regimes (perhaps for a larger sequence) like being output by stumpy.snippets given a snippet subsequence found by the algorithm?

Can you please clarify what you mean by larger "sequence"?

• Do you mean longer time series (compared to the original time series T that was initially used to find snippets)
• Or, do you mean longer snippet (but using the same time series T)?
• Or, do you mean longer subsequence (but using the same time series T)?

Note: In function snippet, the length of snippet can be set by param m, and the length of subseq can be set by param s. If you are not aware of their difference, I think reading the original paper can help you with that.

I am assuming that you are looking for the first one, where you find k snippets in T, and then you want to use the k snippets to find their matches in a new, longer time series (If your question is something else, please try to elaborate your question)

Should one just use MASS for the search, how would you deal with finding the right k to reflect regimes found by the snippets module, if so?

I am going to answer the second part of your question first:
"How to find the right k?" see this snippet tutorial notebook

Now, your second question: "Should one just use MASS for the search?". And,

Update: Tried MASS search with count of regimes found as k. Top k MASS matches are not the same indexes as the regimes.

I still have not found the answer to getting additional regimes location independent of snippet module and how it differs to the distance profile from MASS.

Note that core.mass computes the distance between a subseq and other subseqs. However, IIRC, snippet uses "mpdist" distance measure. What is "mpdist"? In short, "mpdist" distance between two sequence is computed based on the pairwise distances between their subsequences. For further information, try to read the mpdist paper if it is not clearly explained in the snippet paper. Long story short, I think you need to use mpdist (see the "mpdist.py" script. Also, see the function _get_all_profiles in the snippet module and see how it is used. I guess you should find your answer there).

Now, if you have already k snippets and you want to know how these k snippets can segemtize your NEW time series, then I should say that is like a research question. Try to read the original paper as well as the STUMPY code, and see how you can come up with an algorithm that can appropriately(?) segmentize your new time series.

[rnjv]

Hi @seanlaw and @NimaSarajpoor ,

I did something like this:

for monthrange in iterdatebymonth:
    T = df[(df.index>=monthrange[0])&(df.index<=monthrange[1])]['feature']
    snippets, indices, profiles, fractions, areas, regimes = stumpy.snippets(T, m, k, s=s)

There could be month specific snippets, and I am trying to identify them. And, if I use a larger time period, I often get nil results.

Hence, my idea was to get these snippets for each month and search for its top k occurence (or replicate the same regime results in snippets) across a longer time period. At the same time, I wanted to make sure the regimes found by snippets remain consistent in the both the smaller (monthly) and larger space.

[NimaSarajpoor]

@rnjv
Thanks for sharing an example of your code. That is an important step. However, there are still some other steps that should be taken by you. Please see below:

1. Your code is not complete enough. What is iterdatebymonth? (Please note that I can guess but it is better that we have the exact, correct info regarding each part of the code)

2. Can you also provide the data (i.e. df)? You can save it in csv format and just drag and drop it into the comment box.

3. In your code, what is length of T? What are the values for m, k, and s?

4. When you say larger time period, do you mean larger m or s (or maybe T)? Now that you have provided the code, you should try to avoid using vague term such as time period and, instead, just say the exact name. For instance, you can say: larger value for parameter [m or s or T].

Also, when you say larger, does that mean any larger value? Can you provide us with the exact value that you tried?

5. Can you clarify this sentence please?

I often get nil results.

To get better feedback regarding your problem, you should try to be more precise. Instead of the word "often", please provide us with the code that you used and you got nil (empty?) output from stumpy.snippet.

6. Regarding your idea:

Hence, my idea was to get these snippets for each month and search for its top k occurence across a longer time period. At the same time, I wanted to make sure the regimes found by snippets remain consistent in the both the smaller (monthly) and larger space.

Two notes here:
i) please provide an example of the code and data you are using to achieve this. (i.e. the new data and the code you are trying to use to get top-k)

ii) Note that I still do not have enough info about your problem. So, I am going to guess what kind of data you have and what you want to do. I think you want to find snippet for time series {t_1, t_2, ..., t_n} and then use the discovered snippets and find their top-k matches in time series {t_1, t_2, ..., t_n, ...., t_N}, where N>>n. You also do not change the value of m and s in your new case, and you are using the same values that you used when you were computing the snippets of {t_1, t_2, ..., t_n} .
My opinion: I think it is not reasonable to expect to see the same regimes in both cases since you have more data now and the subsequence in {t_1, ..., t_n} can now be compared with new subsequences that contain values from {t_(n+1), ...., t_N}.