couldn't understand behaviour

[RaiAmanRai]

hi, i am trying to use stump, below are two data graphs

In the first case, the initial pattern is recognnized, but it fails to identify the obvious repeating pattern starting around time=50, but it does get recognized when i trim the initial pattern.

keep in mind, i am also using the same configurations/parameters for both the cases.
Is there something that i am doing wrong or am i missing something?

[seanlaw]

@RaiAmanRai Thank you for your question and welcome to the STUMPY community. Can you please provide the same code that you had used to:

1. generate the time series
2. compute the matrix profile(s)
3. plot the figures
4. identify the motifs

In the first case, the initial pattern is recognnized, but it fails to identify the obvious repeating pattern starting around time=50, but it does get recognized when i trim the initial pattern.

Please provide the inputs/code requested above. Looking at your matrix profiles (and ignoring the grey rectangular boxes and dashed lines), STUMPY is doing exactly what is expected. Low (near zero) values in your matrix profile implies that an (near) exact match (i.e., one-nearest neighbor) has been found so the "obvious repeating pattern" is clearly being identified as having a close match (again, evidenced by near-zero matrix profile values). In fact, this is the case in BOTH the upper and lower matrix profiles

[RaiAmanRai]

Hi @seanlaw Yes I agree wioth you that STUMPY is able to generate near(zero) matrix profiles, but i was more of expecting the shaded regions and dashed lines as well. Maybe I am doing something wrong, could you help me out on this.
Here is the I used for generating the graphs.

encoded_char = [ ord(line[26]) for line in lines if len(line) == 28]
l = encoded_char[:]

your_time_series = np.array(l, dtype="float64")
window_size = 5  # Approximately, how many data points might be found in a pattern

mp = stumpy.stump(your_time_series, m=window_size)

motifs = stumpy.motifs(
    your_time_series,
    mp[:, 0],
    max_matches=len(l)//window_size)

motif_idxs = motifs[1][0].tolist()
nearest_neighbor_idxs = [ mp[motif_idx, 1] for motif_idx in motif_idxs ]

# for plotting graphs
fig, axs = plt.subplots(2, sharex=True, gridspec_kw={'hspace': 0})
plt.suptitle('Motif (Pattern) Discovery', fontsize='30')

axs[0].plot(your_time_series)
axs[0].set_ylabel('Steam Flow', fontsize='20')

for i in range(len(motif_idxs)):
    rect = Rectangle((motif_idxs[i], 0), window_size, 40, facecolor='lightgrey')
    axs[0].add_patch(rect)
    rect = Rectangle((nearest_neighbor_idxs[i], 0), window_size, 40, facecolor='lightgrey')
    axs[0].add_patch(rect)
    
axs[1].set_xlabel('Time', fontsize ='20')
axs[1].set_ylabel('Matrix Profile', fontsize='20')
for i in range(len(motif_idxs)):
    axs[1].axvline(x=motif_idxs[i], linestyle="dashed")
    axs[1].axvline(x=nearest_neighbor_idxs[i], linestyle="dashed")
axs[1].plot(mp[:, 0])
plt.show()

The encoded_char consists of ascii value of characters.

For the second graph, I just trimmed the input series by changing line two to
l = encoded_char[50:]

[NimaSarajpoor]

@RaiAmanRai
Could you please provide your data your_time_series?

# import numpy as np
# your_time_series = np.array(l, dtype="float64")

np.save("your_time_series.npy", your_time_series)

And, then I think you should be able to drag and drop the file your_time_series.npy here

---

side note: I think the default value for argument max_motifs in stumpy.motifs is one. I think that means you are getting one motif (and its matches). So, the one you are looking for has different shape. So, probably if you set max_motifs=2 (or higher), you may get the one you are talking about. But, I think it is usually a good idea to share your data if you can so others can try out your code and see the result.

[RaiAmanRai]

@NimaSarajpoor sure, the input file has been attached below
/opt/keystroke_pattern/your_time_series.npy

Also, I tried playing around with max_motifs but still got the same result

[NimaSarajpoor]

@RaiAmanRai
It seems .npy is not supported here. Could you save it as csv and try again? (or maybe just zip the .npy file and drop it here?)

[RaiAmanRai]

@NimaSarajpoor
okay trying it

[NimaSarajpoor]

Something like this (I just drag and dropped the file here)

Example.csv

[RaiAmanRai]

@NimaSarajpoor sorry for inconvinience, I made a mistake, was trying to directly drag and drop from remote machine
your_time_series.csv

[NimaSarajpoor]

@RaiAmanRai
So, after some investigation, I realized the problem is that the default value of cutoff will be set to 0 in your case (cutoff, when not provided, is a function of matrix profile values). This means only motifs that have distance <= 0 are considered. This is an extreme condition because it ONLY considers distance=0 (the lowest possible distance value). So, I think the following code should resolve this issue:

cutoff = np.sort(np.unique(P))[1] # I am choosing the second-smallest value for cutoff
  
motif_dist, motif_IDX = stumpy.motifs(
    T, 
    P, 
    min_neighbors=1,
    max_distance=None,
    cutoff=cutoff,
    max_matches=10,
    max_motifs=3,
)

Please let me know if this works or not.

Btw, a better solution might be to check the default value of cutoff, and if zero, then replace it with what I proposed above.