(workflows) Decide whether to put algorithm in Seer or Sentry

[Mitan]

Sentry:
Pros:

• No need to make extra API calls to Seer
• Easier to re-use for other projects

Cons:

• Need to bring extra dependencies into Sentry or re-create everything in pure python

Seer:
Pros:

• Easier to implement (access to all python libraries)

Cons:

• Extra API call to Seer every time (latency, reliability)

[Mitan]

Libraries in order of importance

1. pandas for data manipulation. Pandas is needed for re-indexing and sorting the data:

distribution = pd.Series({item.get('label', 'value_missing'): item['value'] for item in attr['buckets']})
distribution.reindex(all_keys, fill_value=0)

Computing entropies, KL and sorting the values:

df_results = pd.DataFrame(attrs, columns=['attribute', 'KL score'])
df_results = df_results[df_results['KL score'] > alpha_kl]
df_results['entropy'] = df_results['attribute'].apply(lambda x: entropy(normalized_outlier[x]))
return df_results.sort_values(by='entropy', ascending=True).head(top_k)

2. . scipy (Scipy has numpy as a dependency). The potentially can be implemented manually.

from scipy.stats import entropy