Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to filter fields #4

Open
BenFradet opened this issue Jul 3, 2017 · 3 comments
Open

Add ability to filter fields #4

BenFradet opened this issue Jul 3, 2017 · 3 comments
Assignees

Comments

@BenFradet
Copy link
Contributor

from snowplow/snowplow#3195:

At the moment the index is comprised of many many fields that are not used as a side effect of mapping the atomic definition to Elasticsearch. To reduce the size of the index it would be nice to be able to control what fields we care about storing in the index.

@BenFradet
Copy link
Contributor Author

@jbeemster Did you mean having a filter that would only keep e.g. network_userid and domain_userid?

@jbeemster
Copy link
Member

@BenFradet we were thinking about a bit of a blacklist where after you had processed the enriched event you could just drop the key from the final JSON that gets sent to Elasticsearch.

At the moment because we essentially translate it to a atomic.events definition there are a lot of fields that are always null. If this could be known ahead of time you could just remove these keys and never have them added to the mapping in the Elasticsearch index.

@alexanderdean alexanderdean removed this from the Version 0.9.0 milestone Jul 12, 2017
@alexanderdean
Copy link
Member

De-scheduled - not clear how/when we would do this...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants