figure out a way to remove delimiters from dask.bag

[wikfeldt]

this:

sorted_counts = text.filter(lambda word: word not in DELIMITERS).str.lower().str.strip().str.split().flatten().frequencies().topk(10,key=1).compute()

is not the same thing as the filter function

[ashwinvis]

This was modified in python-perf

https://enccs.github.io/python-perf/parallelize/#dask-bag

text = db.read_text(filename, blocksize='1MiB')
filtered = (
    text
    .filter(lambda word: word not in DELIMITERS)
    .str.lower()
    .str.strip()
    .str.split()
    .flatten()
)
ddf = filtered.to_dataframe(columns=['words'])
ddf['words'].value_counts().compute()[:10]

I need to update it here.

[ashwinvis]

I remember now and understand what you meant. We need to use str.replace with regex or some kind of map + filter.