update indexing notebook

[PhilippeMoussalli]

Updated indexing notebook.

For now we're using the version from this PR to allow using resuable components in notebooks.

• Converted the custom cleaning component and chunking component to lightweight components.
• Added TODOs to change after having a stable release

Future Todos:

• There is an error when using the custom cleaning component related to the way the component text is parsed (related to presence of \n. Requires further investigation. (This PR)
• Change write index components to lightweight component (This PR)
• Switch over the two remaining notebooks ti lightweight components as well (future PRs)

[mrchtr]

I had a look at the changes. Imo, we still need to define which components we are transforming into lightweight components and which we keep as reusable.

I would keep the chunking component and the write to Weaviate as reusable. Any arguments for why we should implement these as custom components?

The aggregate_eval_results and text_cleaning seem to be good examples for custom components. I would only convert them.

[RobbeSneyders]

Thanks @mrchtr!

The not so nice thing about having these components in separate files, is that we still implement them as if they are reusable. We are still wrapping a not-so-nice interface around existing libraries, while I would just "hardcode" the arguments used into the components since they're lightweight. This would simplify the component a lot, but probably works better if the components are inline.

CC: @GeorgesLorre

[RobbeSneyders]

Hmm, we might need this in the other use cases as well. Didn't run into them, but might be because I already had the dependencies in my environment. Will try to re-run with a clean one.

[RobbeSneyders]

Do we need this here if it's in the extra_requires of the component?

[mrchtr]

Yep, we do have langchain as extra requirement. But at the moment we are not installing the extra requirements in the local environment. When I am in the notebook, and want to test the class implementation, I will get an exception that the dependency is missing. I think this is something we should takle in #830

[RobbeSneyders]

But we are not doing the same for eg. Ragas?

Since we are not actually executing the components in the notebook at this moment, I would leave them out of the requirements.txt.

[GeorgesLorre]

The not so nice thing about having these components in separate files, is that we still implement them as if they are reusable. We are still wrapping a not-so-nice interface around existing libraries, while I would just "hardcode" the arguments used into the components since they're lightweight. This would simplify the component a lot, but probably works better if the components are inline.

Thx @mrchtr, looking good. But I do tend to agree here that is almost to nicely abstracted now, they are reusable components with a decorator on top now. I don't feel like we get any of the lightweight component benefits. I would really try to dumb them down by defining them inline (in the notebook). Removing optional functionality. Hardcoding most of the arguments and by doing so removing most of the docstrings. This should make them more compact and understandable at a glance.

Let me know if this does not make sense.

[RobbeSneyders]

But we are not doing the same for eg. Ragas?

Since we are not actually executing the components in the notebook at this moment, I would leave them out of the requirements.txt.

[mrchtr]

The not so nice thing about having these components in separate files, is that we still implement them as if they are reusable. We are still wrapping a not-so-nice interface around existing libraries, while I would just "hardcode" the arguments used into the components since they're lightweight. This would simplify the component a lot, but probably works better if the components are inline.

Thx @mrchtr, looking good. But I do tend to agree here that is almost to nicely abstracted now, they are reusable components with a decorator on top now. I don't feel like we get any of the lightweight component benefits. I would really try to dumb them down by defining them inline (in the notebook). Removing optional functionality. Hardcoding most of the arguments and by doing so removing most of the docstrings. This should make them more compact and understandable at a glance.

Let me know if this does not make sense.

Alright, reduced the notebooks to a minimum.
Should we remove the python pipelines completely? There are not needed anymore when we don't support the parameter search.

[RobbeSneyders]

Should we remove the python pipelines completely? There are not needed anymore when we don't support the parameter search.

I'm a fan of having both in our use cases, the notebooks as explanation, and the python files to actually build on. On the other hand, I see the duplication we'd be introducing.

WDYT @GeorgesLorre?