[Feature Request]: Template to integrate LLMs as PTransform using langchian-beam

[Ganeshsivakumar]

Related Template(s)

New template

What feature(s) are you requesting?

What you would like to accomplish:

• Leverage LLMs capabilities for data processing and create RAG based pipelines in dataflow and apache beam.
• Create vector embeddings for text directly within the pipeline.

How this might work:

langchain-beam java library is a apache beam and langchain integration. it provides transforms to integrate LLMs and embeddings models from gemini, openai and other providers in beam pipeline.

Goal is expose those expose those transform as template for easy integration in Dataflow.

repository link - https://github.com/Ganeshsivakumar/langchain-beam