Update README docs for language transforms

transforms/language/doc_chunk/python/README.md

@@ -1,5 +1,16 @@
 # Chunk documents Transform 
 
+Please see the set of
+[transform project conventions](../../../README.md#transform-project-conventions)
+for details on general project conventions, transform configuration,
+testing and IDE set up.
+
+## Contributors
+
+- Michele Dolfi ([email protected])
+
+## Description 
+
 This transform is chunking documents. It supports multiple _chunker modules_ (see the `chunking_type` parameter).
 
 When using documents converted to JSON, the transform leverages the [Docling Core](https://github.com/DS4SD/docling-core) `HierarchicalChunker`
@@ -9,20 +20,26 @@ which provides the required JSON structure.
 
 When using documents converted to Markdown, the transform leverages the [Llama Index](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#markdownnodeparser) `MarkdownNodeParser`, which is relying on its internal Markdown splitting logic.
 
-## Output format
+
+### Input 
+
+| input column name | data type | description |
+|-|-|-|
+| the one specified in _content_column_name_ configuration | string | the content used in this transform |
+
+
+### Output format
 
 The output parquet file will contain all the original columns, but the content will be replaced with the individual chunks.
 
 
-### Tracing the origin of the chunks
+#### Tracing the origin of the chunks
 
 The transform allows to trace the origin of the chunk with the `source_doc_id` which is set to the value of the `document_id` column (if present) in the input table.
 The actual name of columns can be customized with the parameters described below.
 
 
-## Running
-
-### Parameters
+## Configuration
 
 The transform can be tuned with the following parameters.
 
@@ -40,6 +57,12 @@ The transform can be tuned with the following parameters.
 | `output_pageno_column_name`  | `page_number` | Column name to store the page number of the chunk in the output table. |
 | `output_bbox_column_name`    | `bbox` | Column name to store the bbox of the chunk in the output table. |
 
+
+
+## Usage
+
+### Launched Command Line Options 
+
 When invoking the CLI, the parameters must be set as `--doc_chunk_<name>`, e.g. `--doc_chunk_column_name_key=myoutput`.
 
 
@@ -63,8 +86,32 @@ ls output
 ```
 To see results of the transform.
 
+### Code example
+
+TBD (link to the notebook will be provided)
+
+See the sample script [src/doc_chunk_local_python.py](src/doc_chunk_local_python.py).
+
+
 ### Transforming data using the transform image
 
 To use the transform image to transform your data, please refer to the 
 [running images quickstart](../../../../doc/quick-start/run-transform-image.md),
 substituting the name of this transform image and runtime as appropriate.
+
+## Testing
+
+Following [the testing strategy of data-processing-lib](../../../../data-processing-lib/doc/transform-testing.md)
+
+Currently we have:
+- [Unit test](test/test_doc_chunk_python.py)
+
+
+## Further Resource
+
+- For the [Docling Core](https://github.com/DS4SD/docling-core) `HierarchicalChunker`
+  - <https://ds4sd.github.io/docling/>
+- For the Markdown chunker in LlamaIndex
+  - [Markdown chunking](https://docs.llamaindex.ai/en/stable/module_guides/loading/node_parsers/modules/#markdownnodeparser)
+- For the Token Text Splitter in LlamaIndex
+  - [Token Text Splitter](https://docs.llamaindex.ai/en/stable/api_reference/node_parsers/token_text_splitter/)

transforms/language/pdf2parquet/README.md

@@ -1,10 +1,10 @@
-# PDF2PARQUET Transform 
+# Pdf2Parquet Transform 
 
 
-The PDF2PARQUET transforms iterate through PDF files or zip of PDF files and generates parquet files
-containing the converted document in Markdown format.
+The Pdf2Parquet transforms iterate through PDF, Docx, Pptx, Images files or zip of files and generates parquet files
+containing the converted document in Markdown or JSON format.
 
-The PDF conversion is using the [Docling package](https://github.com/DS4SD/docling).
+The conversion is using the [Docling package](https://github.com/DS4SD/docling).
 
 The following runtimes are available:
 

transforms/language/pdf2parquet/python/README.md

@@ -1,4 +1,15 @@
-# Ingest PDF to Parquet
+# Ingest PDF to Parquet Transform
+
+Please see the set of
+[transform project conventions](../../../README.md#transform-project-conventions)
+for details on general project conventions, transform configuration,
+testing and IDE set up.
+
+## Contributors
+
+- Michele Dolfi ([email protected])
+
+## Description 
 
 This tranforms iterate through document files or zip of files and generates parquet files
 containing the converted document in Markdown or JSON format.
@@ -7,6 +18,9 @@ The PDF conversion is using the [Docling package](https://github.com/DS4SD/docli
 The Docling configuration in DPK is tuned for best results when running large batch ingestions.
 For more details on the multiple configuration options, please refer to the official [Docling documentation](https://ds4sd.github.io/docling/).
 
+
+### Input files
+
 This transform supports the following input formats:
 
 - PDF documents
@@ -17,37 +31,39 @@ This transform supports the following input formats:
 - Markdown documents
 - ASCII Docs documents
 
+The input documents can be provided in a folder structure, or as a zip archive.
+Please see the configuration section for specifying the input files.
 
-## Output format
 
-The output format will contain all the columns of the metadata CSV file,
-with the addition of the following columns
+### Output format
 
-```jsonc
-{
-    "source_filename": "string",  // the basename of the source archive or file
-    "filename": "string",         // the basename of the PDF file
-    "contents": "string",         // the content of the PDF
-    "document_id": "string",      // the document id, a random uuid4 
-    "document_hash": "string",    // the document hash of the input content 
-    "ext": "string",              // the detected file extension
-    "hash": "string",             // the hash of the `contents` column
-    "size": "string",             // the size of `contents`
-    "date_acquired": "date",      // the date when the transform was executing
-    "num_pages": "number",        // number of pages in the PDF
-    "num_tables": "number",       // number of tables in the PDF
-    "num_doc_elements": "number", // number of document elements in the PDF
-    "pdf_convert_time": "float",  // time taken to convert the document in seconds
-}
-```
+The output table will contain following columns
 
+| output column name | data type | description |
+|-|-|-|
+| source_filename | string | the basename of the source archive or file |
+| filename | string | the basename of the PDF file |
+| contents | string | the content of the PDF |
+| document_id | string | the document id, a random uuid4  |
+| document_hash | string | the document hash of the input content |
+| ext | string | the detected file extension |
+| hash | string | the hash of the `contents` column |
+| size | string | the size of `contents` |
+| date_acquired | date | the date when the transform was executing |
+| num_pages | number | number of pages in the PDF |
+| num_tables | number | number of tables in the PDF |
+| num_doc_elements | number | number of document elements in the PDF |
+| pdf_convert_time | float | time taken to convert the document in seconds |
 
-## Parameters
+
+
+## Configuration
 
 The transform can be initialized with the following parameters.
 
 | Parameter  | Default  | Description  |
 |------------|----------|--------------|
+| `data_files_to_use`          | - | The files extensions to be considered when running the transform. Example value `['.pdf','.docx','.pptx','.zip']`. For all the supported input formats, see the section above. |
 | `batch_size`                 | -1 | Number of documents to be saved in the same result table. A value of -1 will generate one result file for each input file. |
 | `artifacts_path`             | <unset> | Path where to Docling models artifacts are located, if unset they will be downloaded and fetched from the [HF_HUB_CACHE](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache) folder. |
 | `contents_type`         | `text/markdown`        | The output type for the `contents` column. Valid types are `text/markdown`, `text/plain` and `application/json`. |
@@ -58,9 +74,68 @@ The transform can be initialized with the following parameters.
 | `pdf_backend`                | `dlparse_v2`  | The PDF backend to use. Valid values are `dlparse_v2`, `dlparse_v1`, `pypdfium2`. |
 | `double_precision`           | `8`           | If set, all floating points (e.g. bounding boxes) are rounded to this precision. For tests it is advised to use 0. |
 
+
+Example
+
+```py
+{
+    "data_files_to_use": ast.literal_eval("['.pdf','.docx','.pptx','.zip']"),
+    "contents_type": "application/json",
+    "do_ocr": True,
+}
+```
+
+## Usage
+
+### Launched Command Line Options 
+
 When invoking the CLI, the parameters must be set as `--pdf2parquet_<name>`, e.g. `--pdf2parquet_do_ocr=true`.
 
 
+### Running the samples
+To run the samples, use the following `make` targets
+
+* `run-cli-sample` - runs src/pdf2parquet_transform_python.py using command line args
+* `run-local-sample` - runs src/pdf2parquet_local.py
+* `run-local-python-sample` - runs src/pdf2parquet_local_python.py
+
+These targets will activate the virtual environment and set up any configuration needed.
+Use the `-n` option of `make` to see the detail of what is done to run the sample.
+
+For example, 
+```shell
+make run-local-python-sample
+...
+```
+Then 
+```shell
+ls output
+```
+To see results of the transform.
+
+
+### Code example
+
+TBD (link to the notebook will be provided)
+
+See the sample script [src/pdf2parquet_local_python.py](src/pdf2parquet_local_python.py).
+
+
+### Transforming data using the transform image
+
+To use the transform image to transform your data, please refer to the 
+[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
+substituting the name of this transform image and runtime as appropriate.
+
+## Testing
+
+Following [the testing strategy of data-processing-lib](../../../../data-processing-lib/doc/transform-testing.md)
+
+Currently we have:
+- [Unit test](transforms/language/pdf2parquet/python/test/test_pdf2parquet_python.py)
+- [Integration test](transforms/language/pdf2parquet/python/test/test_pdf2parquet.py)
+
+
 ## Credits
 
 The PDF document conversion is developed by the AI for Knowledge group in IBM Research Zurich.

transforms/language/pdf2parquet/ray/README.md

@@ -1,7 +1,55 @@
-# PDF2PARQUET Ray Transform 
+# Ingest PDF to Parquet Ray Transform 
+Please see the set of
+[transform project conventions](../../../README.md#transform-project-conventions)
+for details on general project conventions, transform configuration,
+testing and IDE set up.
 
 This module implements the ray version of the [pdf2parquet transform](../python/).
 
+## Summary 
+This project wraps the [Ingest PDF to Parquet transform](../python) with a Ray runtime.
+
+## Configuration and command line Options
+
+Ingest PDF to Parquet configuration and command line options are the same as for the base python transform. 
+
+## Running
+
+### Launched Command Line Options 
+When running the transform with the Ray launcher (i.e. TransformLauncher),
+In addition to those available to the transform as defined in [here](../python/README.md),
+the set of 
+[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.
+
+### Running the samples
+To run the samples, use the following `make` targets
+
+* `run-cli-sample` - runs src/pdf2parquet_transform_ray.py using command line args
+* `run-local-sample` - runs src/pdf2parquet_local_ray.py
+* `run-s3-sample` - runs src/pdf2parquet_s3_ray.py
+    * Requires prior invocation of `make minio-start` to load data into local minio for S3 access.
+
+These targets will activate the virtual environment and set up any configuration needed.
+Use the `-n` option of `make` to see the detail of what is done to run the sample.
+
+For example, 
+```shell
+make run-cli-sample
+...
+```
+Then 
+```shell
+ls output
+```
+To see results of the transform.
+
+
+### Transforming data using the transform image
+
+To use the transform image to transform your data, please refer to the 
+[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
+substituting the name of this transform image and runtime as appropriate.
+
 
 ## Prometheus metrics
 

transforms/language/text_encoder/python/README.md

@@ -1,14 +1,36 @@
 # Text Encoder Transform 
 
-## Summary 
+Please see the set of
+[transform project conventions](../../../README.md#transform-project-conventions)
+for details on general project conventions, transform configuration,
+testing and IDE set up.
+
+## Contributors
+
+- Michele Dolfi ([email protected])
+
+## Description 
+
 This transform is using [sentence encoder models](https://en.wikipedia.org/wiki/Sentence_embedding) to create embedding vectors of the text in each row of the input .parquet table.
 
 The embeddings vectors generated by the transform are useful for tasks like sentence similarity, features extraction, etc which are also at the core of retrieval-augmented generation (RAG) applications.
 
+### Input 
+
+| input column name | data type | description |
+|-|-|-|
+| the one specified in _content_column_name_ configuration | string | the content used in this transform |
+
+
+### Output columns
+
+
+| output column name | data type | description |
+|-|-|-|
+| the one specified in _output_embeddings_column_name_ configuration | `array[float]` | the embeddings vectors of the content |
 
-## Running
 
-### Parameters
+## Configuration
 
 The transform can be tuned with the following parameters.
 
@@ -18,7 +40,11 @@ The transform can be tuned with the following parameters.
 | `model_name`                    | `BAAI/bge-small-en-v1.5` | The HF model to use for encoding the text. |
 | `content_column_name`           | `contents` | Name of the column containing the text to be encoded. |
 | `output_embeddings_column_name` | `embeddings` | Column name to store the embeddings in the output table. |
-| `output_path_column_name`       | `doc_path` | Column name to store the document path of the chunk in the output table. |
+
+
+## Usage
+
+### Launched Command Line Options 
 
 When invoking the CLI, the parameters must be set as `--text_encoder_<name>`, e.g. `--text_encoder_column_name_key=myoutput`.
 
@@ -43,8 +69,20 @@ ls output
 ```
 To see results of the transform.
 
+### Code example
+
+TBD (link to the notebook will be provided)
+
+
 ### Transforming data using the transform image
 
 To use the transform image to transform your data, please refer to the 
 [running images quickstart](../../../../doc/quick-start/run-transform-image.md),
 substituting the name of this transform image and runtime as appropriate.
+
+## Testing
+
+Following [the testing strategy of data-processing-lib](../../../../data-processing-lib/doc/transform-testing.md)
+
+Currently we have:
+- [Unit test](test/test_text_encoder_python.py)