Skip to content

Commit

Permalink
README changes
Browse files Browse the repository at this point in the history
Signed-off-by: SHAHROKH DAIJAVAD <[email protected]>
  • Loading branch information
shahrokhDaijavad committed Dec 10, 2024
1 parent ae79ab5 commit ca3efdb
Showing 1 changed file with 13 additions and 57 deletions.
70 changes: 13 additions & 57 deletions transforms/language/text_encoder/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Text Encoder Transform

Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
[transform project conventions](../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

Expand Down Expand Up @@ -48,41 +48,19 @@ The transform can be tuned with the following parameters.

When invoking the CLI, the parameters must be set as `--text_encoder_<name>`, e.g. `--text_encoder_column_name_key=myoutput`.


### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/text_encoder_transform.py using command line args
* `run-local-sample` - runs src/text_encoder_local.py

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
```shell
make run-cli-sample
...
```
Then
```shell
ls output
```
To see results of the transform.

### Code example

TBD (link to the notebook will be provided)

Here is a sample [notebook](text_encoder-python.ipynb)

### Transforming data using the transform image

To use the transform image to transform your data, please refer to the
[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
[running images quickstart](../../../doc/quick-start/run-transform-image.md),
substituting the name of this transform image and runtime as appropriate.

## Testing

Following [the testing strategy of data-processing-lib](../../../../data-processing-lib/doc/transform-testing.md)
Following [the testing strategy of data-processing-lib](../../../data-processing-lib/doc/transform-testing.md)

Currently we have:
- [Unit test](test/test_text_encoder_python.py)
Expand All @@ -91,50 +69,28 @@ Currently we have:

# TextEncoder Ray Transform
Please see the set of
[transform project conventions](../../../README.md#transform-project-conventions)
[transform project conventions](../../README.md#transform-project-conventions)
for details on general project conventions, transform configuration,
testing and IDE set up.

## Summary
This project wraps the [text_encoder transform](../python) with a Ray runtime.
This project wraps the text_encoder transform with a Ray runtime.

## Configuration and command line Options

Text Encoder configuration and command line options are the same as for the base python transform.

## Running
### Code example

Here is a sample [notebook](text_encoder-ray.ipynb)

### Launched Command Line Options
In addition to those available to the transform as defined in [here](../python/README.md),
the set of
[ray launcher](../../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Running the samples
To run the samples, use the following `make` targets

* `run-cli-sample` - runs src/text_encoder_transform.py using command line args
* `run-local-sample` - runs src/text_encoder_local_ray.py
* `run-s3-sample` - runs src/text_encoder_s3_ray.py
* Requires prior installation of minio, depending on your platform (e.g., from [here](https://min.io/docs/minio/macos/index.html)
and [here](https://min.io/docs/minio/linux/index.html)
and invocation of `make minio-start` to load data into local minio for S3 access.

These targets will activate the virtual environment and set up any configuration needed.
Use the `-n` option of `make` to see the detail of what is done to run the sample.

For example,
```shell
make run-cli-sample
...
```
Then
```shell
ls output
```
To see results of the transform.

In addition to those available to the transform as defined here,
[ray launcher options](../../../data-processing-lib/doc/ray-launcher-options.md) are available.

### Transforming data using the transform image

To use the transform image to transform your data, please refer to the
[running images quickstart](../../../../doc/quick-start/run-transform-image.md),
[running images quickstart](../../../doc/quick-start/run-transform-image.md),
substituting the name of this transform image and runtime as appropriate.

0 comments on commit ca3efdb

Please sign in to comment.