merged with dev

Signed-off-by: Maroun Touma <[email protected]>

.gitignore

@@ -146,3 +146,7 @@ dmypy.json
 
 # Ignore artifacts directory created by the `make save-images` target
 artifacts/
+
+*.lock
+
+*.db

README.md

@@ -79,8 +79,8 @@ conda install gxx_linux-64
 Next, install the data prep toolkit library. This library installs both the python and ray versions of the transforms. For better management of dependencies, it is recommended to install the same tagged version of both the library and the transform. 
 
 ```bash
-pip3 install  'data-prep-toolkit[ray]==0.2.2.dev1'
-pip3 install  'data-prep-toolkit-transforms[ray,all]==0.2.2.dev1'
+pip3 install  'data-prep-toolkit[ray]==0.2.3.dev0'
+pip3 install  'data-prep-toolkit-transforms[ray,all]==0.2.3.dev1'
 pip3 install jupyterlab   ipykernel  ipywidgets
 
 ## install custom kernel
@@ -144,7 +144,7 @@ The matrix below shows the the combination of modules and supported runtimes. Al
 | [Web to Parquet](transforms/universal/web2parquet/README.md)                         | :white_check_mark: |                    |                    |                    |         
 | **Universal (Code & Language)**                                                      |                    |                    |                    |                    | 
 | [Exact dedup filter](transforms/universal/ededup/ray/README.md)                      | :white_check_mark: | :white_check_mark: |                    | :white_check_mark: |
-| [Fuzzy dedup filter](transforms/universal/fdedup/ray/README.md)                      |                    | :white_check_mark: |                    | :white_check_mark: |
+| [Fuzzy dedup filter](transforms/universal/fdedup/ray/README.md)                      | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
 | [Unique ID annotation](transforms/universal/doc_id/ray/README.md)                    | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
 | [Filter on annotations](transforms/universal/filter/python/README.md)                | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |
 | [Profiler](transforms/universal/profiler/ray/README.md)                              | :white_check_mark: | :white_check_mark: | :white_check_mark: | :white_check_mark: |

data-processing-lib/ray/src/data_processing_ray/runtime/ray/ray_utils.py

@@ -15,13 +15,19 @@
 from typing import Any
 
 import ray
-from ray.experimental.state.api import list_actors
 from data_processing.utils import GB, UnrecoverableException
 from ray.actor import ActorHandle
 from ray.exceptions import RayError
+from ray.experimental.state.api import list_actors
 from ray.util.actor_pool import ActorPool
 
 
+# This value matches the constant `RAY_MAX_LIMIT_FROM_API_SERVER` defined in the ray source code here:
+# https://github.com/ray-project/ray/blob/569f7df9067c5654fb57ba7bc4792b3ba5aaa846/python/ray/util/state/common.py#L50-L53
+
+RAY_MAX_ACTOR_LIMIT = 10000
+
+
 class RayUtils:
     """
     Class implementing support methods for Ray execution
@@ -109,11 +115,13 @@ def operator() -> ActorHandle:
             time.sleep(creation_delay)
             return clazz.options(**actor_options).remote(params)
 
-        cls_name = clazz.__class__.__name__.replace('ActorClass(', '').replace(')','')
+        cls_name = clazz.__class__.__name__.replace("ActorClass(", "").replace(")", "")
         actors = [operator() for _ in range(n_actors)]
         for i in range(120):
             time.sleep(1)
-            alive = list_actors(filters=[("class_name", "=", cls_name), ("state", "=", "ALIVE")])
+            alive = list_actors(
+                filters=[("class_name", "=", cls_name), ("state", "=", "ALIVE")], limit=RAY_MAX_ACTOR_LIMIT
+            )
             if len(actors) == len(alive):
                 return actors
         # failed - raise an exception