EVA UDF Registration

[LordDarkula]

EVA UDF Registration

Motivation

Current Approach

Currently, users must create UDFs by doing the following.

1. Create a python file inside eva/udfs
2. Create a class and inherit from AbstractClassifierUDF, PytorchAbstractClassifierUDF, or AbstractNdarrayUDF depending on the type the UDF will operate on
3. Implement some combination of __call__(), transform(), and exec() depending on the class that was inherited from
4. Implement other methods such as name(), labels(), get_predictions(), etc
5. Execute a SQL query in EVA to register the UDF

CREATE UDF IF NOT EXISTS  Array_Count
	INPUT (Input_Array NDARRAY ANYTYPE, Search_Key ANYTYPE)
    OUTPUT (key_count INTEGER)
	TYPE NdarrayUDF
	IMPL "{}/udfs/ndarray_udfs/array_count.py";

Problems

There are a couple problems with the current approach.

• The user has to know how inheritance works
  • Many data scientists are unfamiliar with programming and computer science design principles
• Different types have different class signatures, and inconsistency that leads to confusion
• Mandating the implementation of many methods makes our system less flexible and requires the user to write large custom classes just to interface
• Mandating the use of a SQL query for UDF registration is error-prone

Proposed Design

Many major platforms such as SnorkelAI and BentoML use decorators and that is the approach taken here. The user will write a series of functions and decorate them with metadata about the UDF.
The decorators needed to specify a UDF are setup, preprocess, and forward. Preprocess is optional and can be annotated separately from forward to help EVA optimize queries.

Input and Output Specification

Both preprocess() and forward() take an input and produce an output. EVA needs to know the type and format of these inputs and outputs.

• Input Types
  • Tensor
  • PIL
  • NdArray
• Output Types
  • List[Tuple[Label, Bbox]]
  • Represents a classification result for a specific frame
  • list of hits in a frame

Parameters

In order to add a UDF to EVA, EVA needs to know some information about the UDF to verify queries and execute them efficiently.

• Setup
  • cache: bool
    • Whether or not to cache previously computed results
  • requirements: Optional[List[str] | Path]
    • List of strings specifying pip dependencies or path to requirements.txt file
    • EVA will automatically install these dependencies
  • external_resources: Optional[List[Path]]
    • Paths to external files required for model operation
      • Examples include model files and data
    • EVA will copy these resources into the same directory as the UDF
• Preprocessing (optional)
  • input_type: type
  • input_frame_shape: Tuple[int, int]
• Forward
  • input_type: type
  • input_frame_shape: Tuple[int, int]
  • verify_output: bool
  • batch: bool
• Set UDF type
  • create_udf
    • exists_ok: bool
    • labels: List[str]

Preprocessing

Preprocessing is an optional step the user can define. If it is not defined, the query engine will feed frames into the forward function directly. The output signature of preprocess must match the input signature of forward.

Examples

import eva
from eva.typing import Ndarray, PdDataframe, TorchTensor, PILImage
import torchvision

udf = eva.UDF()

@udf.setup(cache=True, 
		   requirements=["tqdm"],
		   external_resources=["path/to/model"])
def setup(ctx):
	model = torchvision.models.detection.fasterrcnn_resnet50_fpn(
	    pretrained=True, progress=False
    )
    model.eval()
	ctx["model"] = model

@udf.preprocess(input_type=PILImage, 
				input_frame_shape=(3, -1, -1),
				batch=True)
def preprocess(ctx):
	pass

@udf.forward(input_type=TorchTensor,
			input_frame_shape=(3, -1, -1),
			output_type=PdDataframe)
def forward(ctx, frames):
	pass

udf.mode_classify(labels=["cat", "dog"], 
			 output_format=["bbox", "label"])
eva.create_udf(udf, exists_ok=True)

Implementation

EVA is split into two parts: eva_server and eva_client. EVA registration will be client-side. The API will be packaged as a python package. The create_udf() function will send the annotated functions and the associated parameters to the server. The code will internally be converted into a class and saved as a python file in udfs/{udf-name}. The parameters will be stored as a json file.

[xzdandy]

I like the idea. How do you plan to send the annotated functions and the associated parameters (e.g., external_resources) to the server?