Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Simplifying UDF definition #1395

Open
wants to merge 9 commits into
base: staging
Choose a base branch
from

Conversation

Prakhar314
Copy link

@Prakhar314 Prakhar314 commented Nov 25, 2023

Problem

  • Currently declaring UDFs is not simple, requiring a lot of boilerplate code, and obtuse input and output data types (example)
  • Snowflake has a simple interface for UDFs (example), we could use.

Solution

  • Modify load_function_class_from_file to handle regular python functions.
  • Now, if the required instance is a Python function instead of a class extending AbstractFunction, we create a UserDefinedFunction, which is an instance of AbstractFunction, to wrap around the given Python function.
  • Changes in several other files to distinguish between a Python class and a Python instance, from the output of the modified load_function_class_from_file.

Integration Test

See the new integration test for an example usecase.
We load UDFs from a file with the following content:

def mod5(id:int)->int:
    return id%5
def isEven(id:int)->bool:
    return id%2==0

There is no change in the EvaQL interface. This means that the following queries work:

CREATE FUNCTION mod5 IMPL '<file_location>';
CREATE FUNCTION isEven IMPL '<file_location>';
SELECT mod5(Id) from ....

Todo

  • Creating a UserDefinedFunction wrapper is a hack to make the minimum functionality work within the current implementation. We need to explore other options like creating a separate executor for simple functions.
  • We assume that the functions take one row of the input at a time.
  • This implementation is probably not as expressive as extending the AbstractFunction class.

@Prakhar314 Prakhar314 marked this pull request as draft November 25, 2023 06:54
@Prakhar314 Prakhar314 changed the title [Draft] Simplifying UDF definition feat: Simplifying UDF definition Nov 25, 2023
@Prakhar314 Prakhar314 marked this pull request as ready for review December 1, 2023 21:21
@Prakhar314 Prakhar314 marked this pull request as draft December 5, 2023 15:56
@Prakhar314 Prakhar314 marked this pull request as ready for review January 5, 2024 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant