Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure Pdf2Parquet with its own dpk_ namespace #813

Merged
merged 15 commits into from
Dec 17, 2024

Conversation

touma-I
Copy link
Collaborator

@touma-I touma-I commented Nov 19, 2024

Why are these changes needed?

This is a first of a series of restructuring changes that are done to have each transform built as its own module (e.g. dpk_pdf2parquet) with a ray submodule (dpk_pdf2parquet.ray ).

Removed python and ray folders and keep Dockerfile.python and Dockerfile.ray
remove pyproject.toml and Makefiles
move python code under dpk_pdf2parquet
move ray code under dpk_pdf2parquet/ray
change import statement to include module name
replace recursive Makefile and use targets from .make.cicd.targets
adapt kfp_ray/Makefile and other make target

Related issue number (if any).

#774

@touma-I touma-I requested a review from dolfim-ibm November 19, 2024 12:50
@touma-I touma-I changed the title first cut at refactoring pdf2parquet first cut at refactoring dpk_pdf2parquet Nov 19, 2024
Signed-off-by: Maroun Touma <[email protected]>
Copy link
Collaborator

@roytman roytman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@touma-I thank you for your work.

Signed-off-by: Maroun Touma <[email protected]>
@touma-I touma-I changed the title first cut at refactoring dpk_pdf2parquet Restructure Pdf2Parquet with its own dpk_ namespace Nov 24, 2024
@touma-I touma-I requested a review from Param-S November 26, 2024 02:10
touma-I and others added 3 commits December 4, 2024 08:30
Signed-off-by: Maroun Touma <[email protected]>
Signed-off-by: Maroun Touma <[email protected]>
Signed-off-by: SHAHROKH DAIJAVAD <[email protected]>
@shahrokhDaijavad
Copy link
Collaborator

shahrokhDaijavad commented Dec 9, 2024

@touma-I Please check what I have done before I make changes to the others. You seem to have only the run-cli-sample as a target for make, so I got rid of everything else that had to do with make other targets.
Do we need to provide an example of how to run the ray code?

@touma-I touma-I merged commit 664082c into data-prep-kit:dev Dec 17, 2024
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants