-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fast serialization of sql records #812
base: master
Are you sure you want to change the base?
Conversation
@lalithsuresh : please take a look at this API. Note that using the DDlogAPI helpers requires the ddlog dynamic library to be already loaded, and this can happen only once. So the order of the tests is important. |
Thanks @mbudiu-vmw. I'll give this a spin. |
@lalithsuresh, @mbudiu-vmw , shall we close this PR for now? |
I expect this is still useful |
If you're ok maintaining this code, we can merge it. |
Let's first wait for Lalith's comments. Then, if he indeed is using it, it has to be generalized to handle more types. |
If @lalithsuresh is not using this we can perhaps keep it as a draft PR for a while until we figure out whether it will be ever used. |
I think we should keep it around as a draft PR for now. We'll get more data when I try out #817 in DCM (larger tables, each with more columns etc). |
This commit introduces a more efficient way to move data from/to Java when using "SqlRecords" - these are structs with possibly nullable base types as fields. This is implemented by serializing the objects manually into byte[] and transferring the byte[] in a single JNI call, then deserializing it into a DDlogRecord. The reverse path is also supported.
This is a subset of what the flatbufs offers, but the flatbufs do not generate APIs to manipulate records, but rather collections.
Currently the SqlRecord lives in the src project, while the serialization support is in DDlogAPI. This is arguably wrong, we should choose one of the two locations. But this PR is here mostly to solicit an initial review.
We also need to support more base types and probably Tuples as well (needed for primary-key based operation).
The normal way to use the SqlRecord is shown in the test
createREcordFromRelation
: