Skip to content

Sorted dict #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
amascolo wants to merge 80 commits into
base: main
Choose a base branch
from
Draft

Sorted dict #14

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
0636c9c
add sorted dict data structure
amiralikaboli Oct 13, 2024
2085c9b
Fixed clang typename warnings
amascolo Oct 13, 2024
cb35dde
use vector of pairs rather than pair of vectors in SortedDict
amiralikaboli Oct 14, 2024
bbd7d1b
Clion suggestions
amascolo Oct 14, 2024
7125d59
implement at as the logn insert in SortedDict
amiralikaboli Oct 14, 2024
382ef0f
Renamed MAX_STRING -> STRING_MAX in codegen too
amascolo Oct 23, 2024
0615975
Fixed comment
amascolo Oct 24, 2024
c4895c9
Integrated SortedDict in parser
amascolo Oct 24, 2024
509b50f
Integrated Range in parser
amascolo Oct 24, 2024
b5ef787
SortedDict::contains
amascolo Oct 24, 2024
4323d0b
Correct accessors for SortedDict/Range
amascolo Oct 24, 2024
a5d76f8
Sorted indices external function
amascolo Oct 24, 2024
729e803
Range iterator and better error message
amascolo Oct 24, 2024
5d4f1e4
Special case for vec
amascolo Oct 24, 2024
52ecd8a
Sort C++ utilities
amascolo Oct 24, 2024
e5528be
Added support for sort_vec external function
amascolo Oct 24, 2024
69f0677
FJ 3a sorting (almost works)
amascolo Oct 24, 2024
a8ac145
Cleanup
amascolo Oct 25, 2024
5c5fa56
Handled SortedDict read access
amascolo Oct 25, 2024
f6e9eb7
Sorting outside main
amascolo Oct 25, 2024
ab36c13
Use emplace_back
amascolo Oct 25, 2024
e62713f
Benchmark timer scope now determined by SDQL program
amascolo Oct 25, 2024
7929206
Sorted indices external function no longer a special case
amascolo Oct 25, 2024
8b5bc97
Timer no parentheses
amascolo Oct 25, 2024
d7b821d
Load no long treated as special case
amascolo Oct 26, 2024
96cc6c8
Added timer to SDQL programs
amascolo Oct 26, 2024
466b4ed
Added SortedDict cache, so we can call 'at' after 'contains' without …
amascolo Oct 27, 2024
70b8fc7
Fixed sorting performance
amascolo Oct 27, 2024
2322382
Removed unnecessary inline for class methods (clang-tidy)
amascolo Oct 27, 2024
085bb54
Removed TODOs
amascolo Oct 27, 2024
a4369da
Renamed 'SortVec' -> 'SortedVec' for consistency
amascolo Oct 27, 2024
daf3354
Typo
amascolo Oct 27, 2024
a3fa391
Auto-generated sorting 3a & 10a
amascolo Oct 28, 2024
f3d31f1
Removed leftover TODO comment
amascolo Nov 6, 2024
840121b
Added timer to LSQB queries
amascolo Nov 6, 2024
7a9c5cf
Removed TODO for fancier LSQB NaN handling and added comment explaini…
amascolo Nov 6, 2024
4234ba4
Smallvecdict: const fix + equality with bag semantics
amascolo Nov 10, 2024
3eab560
Helper functions for debugging
amascolo Nov 10, 2024
2ebac76
More debugging utilities
amascolo Nov 11, 2024
bc77d78
Arbitrary levels of nesting
amascolo Nov 11, 2024
1cdd150
Simplified + added case
amascolo Nov 11, 2024
600b63d
Codegen: added missing case for iterating over SortedDict
amascolo Nov 11, 2024
51adf34
Codegen: added const qualifiers + fixed newline
amascolo Nov 12, 2024
b125be2
SortedDict == comparison
amascolo Nov 12, 2024
4ad1e37
Runtime: variadic sorted indices
amascolo Nov 12, 2024
f203085
Codegen: variadic sorted indices
amascolo Nov 12, 2024
b55451c
3a GJ hybrid
amascolo Nov 12, 2024
ca1571d
13a GJ hybrid
amascolo Nov 12, 2024
8249ef8
Renamed queries
amascolo Nov 19, 2024
bb01077
13b GJ hybrid + fixed codegen case
amascolo Nov 19, 2024
25c6083
Reorganised sorting queries and added tests
amascolo Nov 19, 2024
4e182a8
Minor fix to sorting in JOB FJ pure 3a, 10a (doesn't impact results)
amascolo Nov 21, 2024
98985fa
Added FJ pure 8a, 12b, 17b, 17f
amascolo Nov 21, 2024
dfe8b4e
Added unit tests for JOB FJ pure new queries
amascolo Nov 21, 2024
60da229
Added codegen unit tests
amascolo Nov 22, 2024
485a4f7
JOB FJ hybrid 3a
amascolo Nov 22, 2024
d35f1c7
JOB FJ hybrid 8a
amascolo Nov 22, 2024
b103fdc
JOB FJ hybrid 12b
amascolo Nov 22, 2024
96e9896
JOB FJ hybrid 17b
amascolo Nov 22, 2024
a8a0502
JOB FJ hybrid 17b
amascolo Nov 22, 2024
c2f3956
FJ pure sorting (all queries)
amascolo Nov 23, 2024
00cc5c8
FJ hybrid sorting (all queries)
amascolo Nov 24, 2024
aa3c8d5
GJ hybrid sorting (all queries)
amascolo Nov 24, 2024
c4dd4d5
Reorganised progs directory + tests
amascolo Nov 24, 2024
312feb4
Small edit to GJ optimised queries
amascolo Nov 25, 2024
fb88faa
GJ hybrid optimised
amascolo Nov 25, 2024
b21b449
Revised queries (new plans for JOB FJ)
amascolo Nov 27, 2024
eb382e1
Bumped munit version
amascolo Nov 27, 2024
0a25db4
LSQB FJ hybrid
amascolo Nov 27, 2024
afaf9e1
LSQB GJ hybrid
amascolo Nov 27, 2024
b9ce440
Added unit tests
amascolo Nov 27, 2024
3cf4224
Updated revised path (for consistency)
amascolo Nov 27, 2024
bac5a36
Performance fix (affected e.g. FJ sorting 18a)
amascolo Nov 28, 2024
cafbf51
Delete old leftover queries from FJ sorting
amascolo Nov 28, 2024
e5a2957
Renamed for consistency
amascolo Nov 28, 2024
225e736
Json free join query plans
amascolo Nov 29, 2024
818a57f
Renamed arg
amascolo Nov 29, 2024
3edcd0c
Moved out plan from path
amascolo Jan 6, 2025
829692a
Pretty print SDQL expressions as source code
amascolo Jan 7, 2025
23b295d
Commented out test
amascolo Jan 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
15 changes: 11 additions & 4 deletions build.sbt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
scalaVersion := "2.13.14"

libraryDependencies ++= Seq(
"com.lihaoyi" %% "fastparse" % "3.1.1",
"junit" % "junit-dep" % "4.11" % "test",
"org.scalatest" % "scalatest_2.13" % "3.2.19" % "test",
"org.scalameta" %% "munit" % "1.0.0" // for pretty printing expressions
"com.lihaoyi" %% "fastparse" % "3.1.1",
"junit" % "junit-dep" % "4.11" % "test",
"org.scalatest" % "scalatest_2.13" % "3.2.19" % "test",
"io.circe" %% "circe-core" % "0.14.10",
"io.circe" %% "circe-generic" % "0.14.10",
"io.circe" %% "circe-parser" % "0.14.10",
"io.circe" %% "circe-generic-extras" % "0.14.4" // follows a separate versioning
)

// Note: IntelliJ accepts -P but SBT requires no. of threads
Expand All @@ -17,6 +20,10 @@ Test / testOptions ++= Seq(
// require converting from Parquet to CSV the JOB datasets of https://github.com/SIGMOD23p561/free-join
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBGJ"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBFJ"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBFJSorting"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBGJHybrid"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBFJHybrid"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestJOBFJRevised"),
// require converting from Parquet to CSV the LSQB datasets of https://github.com/remysucre/gj-vs-binary
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestLSQBGJ0_1"),
Tests.Argument(TestFrameworks.ScalaTest, "-l", "TestLSQBFJ0_1"),
Expand Down
226 changes: 226 additions & 0 deletions plans/job/fj/10a.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
{
"opt_plan": [
{
"node": {
"node": "interm0",
"keys": {
"keys": [
[
0,
{
"relation": "mc",
"attribute": "movie_id"
}
],
[
1,
{
"relation": "mc",
"attribute": "company_type_id"
}
],
[
2,
{
"relation": "mc",
"attribute": "company_id"
}
],
[
3,
{
"relation": "t",
"attribute": "title"
}
]
]
},
"record": [
0
]
},
"build_plan": {
"build_plan": [
[
"mc",
[
[
"movie_id",
"company_type_id",
"company_id"
],
[]
]
],
[
"t",
[
[
"id"
],
[
"title"
]
]
],
[
"ct",
[
[
"id"
],
[]
]
],
[
"cn",
[
[
"id"
],
[]
]
]
]
},
"compiled_plan": {
"compiled_plan": [
[
{
"relation": "mc",
"attribute": "company_type_id"
},
{
"relation": "ct",
"attribute": "id"
}
],
[
{
"relation": "mc",
"attribute": "company_id"
},
{
"relation": "cn",
"attribute": "id"
}
],
[
{
"relation": "mc",
"attribute": "movie_id"
},
{
"relation": "t",
"attribute": "id"
}
]
]
}
},
{
"node": {
"node": "root",
"keys": {
"keys": [
[
5,
{
"relation": "interm0",
"attribute": "col3"
}
],
[
6,
{
"relation": "chn",
"attribute": "name"
}
]
]
},
"record": []
},
"build_plan": {
"build_plan": [
[
"ci",
[
[
"role_id",
"movie_id",
"person_role_id"
],
[]
]
],
[
"rt",
[
[
"id"
],
[]
]
],
[
"interm0",
[
[
"col0"
],
[
"col3"
]
]
],
[
"chn",
[
[
"id"
],
[
"name"
]
]
]
]
},
"compiled_plan": {
"compiled_plan": [
[
{
"relation": "ci",
"attribute": "role_id"
},
{
"relation": "rt",
"attribute": "id"
}
],
[
{
"relation": "ci",
"attribute": "movie_id"
},
{
"relation": "interm0",
"attribute": "col0"
}
],
[
{
"relation": "ci",
"attribute": "person_role_id"
},
{
"relation": "chn",
"attribute": "id"
}
]
]
}
}
]
}
Loading