Skip to content

Commit 4fc5f34

Browse files
JolanrensenJolanrensen
authored
Merge pull request #900 from Kotlin/readmes
READMEs
2 parents 9b88fa5 + 4d96ef1 commit 4fc5f34

File tree

17 files changed

+275
-1
lines changed

17 files changed

+275
-1
lines changed

‎.github/README.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
## GitHub Actions
2+
3+
While publishing and testing of the library is handled by [JetBrains TeamCity](https://www.jetbrains.com/teamcity/),
4+
there are some CI operations that are handled by GitHub actions instead.
5+
6+
### Publishing Docs
7+
8+
The building of the documentation website in [docs](../docs), and the publishing of it along
9+
with the search-indices is handled by the [Build Docs GH Action](./workflows/main.yml). Careful: This action replaces the entire contents
10+
of the documentation website.
11+
12+
### Gradle Wrapper Validation
13+
14+
We run the [Gradle Wrapper Validation action](https://github.com/gradle/wrapper-validation-action)
15+
using a [GitHub Action](./workflows/gradle-wrapper-validation.yml) as well.
16+
This action validates the checksums of all Gradle Wrapper JAR files present in the repository and
17+
fails if any unknown Gradle Wrapper JAR files are found.
18+
19+
### Auto-commit generated code
20+
21+
Anytime the source code changes on [master](https://github.com/Kotlin/dataframe/tree/master),
22+
this [GitHub Action](./workflows/generated-sources-master.yml) makes sure
23+
[`processKDocsMain`](../KDOC_PREPROCESSING.md),
24+
and `korro` are run. If there have been any changes in either [core/generated-sources](../core/generated-sources) or
25+
[docs/StardustDocs/snippets](../docs/StardustDocs/snippets), these are auto-committed to the branch, to keep
26+
it up to date.
27+
28+
### Show generated code in PR
29+
30+
To make sure no unexpected code is auto-committed to [master](https://github.com/Kotlin/dataframe/tree/master),
31+
this [GitHub Action](./workflows/generated-sources.yml) runs the same code-generating tasks but on a separate branch.
32+
If there are changes, it will leave a message in the PR, informing you about the changes that will be done to the master
33+
branch if this PR were merged.

‎core/README.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
## :core
2+
3+
This is the core of the library, published as the `dataframe-core` package.
4+
It contains the DataFrame API and its implementation, as well as plenty of JUnit tests.
5+
6+
I/O operations are split off into other modules, like [:dataframe-excel](../dataframe-excel),
7+
or [:dataframe-jdbc](../dataframe-jdbc), however, this is has not happened yet for all operations
8+
(see [Issue #100](https://github.com/Kotlin/dataframe/issues/100)).
9+
10+
At the moment, these integrations are still part of the `:core` module:
11+
12+
- csv/tsv
13+
- html
14+
- json
15+
- jupyter
16+
17+
### Korro code samples
18+
19+
Code samples for the documentation website reside
20+
in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples that depend solely on `dataframe-core`),
21+
and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
22+
and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
23+
by [Korro](https://github.com/devcrocod/korro).
24+
25+
### Explainer dataframes
26+
27+
Aside from code samples, `@TransformDataFrameExpressions` annotated test functions also generate sample
28+
dataframe HTML files that can be used as iFrames on the documentation website.
29+
They are tested, generated, and copied over to [docs/StardustDocs/snippets](../docs/StardustDocs/snippets) by
30+
our "explainer" [plugin callback proxy](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
31+
which hooks into [the TestBase class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
32+
retrieves the intermediate DataFrame expressions thanks to our "explainer" compiler plugin
33+
[:plugins:expressions-converter](../plugins/expressions-converter).
34+
35+
We can also generate "normal" DataFrame samples for the website. This can be done using the
36+
[OtherSamples class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
37+
HTML files will be stored in [docs/StardustDocs/snippets/manual](../docs/StardustDocs/snippets/manual).
38+
39+
### KDoc Preprocesor
40+
41+
The code you're working on needs to be edited in [src](src), but the KDocs are processed by the
42+
[KDoc Preprocessor](https://github.com/Jolanrensen/docProcessorGradlePlugin) when the project is published (or the task
43+
is run manually). The generated sources with adjusted KDocs will be overwritten
44+
in [generated-sources](generated-sources).
45+
See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information.
46+
47+
KDocs can also be exported to HTML, for them to be reused on the website.
48+
Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
49+
[docs/StardustDocs/snippets/kdocs](../docs/StardustDocs/snippets/kdocs).

‎core/build.gradle.kts

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,13 +14,15 @@ plugins {
1414
alias(serialization)
1515
alias(jupyter.api)
1616
alias(korro)
17-
alias(keywordGenerator)
1817
alias(kover)
1918
alias(ktlint)
2019
alias(docProcessor)
2120
alias(simpleGit)
2221
alias(buildconfig)
2322

23+
// generates keywords using the :generator module
24+
alias(keywordGenerator)
25+
2426
// dependence on our own plugin
2527
alias(dataframe)
2628

‎dataframe-arrow/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## :dataframe-arrow
2+
3+
This module, published as `dataframe-arrow`, contains all logic and tests for DataFrame to be able to work with
4+
Apache Arrow.
5+
6+
See [Read Apache Arrow formats](https://kotlin.github.io/dataframe/read.html#read-apache-arrow-formats) and
7+
[Writing to Apache Arrow formats](https://kotlin.github.io/dataframe/write.html#writing-to-apache-arrow-formats)
8+
for more information about how to use it.

‎dataframe-excel/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## :dataframe-excel
2+
3+
This module, published as `dataframe-excel`, contains all logic and tests for DataFrame to be able to work with
4+
`xls` and `xlsx` files.
5+
6+
See [Read from Excel](https://kotlin.github.io/dataframe/read.html#read-from-excel) and
7+
[Write to Excel spreadsheet](https://kotlin.github.io/dataframe/write.html#write-to-excel-spreadsheet)
8+
for more information about how to use it.

‎dataframe-jdbc/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
## :dataframe-jdbc
2+
3+
This module, published as `dataframe-jdbc`, contains all logic and tests for DataFrame to be able to work with
4+
JDBC data sources.
5+
6+
See [Read from SQL databases](https://kotlin.github.io/dataframe/readsqldatabases.html) for more information
7+
about how to use it.

‎dataframe-openapi-generator/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
## :dataframe-openapi-generator
2+
3+
This module, published as `dataframe-openapi-generator` contains all logic and tests for DataFrame to be able to import
4+
OpenAPI specifications as auto-generated data schemas. This module is a sister module to
5+
[`dataframe-openapi`](../dataframe-openapi):
6+
7+
- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate
8+
data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the
9+
`@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function.
10+
- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas.
11+
12+
See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and
13+
[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html)
14+
for more information about how to use it.

‎dataframe-openapi/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
## :dataframe-openapi
2+
3+
This module, published as `dataframe-openapi` contains some functions to be able to use auto-generated
4+
data schemas from OpenAPI specifications. This module is a sister module to
5+
[`dataframe-openapi-generator`](../dataframe-openapi-generator):
6+
7+
- `dataframe-openapi-generator` is used as a dependency of the Gradle plugin and Jupyter plugin to be able to generate
8+
data schemas from OpenAPI specifications. In the Gradle plugin it adds support for the `dataschemas {}` DSL and the
9+
`@file:ImportDataSchema()` annotation. In Jupyter, it adds support for the `importDataSchema()` function.
10+
- `dataframe-openapi` must be used as a dependency of a user-project to be able to use the generated data schemas.
11+
12+
See [Import OpenAPI Schemas in Gradle project](https://kotlin.github.io/dataframe/schemasimportopenapigradle.html) and
13+
[Import Data Schemas, e.g. from OpenAPI, in Jupyter](https://kotlin.github.io/dataframe/schemasimportopenapijupyter.html)
14+
for more information about how to use it.

‎docs/README.md

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
## The Documentation Website
2+
3+
This folder holds the source code of our documentation website:
4+
[kotlin.github.io/dataframe](https://kotlin.github.io/dataframe).
5+
6+
It's built using [Writerside](https://www.jetbrains.com/writerside/) and published
7+
by a [Github Action](../.github/workflows/main.yml).
8+
The file structure largely mirrors the default Writerside structure.
9+
For instance, if you want to add a new page to the website, this needs to be stored as an `.md` file in the
10+
[StardustDocs/topics](./StardustDocs/topics) folder,
11+
and included in the [StardustDocs/d.tree](./StardustDocs/d.tree) file.
12+
13+
Images all README files can be stored in [docs/imgs](./imgs).
14+
15+
### Explainer dataframes
16+
`@TransformDataFrameExpressions` annotated test functions generate sample
17+
dataframe HTML files that can be used as iFrames on the documentation website.
18+
They are tested, generated, and copied over to [docs/StardustDocs/snippets](./StardustDocs/snippets) by
19+
our "explainer" [plugin callback proxy](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
20+
which hooks into [the TestBase class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
21+
retrieves the intermediate DataFrame expressions thanks to
22+
our "explainer" compiler plugin [:plugins:expressions-converter](../plugins/expressions-converter).
23+
24+
We can also generate "normal" DataFrame samples for the website. This can be done using the
25+
[OtherSamples class](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
26+
HTML files will be stored in [docs/StardustDocs/snippets/manual](./StardustDocs/snippets/manual).
27+
28+
### KDoc Preprocessor
29+
KDocs can also be exported to HTML, for them to be reused on the website.
30+
Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
31+
[docs/StardustDocs/snippets/kdocs](./StardustDocs/snippets/kdocs).
32+
33+
### Korro code samples
34+
Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
35+
(for samples that depend solely on `dataframe-core`),
36+
and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
37+
and they are copied over to Markdown files in [docs/StardustDocs/topics](./StardustDocs/topics)
38+
by [Korro](https://github.com/devcrocod/korro).
39+
40+

‎examples/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,5 @@ ___
4444
---
4545
* 0.14.0 Dev Feature Overview [notebook](notebooks/feature_overviews/0.14.0/new_features.ipynb)
4646
Overview of new features available in 0.14.0-dev
47+
48+
These example notebooks are tested by the [:core module](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/jupyter).

‎generator/README.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
## :generator
2+
3+
This module holds a little Gradle plugin whose sole purpose is to provide
4+
[:core](../core) with the `generateKeywordsSrc` task.
5+
6+
This task, generates three enum classes: `HardKeywords`, `ModifierKeywords`, and `SoftKeywords`.
7+
These enums together contain all restricted Kotlin keywords to be taken into account when generating our own
8+
code in Notebooks or any of our [plugins](../plugins). Words like "package", "fun", "suspend", etc...
9+
10+
As the Kotlin language can change over time, this task ensures that any changes to the language
11+
will be reflected in our code generation.
12+
13+
This module will likely be moved under [:plugins](../plugins):
14+
[Issue #899](https://github.com/Kotlin/dataframe/issues/899).

‎plugins/README.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
## :plugins
2+
3+
This folder holds all our Gradle- and Kotlin Compiler plugins:
4+
5+
### [:plugins:dataframe-gradle-plugin](./dataframe-gradle-plugin)
6+
The Gradle plugin for DataFrame that can generate data schemas from a data sample using the simple
7+
task `dataframes {}`. It uses [:plugins:symbol-processor](./symbol-processor) to generate column accessors.
8+
9+
### [:plugins:symbol-processor](./symbol-processor)
10+
The KSP plugin that can generate data schemas from a data sample using the `@file:ImportDataSchema` annotation.
11+
It is also used to generate column accessors for in the form of extension properties for
12+
(both manually written- or generated) `@DataSchema` annotated classes/interfaces.
13+
14+
### [:plugins:kotlin-dataframe](./kotlin-dataframe)
15+
The Kotlin 2.x Compiler plugin of DataFrame. A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704)
16+
plugin for your Gradle project that can generate on-the-fly column accessors for the compiler and IDE even without
17+
having to provide data schemas!
18+
19+
### [:plugins:expressions-converter](./expressions-converter)
20+
A small Kotlin Compiler plugin that provides intermediate expressions of DataFrame
21+
operation chains, used internally by [:core](../core) to generate "explainer dataframes" on the documentation website.
22+
23+
### [generator](../generator)
24+
A small Gradle plugin that is used internally to generate enums with restricted Kotlin keywords for the
25+
[:core](../core) module.
26+
27+
This module can probably be moved under [:plugins](../plugins):
28+
[Issue #899](https://github.com/Kotlin/dataframe/issues/899).

‎plugins/dataframe-gradle-plugin/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## :plugins:dataframe-gradle-plugin
2+
3+
This module holds the Gradle plugin for DataFrame, published as "org.jetbrains.kotlinx.dataframe" on the
4+
[Gradle Plugin Portal](https://plugins.gradle.org/plugin/org.jetbrains.kotlin.plugin.dataframe).
5+
6+
This plugin can let the user generate data schemas from a data sample using the simple
7+
Gradle task `dataframes {}`. It also provides an automatic dependency on [:plugins:symbol-processor](./symbol-processor)
8+
to generate column accessors and support the `@file:ImportDataSchema` notation.
9+
10+
Read more about how to use the Gradle plugin at
11+
[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html).

‎plugins/expressions-converter/README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
## :plugins:expressions-converter
2+
3+
This Kotlin Compiler plugin, used by [:core](../../core), can extract intermediate
4+
DataFrame expressions from `@TransformDataFrameExpressions` annotated functions.
5+
6+
It is used to generate sample "explainer dataframe" HTML files that can be used as iFrames on the documentation website.
7+
8+
Annotated functions in [core/.../test/.../samples/api](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
9+
are tested, generated, and copied over to [docs/StardustDocs/snippets](../../docs/StardustDocs/snippets) by
10+
our "explainer" [plugin callback proxy](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/explainer),
11+
which hooks into [the TestBase class](../../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/TestBase.kt) and
12+
retrieves the intermediate DataFrame expressions thanks to this module.

‎plugins/kotlin-dataframe/README.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
## :plugins:kotlin-dataframe
2+
3+
This module, published as "compiler-plugin-all", holds the Kotlin 2.x Compiler plugin of DataFrame.
4+
5+
A [work-in-progress](https://github.com/Kotlin/dataframe/issues/704)
6+
plugin for your Kotlin project that can generate on-the-fly column accessors for the compiler and IDE even without
7+
having to provide data schemas!
8+
9+
Details of how to use this compiler plugin for your own project will follow, but there is a
10+
[demo project](https://github.com/koperagen/df-plugin-demo) you can try already.

‎plugins/symbol-processor/README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
## :plugins:symbol-processor
2+
3+
This module holds the KSP plugin, published as the library "symbol-processor-all" that can generate data schemas from
4+
a data sample using the `@file:ImportDataSchema` annotation. This annotation can be used as an alternative to
5+
the `dataframes {}` syntax of the [Gradle plugin](../dataframe-gradle-plugin) that's declared closer to the source code.
6+
It is also used to detect (both manually written- or generated) `@DataSchema` annotated classes/interfaces to generate
7+
column accessors for in the form of extension properties.
8+
9+
If you use the [Gradle plugin](../dataframe-gradle-plugin), this module and KSP are added as a dependency automatically.
10+
11+
Read more about how to use this at
12+
[Data Schemas in Gradle projects](https://kotlin.github.io/dataframe/schemasgradle.html) and
13+
[Extension Properties API](https://kotlin.github.io/dataframe/extensionpropertiesapi.html).

‎tests/README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
## :tests
2+
3+
Code samples for the documentation website reside in [core/.../test/.../samples/api](../core/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api)
4+
(for samples that depend solely on `dataframe-core`),
5+
and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
6+
and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
7+
by [Korro](https://github.com/devcrocod/korro).
8+
9+
This module might be merged with [:core](../core): [Issue #898](https://github.com/Kotlin/dataframe/issues/898).

0 commit comments

Comments
 (0)