Architecture Documentation (#42)

Author: bbengfort

docs/.gitignore

@@ -2,5 +2,4 @@
 date = '{{ .Date }}'
 draft = true
 title = '{{ replace .File.ContentBaseName "-" " " | title }}'
-disableTOC = false
 +++

docs/assets/.gitkeep docs/.hugo_build.lock

@@ -0,0 +1,64 @@
+---
+title: HonuDB Documentation
+weight: 0
+geekdocNav: false
+geekdocAlign: center
+geekdocAnchor: false
+geekdocBreadcrumb: true
+---
+
+<!-- markdownlint-capture -->
+<!-- markdownlint-disable MD033 -->
+
+<span class="badge-placeholder">[![Build Status](https://github.com/rotationalio/honu/actions/workflows/tests.yaml/badge.svg)](https://github.com/rotationalio/honu/actions/workflows/tests.yaml)</span>
+<span class="badge-placeholder">[![GitHub Release](https://img.shields.io/github/v/release/rotationalio/honu)](https://github.com/rotationalio/honu/releases/latest)</span>
+<span class="badge-placeholder">[![GitHub Contributors](https://img.shields.io/github/contributors/rotationalio/honu)](https://github.com/rotationalio/honu/graphs/contributors)</span>
+<span class="badge-placeholder">[![License: BSD3](https://img.shields.io/github/license/rotationalio/honu)](https://github.com/rotationalio/honu/blob/main/LICENSE)</span>
+
+<!-- markdownlint-restore -->
+
+The HonuDB Database is the first AI native distributed database intended for an audience of AI developers who need to manage multi-modal datasets with snapshots that can map to models and model training. A replicated document database, HonuDB provides rapid data ingestion and collection management for different mimetypes including JSON, Parquet, images, video, and more. With privacy in mind from the start, HonuDB has data governance features such as provenance and lineage tracking (including by geographic location), and fine-grain access controls. Data scientists and machine learning engineers can rely on Honu to manage small to extremely large datasets replicated over multiple geographic areas.
+
+{{< button size="large" relref="quickstart/" >}}Get Started Now!{{< /button >}}
+
+## Feature overview
+
+{{< columns >}}
+
+### Collections &amp; Datasets
+
+Collections allow you to manage related data together; Datasets are snapshots of collections that indicate exactly what data was usd to train a model.
+
+<--->
+
+### Full Versioning
+
+All objects in the database are fully versioned to prevent an update from changing the view of a dataset from a model perspective.
+
+<--->
+
+### Provenance Awareness
+
+Regions and unique writers are tracked across all updates so you can monitor how data is changing in your system and implement privacy controls.
+
+{{< /columns >}}
+
+{{< columns >}}
+
+### Smart Replication
+
+Honu uses reinforcement learning anti-entropy to maximize consistency and scale replication to hundreds of nodes without increasing your cloud costs.
+
+<--->
+
+### Fine-Grain Access Control
+
+Collections, objects, and datasets have a hierarchical permission model specifically for AI workloads including training and inferencing permissions.
+
+<--->
+
+### Model Context Protocol
+
+Honu supports the [Model Context Protocol](https://modelcontextprotocol.io/introduction) so that you can directly add data to your LLM contexts using semantic similarity indexes.
+
+{{< /columns >}}

docs/archetypes/default.md

@@ -0,0 +1,24 @@
+---
+title: Architecture
+weight: 500
+---
+
+This section is primarily for contributors and developers of the HonuDB replicated database. In it, we will describe the design principles of the database system, how features are implemented and integrated, and the path towards creating complex systems designed from many simple components.
+
+**Design Goal**<br />
+The goal of the database is to provide scalable data retrieval both in terms of number of nodes (e.g. scale to 100s of nodes) and amount of data (hundreds of terabytes). In addition to scale, this database provides data access controls, privacy and provenance, and other security related features. Finally, HonuDB provides artifacts and features to support the traning and inferencing model lifecycle. In short, HonuDB is a distributed data governance database for machine learning and artificial intelligence workloads.
+
+**On This Page**
+
+{{< toc >}}
+
+## System Diagram
+
+
+
+## Key Terms
+
+<dl>
+    <dt>Engine</dt>
+    <dd>A database engine is a component that manages how data is stored and cached on disk.</dd>
+</dl>

docs/content/_index.md

@@ -0,0 +1,14 @@
+---
+title: Data Model
+weight: 10
+resources:
+- name: keylayout
+  src: figures/keylayout.svg
+  title: Byte Layout of Keys
+---
+
+The database engine manages the data as key/value pairs on disk such that the keys are ordered in byte-sort order for fast iteration. Generally speaking the engine uses an LSM-Tree (log structured merge tree) or similar structure for fast appends to the database and routine compaction and merging.
+
+The layout of the `keys` and the data objects are important to understand.
+
+{{< img name="keylayout" size="large" lazy=false >}}

docs/content/architecture/_index.md

@@ -0,0 +1,20 @@
+---
+title: Contributing
+weight: 450
+---
+
+HonuDB is an open source project that is supported by a community who will gratefully accept any contributions you might make to the project. Large or small, any contribution makes a difference! While Rotational Labs funds the development of HonuDB, we have committed to ensuring that HonuDB always remains open source; please rest assured that your contributions build the community and is not simply free work.
+
+Database development is notoriously difficult -- but hopefully contributing to database code is not. For example, the implementation of an algorithm or component can often be accomplished without affecting the rest of the database code. If you're not sure where to start, look for `TODO` comments in the code or [get in touch with us](https://rotational.io/contact)!
+
+Beyond the code, there are many ways to contribute:
+
+- Submit a feature request or bug report on our [GitHub Issues](https://github.com/rotationalio/honu/issues).
+- Add to the documentation or help us with our website, [honudb.dev](https://honudb.dev).
+- Write a blog post, tweet, or share our project with others.
+- Star our [https://github.com/rotationalio/honu] on GitHub!
+- Translate our documentation into another language.
+- Write unit or integration tests for Honu.
+- Tell us about how you're using HonuDB!
+
+There are lots of ways to get involved, and we'd love to have you be a part of our community.

docs/content/architecture/data-model/_index.md

@@ -0,0 +1,20 @@
+---
+title: Introducing HonuDB
+type: posts
+date: 2025-02-27
+tags:
+  - Updates
+  - Informational
+---
+
+Why does the world need yet another database? In short, because no one database can support all use cases all the time. HonuDB is for machine learning engineers, a group that needs data management more than most, but is often overlooked as users for database management systems.
+
+<!--more-->
+
+Instead of creating a single purpose tool like a vector database, or augmenting an existing database with vector capabilities like Elastic; HonuDB is focused on the workflow of AI and model development **from training datasets to inferencing context**.
+
+The AI/ML workflow has specialized features that are not generally found together in a single system. To support reproducibility, datasets must be versioned and snapshotted so that training datasets can be mapped to their models, and datasets are a first class access pattern in HonuDB. We also understand how important _privacy_ and _data governance_ is, especially when it comes to AI -- so HonuDB is built to support provenance based investigations and geographic access controls. ML datasets range from the very small to the very large, so HonuDB can operate as a single node or scale to replicate to hundreds of nodes across multiple geographic regions. Finally, vector queries and model context protocols are needed for inferencing, and Honu is ready to support these protocols for RAG workflows.
+
+HonuDB not only supports AI/ML workflows but uses ML under the hood to improve its performance. Based on the academic papers [Anti-Entropy Bandits for Geo-Replicated Consistency](https://ieeexplore.ieee.org/document/8416408) and [Bilateral Anti-Entropy for Eventual Consitency](https://dl.acm.org/doi/10.1145/3517209.3524083), HonuDB uses reinforcement learning to optimize replication and consistency in the wide area!
+
+Always open-source, we hope HonuDB will accelerate your projects and that you'll enjoy using it as much as we do.

docs/content/architecture/data-model/figures/keylayout.png

@@ -0,0 +1,6 @@
+---
+title: News
+type: posts
+weight: 10
+geekdocHidden: true
+---

docs/content/architecture/data-model/figures/keylayout.svg

@@ -0,0 +1,9 @@
+---
+title: Getting Started
+weight: -20
+---
+
+{{< hint type="caution" icon="gdoc_fire" title="Coming Soon" >}}
+**We're still in development**\
+HonuDB is still in an Alpha development phase and is not ready for prime time use. We're excited that you're excited to try it out; if you're willing to be a Beta tester, please get in [contact with us](https://rotational.io/contact/)!
+{{< /hint >}}

docs/content/contributing/_index.md

@@ -0,0 +1,6 @@
+---
+header:
+  - name: GitHub
+    ref: https://github.com/rotationalio/honu
+    icon: gdoc_github
+    external: true

docs/content/en/_index.md

@@ -0,0 +1,13 @@
+---
+more:
+  - name: News
+    ref: "/posts"
+    icon: "gdoc_gitea"
+  - name: Releases
+    ref: "https://github.com/rotationalio/honu/releases"
+    external: true
+    icon: "gdoc_download"
+  - name: View Source
+    ref: "https://github.com/rotationalio/honu"
+    external: true
+    icon: "gdoc_github"

docs/content/posts/2025-02-27-introducing-honudb.md

@@ -0,0 +1,147 @@
+baseURL = 'https://honudb.dev/'
+languageCode = 'en-us'
+title = 'HonuDB'
+theme = "hugo-geekdoc"
+
+pluralizeListTitles = false
+
+# Geekdoc required configuration
+# Ensures well formatted code blocks
+pygmentsUseClasses = true
+pygmentsCodeFences = true
+disablePathToLower = true
+enableGitInfo = true
+
+# Required if you want to render robots.txt template
+enableRobotsTXT = true
+
+# Needed for mermaid shortcodes
+[markup]
+  [markup.goldmark.renderer]
+    # Needed for mermaid shortcode or when nesting shortcodes
+    unsafe = true
+  [markup.tableOfContents]
+    startLevel = 1
+    endLevel = 9
+
+[taxonomies]
+  tag = "tags"
+
+[params]
+  description = "HonuDB is the worlds first AI native distributed database that provides data support for both training and inferencing workloads. Designed to scale to hundreds of nodes, HonuDB provides machine learning engineers practical dataset management, a model context protocol, vector search, and annotation references."
+  keywords = [
+    "HonuDB",
+    "database",
+    "distributed database",
+    "vector database",
+    "AI native database",
+    "dataset management",
+    "model context protocol",
+    "vector search",
+    "data governance",
+    "provenance",
+    "lineage",
+    "reinforcement learning",
+    "anti-entropy replication"
+  ]
+
+  images = [
+    "socialmedia.png"
+  ]
+
+  # (Optional, default 6) Set how many table of contents levels to be showed on page.
+  # Use false to hide ToC, note that 0 will default to 6 (https://gohugo.io/functions/default/)
+  # You can also specify this parameter per page in front matter.
+  geekdocToC = 3
+
+  # (Optional, default static/brand.svg) Set the path to a logo for the Geekdoc
+  # relative to your 'static/' folder.
+  geekdocLogo = "logo.png"
+
+  # (Optional, default false) Render menu from data file in 'data/menu/main.yaml'.
+  # See also https://geekdocs.de/usage/menus/#bundle-menu.
+  geekdocMenuBundle = false
+
+  # (Optional, default false) Collapse all menu entries, can not be overwritten
+  # per page if enabled. Can be enabled per page via 'geekdocCollapseSection'.
+  geekdocCollapseAllSections = true
+
+  # (Optional, default true) Show page navigation links at the bottom of each docs page.
+  geekdocNextPrev = true
+
+  # (Optional, default true) Show a breadcrumb navigation bar at the top of each docs page.
+  # You can also specify this parameter per page in front matter.
+  geekdocBreadcrumb = true
+
+  # (Optional, default none) Set source repository location. Used for 'Edit page' links.
+  # You can also specify this parameter per page in front matter.
+  geekdocRepo = "https://github.com/rotationalio/honu"
+
+  # (Optional, default none) Enable 'Edit page' links. Requires 'geekdocRepo' param
+  # and the path must point to the parent directory of the 'content' folder.
+  # You can also specify this parameter per page in front matter.
+  geekdocEditPath = "edit/main/docs"
+
+  # (Optional, default false) Show last modification date of the page in the header.
+  # Keep in mind that last modification date works best if `enableGitInfo` is set to true.
+  geekdocPageLastmod = false
+
+  # (Optional, default true) Enables search function with flexsearch.
+  # Index is built on the fly and might slow down your website.
+  geekdocSearch = true
+
+  # (Optional, default false) Display search results with the parent folder as prefix. This
+  # option allows you to distinguish between files with the same name in different folders.
+  # NOTE: This parameter only applies when 'geekdocSearch = true'.
+  geekdocSearchShowParent = true
+
+  # (Optional, default none) Add a link to your Legal Notice page to the site footer.
+  # It can be either a remote url or a local file path relative to your content directory.
+  geekdocLegalNotice = "https://rotational.io/terms"
+
+  # (Optional, default none) Add a link to your Privacy Policy page to the site footer.
+  # It can be either a remote url or a local file path relative to your content directory.
+  geekdocPrivacyPolicy = "https://rotational.io/privacy"
+
+  # (Optional, default true) Add an anchor link to headlines.
+  geekdocAnchor = true
+
+  # (Optional, default true) Copy anchor url to clipboard on click.
+  geekdocAnchorCopy = true
+
+  # (Optional, default true) Enable or disable image lazy loading for images rendered
+  # by the 'img' shortcode.
+  geekdocImageLazyLoading = true
+
+  # (Optional, default false) Set HTMl <base> to .Site.Home.Permalink if enabled. It might be required
+  # if a subdirectory is used within Hugos BaseURL.
+  # See https://developer.mozilla.org/de/docs/Web/HTML/Element/base.
+  geekdocOverwriteHTMLBase = false
+
+  # (Optional, default true) Enable or disable the JavaScript based color theme toggle switch. The CSS based
+  # user preference mode still works.
+  geekdocDarkModeToggle = true
+
+  # (Optional, default false) Auto-decrease brightness of images and add a slightly grayscale to avoid
+  # bright spots while using the dark mode.
+  geekdocDarkModeDim = false
+
+  # (Optional, default false) Enforce code blocks to always use the dark color theme.
+  geekdocDarkModeCode = false
+
+  # (Optional, default true) Display a "Back to top" link in the site footer.
+  geekdocBackToTop = true
+
+  # (Optional, default false) Enable or disable adding tags for post pages automatically to the navigation sidebar.
+  geekdocTagsToMenu = true
+
+  # (Optional, default 'title') Configure how to sort file-tree menu entries. Possible options are 'title', 'linktitle',
+  # 'date', 'publishdate', 'expirydate' or 'lastmod'. Every option can be used with a reverse modifier as well
+  # e.g. 'title_reverse'.
+  geekdocFileTreeSortBy = "title"
+
+  # (Optional, default none) Adds a "Content licensed under <license>" line to the footer.
+  # Could be used if you want to define a default license for your content.
+  # [params.geekdocContentLicense]
+  #   name = "CC BY-SA 4.0"
+  #   link = "https://creativecommons.org/licenses/by-sa/4.0/"