Skip to content

Latest commit

 

History

History
196 lines (185 loc) · 5.98 KB

TODOs.org

File metadata and controls

196 lines (185 loc) · 5.98 KB
Raw

wip notes:

  • RocksDB-level metrics would be useful

!! Replica set failover causes any in-flight requests to time out (and get dropped) !! RPC error handling is bad. Need to have a way to serialize app-level errors across the boundary

Thoughts on “Leased” operations:

  • Must skip raft loop for acceptable perf
  • need to handle failover case. Shard failover could cause “lost updates” if we skip raft
  • Gain a lease to a specific shard in the replica set. This shard serves all read traffic?
  • can clients read from a replica? ES seems to think this is OK
  • if so, still need to obtain a lease to know when the replica is fully up to date
  • Ultimately, need to guarantee all reads after a refresh are consistent

Current Notes

Missing Validation:

  • Internal mappings fields

Release Tasks

0.1 [50%]

Search [4/4]

Term Query

Bool Query

Respond with document information

match_all

APIs [6/9]

Get Index

Refresh API

Get Document by ID

Delete

Bulk API

Cluster Health API

Proper errors

Index Exists

Clustering [5/7]

Allow nodes to join without joining the master raft group

Allocate unallocated shards when a node joins the cluster

Detect when a node leaves the cluster

De-allocate shard when a node leaves the cluster

Remove inter-node polling

  • Gossip shard/index routing information

Gossip infrastructure - Discovery

Cache shard routing information

  • Done for indices. Need to figure out when to refresh it and do it for shard-level routing
  • Update local cache from gossips
  • Check meta layer synchronously if cache missed / was wrong

Indexing [3/5]

Commit on an interval instead of per-operation

Upsert semantics

Create semantics

keep track of indexed versions when commit(), trim the in-memory map when looking up, check the map for a version. If absent, use a searcher

Optimistic Locking

Deletes are hard - elasticsearch uses a time window

Response format

Schema [1/1]

Basic data types

  • Strings
  • Longs
  • Objects

Operations [1/5]

Terraform configuration

Deploy cluster to AWS

Baseline perf test

Structured logging

Expose prometheus metrics

Storage [1/2]

Raft Log Compaction

Indexing should be async, not in raft loop

Tests [1/7]

Search Storage

Integration

Raft Storage

Node Router

Gossip

Web API

Meta State Machine

Search State Machine

Polish [1/6]

Reasonable test suite (broken out)

CI

pre-commit script

Figure out license

README

Contributor guidelines

0.2 [0%]

Search [0/5]

Specify document count limit

Sepcify document offset

Sorting

Aggregations

API [0/5]

Multi Get

Get Mapping

Update index settings

API docs

Per-shard response information

Nodes cat API

Indexing [0/4]

Partial Failure in bulk requests

Update?

Schema [0/13]

Text

Float

Boolean

Dates

DateTimes

Integer

Short

Byte

Double

Float

Half Float

Scaled Float

Binary

Clustering [0/4]

Replica sets should have a lease holder

Direct all reads at the lease holder

Heartbeat at the node level, not the shard level

Remove ‘node id’ configuration

Shouldn’t have to force the user to configure this, we should figure out node ids from address or something else

0.3 [0%]

Search [0/2]

Cross-index search

API [0/1]

Update mappings

Cluster settings API

Indexing [/]

Percolation [0/1]

Implment it

Schema [0/3]

Nested Documents

Range Types?

Clustering [0/3]

Split / Merge(?)

Auto-scale replication

Icebox

Tooling

tower-grpc is interesting, but ergonomics of pingcap’s grpc are currently better. Both suck.

Tantivy Issues

  • Max field count is 255
  • Threading is weird. I want to manage the thread pool across many indices
  • No way to have a user-defined doc id

Neat Ideas

Impl Notes

Elasticsearch ids: https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/mapper/Uid.java#L178