Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EventStore source support to sync Template #16

Merged
merged 360 commits into from
Jun 18, 2019
Merged

Add EventStore source support to sync Template #16

merged 360 commits into from
Jun 18, 2019

Conversation

bartelink
Copy link
Collaborator

@bartelink bartelink commented Mar 8, 2019

While this works, provides good throughput and other things, there's still stuff to do so it's really not ready to "review" in the sense of evaluating "is this a clean simple ingester?" atm (clue: atm the code is too messy, and there is an obvious need to add support for doing multiple concurrent readers)

TL;DR at the moment its a POC EventStore->Equinox.CosmosDb one-way ingester, but the general principle is that there are lots of uses for an efficient bulk sync tool working at the stream level in order to optimize ingesting performance (and batch writes to same stream) - this allows one to subset by Category and/or do transforms when propagating data onward

This is approaching the final shape of things in that - the Sync program is intended to be a continuous app you run in prod that can support Cosmos -> Cosmos and EventStore -> Cosmos ingestion - the Ingest Program only applies to EventStore -> Cosmos loading and is intended to be run once on a standalone basis

The equinox-sync template's Sync app includes support for both the striped reading (which was originally pioneered in an Ingest application) (badged gorge mode ,-g) and working as a more traditional projector (which it reverts to after any gorge mode)

Work list:

  • Make it ingest from EventStore into Equinox.Cosmos
  • Efficiently (processing can keep up with a reader getting 4096 event batches from a local ES)
  • Reporting status cleanly (status messages etc are clear and concise)
  • provide striped mode when ingesting from EventStore (multiple readers, using offset computation based on @ scarvel8's event_global_position = 256 x 1024 x 1024 x chunk_number + chunk_header_size (128) + event_position_offset_in_chunk)
  • dont report malformed exceptions
  • report stream state
  • implement parallel readers
  • extend to also provide continuous ingestion
  • support ingesting from CosmosDb into Cosmos now in dotnet new eqxsync
  • Make 429 exceptions more succint
  • extract EventStore walk and enumeration code to a separate file
  • extend to record EventStore -> Cosmos progress in destination and/or aux collection
  • unify -i and -g impls using max of both and then adjusting when switching to tailing
  • implement cosmos connection pooling
  • enhance slipstream submission protocol to let cosmosingester grab stuff without locking (or keeping message bus busy)
  • support ingesting into EventStore by building an EventStoreSink
  • fix tailing switch for ES where >1 discover EOF Fix StripedIngester transition to tailing propulsion#10
  • re-add stream catchup reading for ES Add Streamwise Catchup mode propulsion#11

@bartelink bartelink force-pushed the ingester branch 5 times, most recently from 8daaab8 to cebc41f Compare March 15, 2019 10:49
@bartelink bartelink force-pushed the ingester branch 3 times, most recently from 05616e0 to 52d7200 Compare March 21, 2019 01:31
@bartelink bartelink mentioned this pull request Mar 24, 2019
@bartelink bartelink force-pushed the ingester branch 2 times, most recently from 71152ef to 893129b Compare March 26, 2019 17:03
@bartelink bartelink force-pushed the ingester branch 2 times, most recently from c2c63af to b2b82d1 Compare April 3, 2019 20:54
@bartelink bartelink changed the title Add eqxsync template Add eqxingest template Apr 3, 2019
@bartelink bartelink force-pushed the ingester branch 2 times, most recently from 7b82a39 to 269c2ab Compare April 14, 2019 14:59
@bartelink bartelink changed the title Add eqxingest template Add EventStore source support to sync Template Apr 14, 2019
@bartelink
Copy link
Collaborator Author

Marking as ready for review - I still have some tidying to do, but the basic split between Sync's EventStore mode and the Ingest Program (which started life as a Template) is now in place.

@bartelink bartelink marked this pull request as ready for review April 14, 2019 15:09
@bartelink bartelink force-pushed the ingester branch 5 times, most recently from 4797c06 to 26c4985 Compare April 17, 2019 23:50
@bartelink bartelink merged commit 2928c5a into master Jun 18, 2019
@bartelink bartelink deleted the ingester branch June 18, 2019 16:05
@bartelink bartelink restored the ingester branch June 18, 2019 16:10
bartelink added a commit that referenced this pull request Jun 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant