Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EventStore source support to sync Template #16

Merged
merged 360 commits into from
Jun 18, 2019
Merged

Add EventStore source support to sync Template #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
360 commits
Select commit Hold shift + click to select a range
3f1cb1a
Adjust fudge
bartelink Apr 18, 2019
0abdcd1
Fix fudge
bartelink Apr 18, 2019
81ade3e
Another
bartelink Apr 18, 2019
172913a
Reorder progress
bartelink Apr 18, 2019
d2c4a67
Even more reorder
bartelink Apr 18, 2019
3138ed9
Naming consistency for args
bartelink Apr 23, 2019
b423251
Fix: remove temp logging
bartelink Apr 23, 2019
275a85a
Fix comment
bartelink Apr 23, 2019
88cbeb0
Add Projector State
bartelink Apr 24, 2019
d3298a9
Fix bugs
bartelink Apr 24, 2019
b770187
No struct tuples
bartelink Apr 24, 2019
aa0d3d1
Tidy ingest effiency
bartelink Apr 24, 2019
73b0b42
Partition alignment
bartelink Apr 24, 2019
c9bb1eb
Tidy RC
bartelink Apr 24, 2019
5c4c88c
Lag order
bartelink Apr 24, 2019
9624c7f
Fix lag display
bartelink Apr 24, 2019
6d4f157
Batch blocking by semaphore
bartelink Apr 25, 2019
da79921
Fixes
bartelink Apr 25, 2019
e2277c7
Fix delay layout
bartelink Apr 25, 2019
3f1b147
Async.Start instead of StartChild
bartelink Apr 25, 2019
3caea3d
Log in flight batches
bartelink Apr 25, 2019
c0190c7
MaxThreads
bartelink Apr 25, 2019
6c03a3d
Move work back to main thread
bartelink Apr 25, 2019
8cbb31c
Drop back to 10x delay
bartelink Apr 25, 2019
6baf343
Fix MinBatch naming
bartelink Apr 25, 2019
223257f
Generalize Projector and Sync
bartelink Apr 26, 2019
adc6ec0
Filter log to console
bartelink Apr 26, 2019
ab4b756
Fix malformed log
bartelink Apr 26, 2019
ecaeaa3
Limit batches
bartelink Apr 26, 2019
97f54cd
Use CosmosStats
bartelink Apr 26, 2019
86789f5
Tidy result handling
bartelink Apr 26, 2019
793eee6
Skip empty batches
bartelink Apr 26, 2019
8994cd6
Fix
bartelink Apr 26, 2019
5aa7ecd
Handle empty batch completion
bartelink Apr 26, 2019
0cb68e8
Fix Batch counting
bartelink Apr 26, 2019
92b2420
Fix exception stats
bartelink Apr 26, 2019
08a46ca
Tidy labels
bartelink Apr 26, 2019
354ae7c
Event counts+sizes
bartelink Apr 27, 2019
c7b9997
Tidy req stats
bartelink Apr 27, 2019
18dbc9b
Tidy progress
bartelink Apr 27, 2019
e067ce1
Tidy
bartelink Apr 27, 2019
b9c0d44
Tidy stream list
bartelink Apr 27, 2019
84a83bb
start Cosmos sync
bartelink Apr 27, 2019
4d4be0f
Default to not from tail
bartelink Apr 27, 2019
d2f6e83
Fix start message
bartelink Apr 27, 2019
393ec84
Generalize Ingestion
bartelink Apr 27, 2019
35e1008
Tidy
bartelink Apr 27, 2019
5accf32
Rename to Engine
bartelink Apr 27, 2019
788dbaa
Fix
bartelink Apr 27, 2019
650ff9b
reorder
bartelink Apr 27, 2019
cbf87d2
Split read/progress batching from ingestion
bartelink Apr 29, 2019
c004adf
fix
bartelink Apr 29, 2019
606dae5
Show max batch limits
bartelink Apr 29, 2019
355242b
Drain Active
bartelink Apr 29, 2019
d076e5c
fix
bartelink Apr 29, 2019
b8026e2
fix collection modification
bartelink Apr 29, 2019
b90b7c1
release Read and write
bartelink Apr 29, 2019
61aebbd
Fix buffering stats
bartelink Apr 30, 2019
ec528fe
Hook stats
bartelink Apr 30, 2019
a681882
Comments, naming
bartelink Apr 30, 2019
241d181
Fix length calcs
bartelink Apr 30, 2019
95fefcc
Up default max batches
bartelink Apr 30, 2019
d557b42
Add Gorge Mode
bartelink Apr 30, 2019
5f6770d
blacklist stream
bartelink May 1, 2019
04c1f00
Add Gorge mode impl
bartelink May 1, 2019
079073c
Fix stripe count off by one count
bartelink May 1, 2019
6a67db2
Log gorge mode
bartelink May 1, 2019
9d6c687
Add chunk buffer management + logging
bartelink May 1, 2019
46c0740
Fix chunk numbering
bartelink May 1, 2019
f02df11
Fix queue management
bartelink May 1, 2019
9a625e0
Log mode info
bartelink May 1, 2019
3eeb083
Fix chunk numbering
bartelink May 1, 2019
4f325c3
Doh
bartelink May 1, 2019
13706f7
Log buffering counts
bartelink May 1, 2019
c3aa973
Add jitter
bartelink May 1, 2019
9dad3de
Fix jitter
bartelink May 1, 2019
0cd52b0
More jitter work
bartelink May 1, 2019
4bb8d1c
Add universal jitter
bartelink May 1, 2019
d0dde02
Connection per stripe
bartelink May 1, 2019
211a7fc
Blacklist inventory
bartelink May 1, 2019
95a208f
Blacklist InvesntoryCount
bartelink May 1, 2019
bfefbab
Tidy ES ingestion, remove Ingester
bartelink May 1, 2019
1e4dc94
Reorg Engine components
bartelink May 1, 2019
7404730
Transition from Gorging to Tailing at end
bartelink May 1, 2019
75b8d7f
Split out CosmosSource
bartelink May 1, 2019
e701d15
Fix cosmos switching args
bartelink May 1, 2019
5fd7791
Tag logs
bartelink May 2, 2019
ac1837d
Rewrite :allthethings:
bartelink May 3, 2019
185cafc
Tune batch size defaults
bartelink May 3, 2019
184175c
Important !
bartelink May 3, 2019
aca53c9
Add missing Release
bartelink May 3, 2019
1312802
Fix batch Releasing
bartelink May 3, 2019
d241a49
Tmp
bartelink May 3, 2019
e568a74
Add unprefixed stats
bartelink May 3, 2019
b3458c8
fix
bartelink May 3, 2019
449c4e5
Fix locking; extend ordering
bartelink May 3, 2019
d3fff06
Fix transition
bartelink May 3, 2019
cc89893
fix
bartelink May 3, 2019
407c70a
Omit busy
bartelink May 4, 2019
0ef8512
Add filled metric
bartelink May 4, 2019
41bfddb
Fix percentage
bartelink May 4, 2019
65b2db4
Trim first batch better
bartelink May 4, 2019
e6cc9df
Slipstream repeats as fillers
bartelink May 4, 2019
7937686
Prevent too many tails
bartelink May 4, 2019
52992b5
fix
bartelink May 4, 2019
6c2471c
Add trimming
bartelink May 4, 2019
28474d0
Relabel logs
bartelink May 4, 2019
2a840df
Update to Equinox 2.0.0-preview6
bartelink May 6, 2019
5156e04
Update with master 2.0.0-preview5 changes
bartelink May 6, 2019
6152cd3
Remove tests
bartelink May 6, 2019
15a70f5
remove redundant metrics
bartelink May 6, 2019
c97311e
Fix filter
bartelink May 7, 2019
35c480f
Another blacklist
bartelink May 7, 2019
243d0df
Tweaks
bartelink May 7, 2019
72a532c
Merge branch 'master' into ingester
bartelink May 7, 2019
29c2929
Consistency tweaks
bartelink May 7, 2019
4dd3ac3
Update all to 2.0.0-preview6
bartelink May 7, 2019
a349659
Correct eqxsync progress management, rebasing on Equinox.Projection p…
bartelink May 7, 2019
cb0c2dd
Tidy lag logging
bartelink May 8, 2019
ae60dc9
Add local copy of Projection[2]
bartelink May 8, 2019
0900216
More reformatting
bartelink May 8, 2019
9fb9c0c
Rewrite projection algorithm
bartelink May 8, 2019
2ddf2f4
Fix read vs submit counts
bartelink May 8, 2019
3f674ab
Tidy builders
bartelink May 8, 2019
93771e4
CosmosIngester lifetime
bartelink May 8, 2019
149ece8
Fix dispatcher logic
bartelink May 8, 2019
6bed13e
Remove Partial Dispatcher state
bartelink May 8, 2019
e44ea38
Destruct states
bartelink May 8, 2019
6275aa1
Logging @
bartelink May 8, 2019
b61cf19
Delay materialize
bartelink May 8, 2019
63942b8
ES
bartelink May 8, 2019
3e6ed71
Disable checkpoint logging
bartelink May 8, 2019
3992d0f
Terminate slipstreaming state
bartelink May 8, 2019
6d536da
Log releases
bartelink May 8, 2019
8bb7f4e
logging
bartelink May 8, 2019
01db2eb
remove logging
bartelink May 8, 2019
0333adc
tidy scheduler
bartelink May 8, 2019
cc557cd
Reorg MaxProcessing
bartelink May 8, 2019
4a7ba98
Reorg commandline parsing
bartelink May 9, 2019
9de6980
Cosmos pooling connection pooling
bartelink May 9, 2019
ec06a50
Delay presubmits
bartelink May 9, 2019
b0e67ec
Remove false idle state
bartelink May 9, 2019
a5d1957
Submit all things
bartelink May 9, 2019
373cf4d
Render holding buffers
bartelink May 9, 2019
dbf2cc2
reorder log output
bartelink May 9, 2019
5d09abb
Render preloading separately
bartelink May 9, 2019
29c6ec1
Fix buffered message
bartelink May 9, 2019
9c3de1a
512MB
bartelink May 9, 2019
90e95ba
Add catch to buffer thread
bartelink May 9, 2019
4ffd99e
Limit count in holding loop
bartelink May 9, 2019
018f6c9
Raise thread limits
bartelink May 9, 2019
d743dcc
remove threads, limit
bartelink May 9, 2019
98470c6
default to cosmos
bartelink May 9, 2019
3cf5998
fix default read
bartelink May 9, 2019
40d3bc1
Fix maxprocessing
bartelink May 9, 2019
81a3aa9
More cosmos arg fixes
bartelink May 9, 2019
2899953
512MB again
bartelink May 9, 2019
3bb8318
256K
bartelink May 9, 2019
53e4a88
16384
bartelink May 9, 2019
7949c67
more threads
bartelink May 9, 2019
d986384
64K blocks
bartelink May 9, 2019
63eeae9
zero len writes
bartelink May 10, 2019
3b6687e
Use ConcurrentBags
bartelink May 10, 2019
3b12fff
Tune concurrency
bartelink May 10, 2019
85441fa
Make log context Ingest
bartelink May 10, 2019
eea552b
Off by one!
bartelink May 10, 2019
7586c0e
Tidy ingest; better buffer size
bartelink May 10, 2019
db64cc2
Tidy stats
bartelink May 10, 2019
8281ec2
512K
bartelink May 10, 2019
490a05f
Fix budgets at 16K events or 512KB
bartelink May 10, 2019
e1961ec
Add Stream counts
bartelink May 10, 2019
9c22045
Fix typo
bartelink May 10, 2019
f6161e0
Tidy results breakdown
bartelink May 10, 2019
b17d68d
Fix categorization
bartelink May 10, 2019
0154d8a
Fix
bartelink May 10, 2019
da2a94b
Inline state
bartelink May 10, 2019
660c796
Logging fix
bartelink May 10, 2019
2720355
Fix Batch stats
bartelink May 10, 2019
525fe94
Fix log typo
bartelink May 10, 2019
46db864
reinstate bodies
bartelink May 10, 2019
9a52563
Stack Pop fix
bartelink May 10, 2019
a651b30
Lengthen timeouts
bartelink May 11, 2019
d492d34
Correct categorization
bartelink May 12, 2019
cc8ea96
Clean slipstreaming
bartelink May 12, 2019
e786da2
Tidy completed log
bartelink May 13, 2019
b8a8259
Improve full buffers handling
bartelink May 13, 2019
dc2db73
Tidy, reinstate IsEmpty check
bartelink May 13, 2019
d4f877a
Take multiple batches in inner loop
bartelink May 13, 2019
839c548
Fix exit condition
bartelink May 13, 2019
0c9d99e
32 batches
bartelink May 13, 2019
3625747
Remove redundant ingest
bartelink May 13, 2019
ab17e63
64 batches
bartelink May 13, 2019
5325985
Change queueWeight to event count
bartelink May 13, 2019
18061c1
Change max size from 512KB to 1MB
bartelink May 13, 2019
e563e60
Roll back to 32 batches
bartelink May 13, 2019
c31db26
Add phase timing logging
bartelink May 13, 2019
8f7203d
Remove GSW
bartelink May 13, 2019
c2af38e
Remove async from dispatch
bartelink May 13, 2019
2c53849
Remove more Async
bartelink May 13, 2019
f31cdf4
Remove IsEmpty
bartelink May 13, 2019
dcdb156
reinstate weight
bartelink May 13, 2019
aeb8551
Fix sort perf
bartelink May 14, 2019
38f7011
Fix warning
bartelink May 14, 2019
1d402ce
Final fix
bartelink May 14, 2019
7e760c4
Final indignity
bartelink May 14, 2019
4771597
No tmp
bartelink May 14, 2019
9008cd3
Reduce maxBatches a lot
bartelink May 14, 2019
015a2b9
Background stream merge
bartelink May 14, 2019
5349f21
Forward merges immediately
bartelink May 14, 2019
2325b5f
Reduce merge frequency
bartelink May 14, 2019
3dfe73d
Throttle batch submission
bartelink May 14, 2019
c718336
4s submit interval
bartelink May 14, 2019
d8eca45
16 batches
bartelink May 14, 2019
3f3f29b
Grayscale
bartelink May 14, 2019
6228d42
Rebase on Equinox 2.0.0-preview7
bartelink May 14, 2019
0c04058
Drop level on messages that should not be Warnings
bartelink May 14, 2019
2df06c6
Add Exception stats
bartelink May 14, 2019
0330396
Reset exception counters
bartelink May 14, 2019
46ff26c
Rebadge timings
bartelink May 14, 2019
fcb9f3d
Kafka projector rewrite
bartelink May 15, 2019
36da634
Kafka fixes
bartelink May 16, 2019
81fc65e
WIP
bartelink May 16, 2019
497c68a
Merge branch 'master' into ingester
bartelink May 17, 2019
69a5350
Target 2.0.0-preview8
bartelink May 17, 2019
3a99e01
Merge branch 'master' into ingester
bartelink May 20, 2019
205cee1
Merge branch 'master' into ingester
bartelink Jun 3, 2019
05deff3
Replace ifdefs with matches
bartelink Jun 5, 2019
e375909
Add ChangeLog info
bartelink Jun 5, 2019
87ffa03
Tidy for consistency with Sync
bartelink Jun 5, 2019
4cd9be7
Consistency tweaks
bartelink Jun 5, 2019
f23ac00
Adjust to propulsion naming
bartelink Jun 5, 2019
e7531aa
fix typo
bartelink Jun 6, 2019
5197a13
Add #serial filtering
bartelink Jun 7, 2019
d5e91cb
Log
bartelink Jun 7, 2019
397ae45
LOG
bartelink Jun 7, 2019
064d787
Merge branch 'master' into ingester
bartelink Jun 7, 2019
dd17999
kafka
bartelink Jun 8, 2019
0045a0d
Ref local
bartelink Jun 10, 2019
d570cb4
log exns
bartelink Jun 11, 2019
bdc884e
low compress
bartelink Jun 11, 2019
1f80df7
compress off
bartelink Jun 11, 2019
ae09642
fix
bartelink Jun 11, 2019
1dbf5b9
compress again
bartelink Jun 11, 2019
f81a32f
Parameterize producers
bartelink Jun 11, 2019
98136d9
Deal with renamespacings
bartelink Jun 14, 2019
c2befdf
Use separated Kafka config
bartelink Jun 17, 2019
16467a8
ref local propulsionI
bartelink Jun 17, 2019
effe77f
Update after Kafka stats reorg
bartelink Jun 17, 2019
f5e3864
Merge branch 'master' into ingester
bartelink Jun 18, 2019
3abdb54
Remove propuslision refs
bartelink Jun 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The `Unreleased` section name is replaced by the expected version of next releas
- `eqxprojector -k`'s `Consumer` summarizes processing outcomes in example using new support for same in `Propulsion.Kafka` [#25](https://github.com/jet/dotnet-templates/pull/25)
- `eqxprojector -k -n`'s `Producer` offers a parallel producer mode which runs all projections in parallel without constraints (or need to synthesize streams) [#24](https://github.com/jet/dotnet-templates/pull/24)
- `eqxsync` has EventStore Sink support via `cosmos` ... `es` commandline option [#23](https://github.com/jet/dotnet-templates/pull/23)
- `eqxsync` has EventStore Source support via `es` ... `cosmos` commandline option [#16](https://github.com/jet/dotnet-templates/pull/16)

### Changed

Expand Down
Loading