Skip to content

Commit

Permalink
Cleanse non-Propulsion artifacts
Browse files Browse the repository at this point in the history
  • Loading branch information
bartelink committed Jun 5, 2019
1 parent d962230 commit a43c234
Show file tree
Hide file tree
Showing 8 changed files with 74 additions and 169 deletions.
150 changes: 6 additions & 144 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,7 @@ The `Unreleased` section name is replaced by the expected version of next releas

### Changed

- Targets `Confluent.Kafka` v `1.0.1`, `librdkafka.redist` v `1.0.1`

### Removed
### Fixed
- Targets `Jet.ConfluentKafka.FSharp` v `1.0.1`

<a name="1.0.0-rc13"></a>
## [1.0.0-rc13] - 2019-06-01
Expand All @@ -42,157 +39,22 @@ The `Unreleased` section name is replaced by the expected version of next releas

- Split reusable components of `ParallelConsumer` out into independent `Propulsion` and `Propulsion.Kafka` libraries [#34](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/34)

<a name="1.0.0-rc12"></a>
## [1.0.0-rc12] - 2019-05-31

### Added

- Included `totalLag` in Consumer Stats

### Changed

- Default `minInFlightBytes` is now 2/3 of `maxInFlightBytes`
- Reduced `Thread.Sleep` when over `maxInFlightBytes` threshold from `5` to `1` ms
## 1.0.0-rc12 - 2019-05-31

### Fixed

- Significant tuning / throughput improvements for `ParallelConsumer`

<a name="1.0.0-rc11"></a>
## [1.0.0-rc11] - 2019-05-27
## 1.0.0-rc11 - 2019-05-27

### Added

- `ParallelConsumer` [#33](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/33)

### Fixed

- reinstated `AutoOffsetReset` logging in `KafkaConsumerConfig`

<a name="1.0.0-rc10"></a>
## [1.0.0-rc10] - 2019-05-22

### Added

- mechanism to remove logging regarding polling backoff [#32](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/32) HT [@szer](https://github.com/Szer) re [#31](https://github.com/jet/Jet.ConfluentKafka.FSharp/issues/31)

### Changed

- split batching behaviors out into `BatchedProducer`/`BatchedConsumer` [#30](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/30)
- default auto-commit interval dropped from 10s to 5s (which is the `Confluent.Kafka` default) [#30](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/30)
- removed curried `member` Method arguments in `Start` methods

<a name="1.0.0-rc9"></a>
## [1.0.0-rc9] - 2019-05-22

### Added

- each configuration DSL now has a `customize` function to admit post-processing after defaults and `custom` have taken effect [#29](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/29)
- Producer/Consumer both have an `Inner` to enable custom logic [#29](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/29)

### Changed

- default auto-commit interval dropped from 10s to 5s (which is the `Confluent.Kafka` default) [#29](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/29)
- default `fetchMinBytes` dropped from 10 to 1 (which is the `Confluent.Kafka` default) [#29](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/29)

<a name="1.0.0-rc8"></a>
## [1.0.0-rc8] - 2019-05-21

### Fixed

- Make custom parameters in consumer config a seq once again [#28](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/28) [@szer](https://github.com/Szer)

<a name="1.0.0-rc7"></a>
## [1.0.0-rc7] - 2019-05-16

### Added

- Exposed [single-item] `ProduceAsync` in `KafkaProducer`

<a name="1.0.0-rc6"></a>
## [1.0.0-rc6] - 2019-04-24

### Changed

- Updated to target `Confluent.Kafka 1.0.0`

<a name="1.0.0-rc5"></a>
## [1.0.0-rc5] - 2019-04-24

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC7`

<a name="1.0.0-rc4"></a>
## [1.0.0-rc4] - 2019-04-23

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC6`

<a name="1.0.0-rc3"></a>
## [1.0.0-rc3] - 2019-04-12

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC4`

### Fixed

- Cleaned minor logging inconsistency wrt `CompressionType`

<a name="1.0.0-rc2"></a>
## [1.0.0-rc2] - 2019-04-02

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC3` [#24](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/24)

<a name="1.0.0-rc1"></a>
## [1.0.0-rc1] - 2019-03-27

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC2` (which references `librdkafka.redist 1.0.0`) [#23](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/23)
- Pins `rdkafka` and `Confluent.Kafka` dependencies to specific known good versions as above [#22](https://github.com/jet/Jet.ConfluentKafka.FSharp/issues/22)

<a name="1.0.0-preview2"></a>
## [1.0.0-preview2] - 2019-03-26

### Changed

- Updated to target `Confluent.Kafka 1.0.0-RC1` (triggered relatively minor changes internally due to sane API fixes, does not update to rdkafka 1.0.0, still `1.0.0-RC9`) [#21](https://github.com/jet/Jet.ConfluentKafka.FSharp/pull/21)

<a name="1.0.0-preview1"></a>
## [1.0.0-preview1] - 2019-03-05

(Extracted from [Equinox Changelog](https://github.com/jet/equinox/blob/master/CHANGELOG.md) - this codebase was maintained within that repo originally)

### Added

- `Equinox.Projection.Kafka` consumer metrics emission, see [Equinox #94](https://github.com/jet/equinox/pull/94) @michaelliao5
- Initial release as part of `Equinox.Cosmos` projection facilities, see [Equinox #87](https://github.com/jet/equinox/pull/87) @michaelliao5

<a name="1.0.0-bare"></a>
## [1.0.0-bare]
## squashed prior to initial relevant commit

(Stripped down repo for history purposes, see [`v0` branch](tree/v0) for implementation targeting `Confluent.Kafka` v `0.9.4`)
(Stripped down repo for history purposes, see [`master` branch of Jet.ConfluentKafka.FSharp for complete history prior to 1.0.0-rc13](https://github.com/jet/Jet.ConfluentKafka.FSharp/blob/master/CHANGELOG.md))

[Unreleased]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.1-rc1...HEAD
[1.0.1-rc1]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc13...1.0.1-rc1
[1.0.0-rc13]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc12...1.0.0-rc13
[1.0.0-rc12]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc11...1.0.0-rc12
[1.0.0-rc11]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc10...1.0.0-rc11
[1.0.0-rc10]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc9...1.0.0-rc10
[1.0.0-rc9]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc8...1.0.0-rc9
[1.0.0-rc8]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc7...1.0.0-rc8
[1.0.0-rc7]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc6...1.0.0-rc7
[1.0.0-rc6]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc5...1.0.0-rc6
[1.0.0-rc5]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc4...1.0.0-rc5
[1.0.0-rc4]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc3...1.0.0-rc4
[1.0.0-rc3]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc2...1.0.0-rc3
[1.0.0-rc2]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-rc1...1.0.0-rc2
[1.0.0-rc1]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-preview2...1.0.0-rc1
[1.0.0-preview2]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-preview1...1.0.0-preview2
[1.0.0-preview1]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/1.0.0-bare...1.0.0-preview1
[1.0.0-bare]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/e4bc8ff53b4f4400308b09c02fe8da6fc7e61d82...1.0.0-bare
[1.0.0-rc13]: https://github.com/jet/Jet.ConfluentKafka.FSharp/compare/d2caf9a007a137994e91ab709c87eb29fe32489b...1.0.0-rc13
10 changes: 5 additions & 5 deletions Directory.Build.props
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
<Project ToolsVersion="15.0">
<PropertyGroup>
<Authors>@jet @eulerfx @eiriktsarpalis @michaelliao5 and contributors</Authors>
<Authors>@jet @bartelink @eiriktsarpalis and contributors</Authors>
<Company>Jet.com</Company>
<Description>Lightweight F# wrapper for Confluent.Kafka</Description>
<PackageProjectUrl>https://github.com/jet/Jet.ConfluentKafka.FSharp</PackageProjectUrl>
<PackageTags>kafka fsharp</PackageTags>
<Description>Composable high performance event streaming pipelines</Description>
<PackageProjectUrl>https://github.com/jet/propulsion</PackageProjectUrl>
<PackageTags>fsharp eventsourcing cosmosdb changefeedprocessor eventstore kafka</PackageTags>
<PackageLicense>Apache-2.0</PackageLicense>
<Copyright>Copyright © 2016-9</Copyright>
<Copyright>Copyright © 2018-9</Copyright>

<RepoDir>$([System.IO.Path]::GetFullPath("$(MSBuildThisFileDirectory)"))</RepoDir>

Expand Down
14 changes: 0 additions & 14 deletions Jet.ConfluentKafka.FSharp.sln → Propulsion.sln
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{302B09C4
tests\xunit.runner.json = tests\xunit.runner.json
EndProjectSection
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Jet.ConfluentKafka.FSharp.Integration", "tests\Jet.ConfluentKafka.FSharp.Integration\Jet.ConfluentKafka.FSharp.Integration.fsproj", "{1D205901-2742-45EA-8255-E6322B49E3BB}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{6EADC23C-D6CA-4CBB-AB71-DA4A218AE24D}"
ProjectSection(SolutionItems) = preProject
.gitattributes = .gitattributes
Expand All @@ -31,8 +29,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898}"
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Jet.ConfluentKafka.FSharp", "src\Jet.ConfluentKafka.FSharp\Jet.ConfluentKafka.FSharp.fsproj", "{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}"
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Propulsion", "src\Propulsion\Propulsion.fsproj", "{0F72360F-1C14-46E3-9A60-B6BF87BD726D}"
EndProject
Project("{6EC3EE1D-3C4E-46DD-8F32-0CC8E7565705}") = "Propulsion.Kafka", "src\Propulsion.Kafka\Propulsion.Kafka.fsproj", "{5F176C54-609B-4D2E-804B-3C1F60ADDAF4}"
Expand All @@ -51,14 +47,6 @@ Global
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{1D205901-2742-45EA-8255-E6322B49E3BB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{1D205901-2742-45EA-8255-E6322B49E3BB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{1D205901-2742-45EA-8255-E6322B49E3BB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{1D205901-2742-45EA-8255-E6322B49E3BB}.Release|Any CPU.Build.0 = Release|Any CPU
{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Debug|Any CPU.Build.0 = Debug|Any CPU
{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Release|Any CPU.ActiveCfg = Release|Any CPU
{76802BE3-00C2-4B1D-96A2-95A3E2136DBE}.Release|Any CPU.Build.0 = Release|Any CPU
{0F72360F-1C14-46E3-9A60-B6BF87BD726D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0F72360F-1C14-46E3-9A60-B6BF87BD726D}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0F72360F-1C14-46E3-9A60-B6BF87BD726D}.Release|Any CPU.ActiveCfg = Release|Any CPU
Expand Down Expand Up @@ -88,8 +76,6 @@ Global
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{1D205901-2742-45EA-8255-E6322B49E3BB} = {302B09C4-7F38-4CF7-93B9-1B7A6035386E}
{76802BE3-00C2-4B1D-96A2-95A3E2136DBE} = {4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898}
{0F72360F-1C14-46E3-9A60-B6BF87BD726D} = {4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898}
{5F176C54-609B-4D2E-804B-3C1F60ADDAF4} = {4670F7C4-A4FD-4E3F-B97C-99F9B3FC1898}
{BBD50DA2-7932-4D24-888D-C168F4788705} = {302B09C4-7F38-4CF7-93B9-1B7A6035386E}
Expand Down
62 changes: 62 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# Propulsion [![Build Status](https://dev.azure.com/jet-opensource/opensource/_apis/build/status/jet.Propulsion)](https://dev.azure.com/jet-opensource/opensource/_build/latest?definitionId=7) [![release](https://img.shields.io/github/release-pre/jet/propulsion.svg)](https://github.com/jet/propulsion/releases) [![NuGet](https://img.shields.io/nuget/vpre/Propulsion.svg?logo=nuget)](https://www.nuget.org/packages/Propulsion/) [![license](https://img.shields.io/github/license/jet/propulsion.svg)](LICENSE) ![code size](https://img.shields.io/github/languages/code-size/jet/equinox.svg)

This is pre-production code; unfortunately it's also pre-documentation atm (will sadd some info over the coming weeks)

## QuickStart

See [the Equinox QuickStart](https://github.com/jet/equinox#quickstart) covers the Kafka aspect to a minor degree, but really, this is still a TODO item.

## Components

The components within this repository are delivered as a multi-targeted Nuget package targeting `net461` (F# 3.1+) and `netstandard2.0` (F# 4.5+) profiles

- [![NuGet](https://img.shields.io/nuget/vpre/Propulsion.svg)](https://www.nuget.org/packages/Propulsion/) `Propulsion`. Implements core functionality in a channel-independent fashion including `ParallelProjector`, `StreamsProjector`. [Depends](https://www.fuget.org/packages/Propulsion) on `Serilog`
- [![NuGet](https://img.shields.io/nuget/vpre/Propulsion.Cosmos.svg)](https://www.nuget.org/packages/Propulsion.Cosmos/) `Propulsion.Cosmos`. Provides bindings to Azure CosmosDb a) writing to `Equinox.Cosmos` :- `CosmosSink` b) reading from CosmosDb's changefeed by wrapping the [`dotnet-changefeedprocessor` library](https://github.com/Azure/azure-documentdb-changefeedprocessor-dotnet) :- `CosmosSource`. [Depends](https://www.fuget.org/packages/Propulsion.Cosmos) on `Equinox.Cosmos`, `Serilog`, `Microsoft.Azure.DocumentDB.ChangeFeedProcessor`
- [![NuGet](https://img.shields.io/nuget/vpre/Propulsion.EventStore.svg)](https://www.nuget.org/packages/Propulsion.EventStore/) `Propulsion.EventStore`. Provides bindings to [EventStore](eventstore.org), writing via `Propulsion.EventStore.EventStoreSink` [Depends](https://www.fuget.org/packages/Propulsion.EventStore) on `Equinox.EventStore`, `Serilog`
- [![NuGet](https://img.shields.io/nuget/vpre/Propulsion.Kafka.svg)](https://www.nuget.org/packages/Propulsion.Kafka/) `Propulsion.Kafka`. Provides bindings for producing and consuming both streamwise and in parallel. Includes a standard codec for use with streamwise projection and consumption, `Propulsion.Kafka.Codec.RenderedSpan`. [Depends](https://www.fuget.org/packages/Propulsion.Kafka) on `Jet.ConfluentKafka.FSharp`, `Serilog`

The ubiquitous Serilog dependency is solely on the core module, not any sinks, i.e. you configure to emit to `NLog` etc.

## CONTRIBUTING

Please raise GitHub issues for any questions so others can benefit from the discussion.

This is an Open Source project for many reasons, with some central goals:

- quality reference code (the code should be clean and easy to read; where it makes sense, it should remain possible to clone it locally and use it in tweaked form)
- optimal resilience and performance (getting performance right can add huge value for some systems, i.e., making it prettier but disrupting the performance would be bad)
- this code underpins non-trivial production systems (having good tests is not optional for reasons far deeper than having impressive coverage stats)

We'll do our best to be accomodating to PRs and issues, but please appreciate that [we emphasize decisiveness for the greater good of the project and its users](https://www.hanselman.com/blog/AlwaysBeClosingPullRequests.aspx); _new features [start with -100 points](https://blogs.msdn.microsoft.com/oldnewthing/20090928-00/?p=16573)_.

Within those constraints, contributions of all kinds are welcome:

- raising [Issues](https://github.com/jet/propulsion/issues) is always welcome (but we'll aim to be decisive in the interests of keeping the list navigable).
- bugfixes with good test coverage are always welcome; in general we'll seek to move them to NuGet prerelease and then NuGet release packages with relatively short timelines (there's unfortunately not presently a MyGet feed for CI packages rigged).
- improvements / tweaks, _subject to filing a GitHub issue outlining the work first to see if it fits a general enough need to warrant adding code to the implementation and to make sure work is not wasted or duplicated_

## TEMPLATES

The best place to start, sample-wise is with the [QuickStart](#quickstart), which walks you through sample code, tuned for approachability, from `dotnet new` templates stored [in a dedicated repo](https://github.com/jet/dotnet-templates).

## BUILDING

Please note the [QuickStart](#quickstart) is probably the best way to gain an overview, and the templates are the best way to see how to consume it; these instructions are intended mainly for people looking to make changes.

NB The `Propulsion.Kafka.Integration` tests are reliant on a `TEST_KAFKA_BROKER` environment variable pointing to a Broker that has been configured to auto-create ephemeral Kafka Topics as required by the tests (each test run writes to a guid-named topic)

### build, including tests on net461 and netcoreapp2.1

```powershell
dotnet build build.proj -v n
```

## FAQ

### What's the deal with the history of this repo?

This repo is derived from [`Jet.ConfluentKafka.FSharp`](https://github.com/jet/Jet.ConfluentKafka.FSharp); the history has been edited to focus only on edits to the `Propulsion` libraries.

### Your question here

- Please feel free to log question-issues; they'll get answered here
2 changes: 0 additions & 2 deletions build.proj
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
</PropertyGroup>

<Target Name="Pack">
<Exec Command="dotnet pack src/Jet.ConfluentKafka.FSharp $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Propulsion $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Propulsion.Cosmos $(Cfg) $(PackOptions)" />
<Exec Command="dotnet pack src/Propulsion.EventStore $(Cfg) $(PackOptions)" />
Expand All @@ -24,7 +23,6 @@

<Target Name="VSTest">
<Exec Command="dotnet test tests/Propulsion.Tests $(Cfg) $(TestOptions)" />
<Exec Command="dotnet test tests/Jet.ConfluentKafka.FSharp.Integration $(Cfg) $(TestOptions)" />
<Exec Command="dotnet test tests/Propulsion.Kafka.Integration $(Cfg) $(TestOptions)" />
</Target>

Expand Down
1 change: 0 additions & 1 deletion src/Propulsion.Kafka/Consumers.fs
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,6 @@ type ConsumerPipeline private (inner : IConsumer<string, string>, task : Task<un
let task = Async.StartAsTask machine
new ConsumerPipeline(consumer, task, triggerStop)

[<AbstractClass; Sealed>]
type ParallelConsumer private () =
/// Builds a processing pipeline per the `config` running up to `dop` instances of `handle` concurrently to maximize global throughput across partitions.
/// Processor pumps until `handle` yields a `Choice2Of2` or `Stop()` is requested.
Expand Down
3 changes: 1 addition & 2 deletions src/Propulsion.Kafka/Propulsion.Kafka.fsproj
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,10 @@
<PackageReference Include="FSharp.Core" Version="3.1.2.5" Condition=" '$(TargetFramework)' == 'net461' " />
<PackageReference Include="FSharp.Core" Version="4.3.4" Condition=" '$(TargetFramework)' != 'net461' " />

<PackageReference Include="Serilog" Version="2.7.1" />
<PackageReference Include="Jet.ConfluentKafka.FSharp" Version="1.0.1-rc1" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\Jet.ConfluentKafka.FSharp\Jet.ConfluentKafka.FSharp.fsproj" />
<ProjectReference Include="..\Propulsion\Propulsion.fsproj" />
</ItemGroup>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\..\src\Jet.ConfluentKafka.FSharp\Jet.ConfluentKafka.FSharp.fsproj" />
<ProjectReference Include="..\..\src\Propulsion.Kafka\Propulsion.Kafka.fsproj" />
</ItemGroup>

Expand Down

0 comments on commit a43c234

Please sign in to comment.