Feat/trades and bitstamp #606

jdupas22 · 2025-11-28T10:32:54Z

Summary by cubic

Add first-class Trades support across the API and engine, and introduce a Bitstamp connector with trade ingestion. Also adds an initial Coinbase Prime connector with accounts, balances, payments, and wallet-to-wallet transfers.

New Features
- Trade domain models and validation (status, side, order type, fees, liquidity).
- New v3 endpoints: POST /trades, GET /trades, GET /trades/{tradeID}, PATCH /trades/{tradeID}/metadata.
- Engine workflows/activities to fetch trades from connectors, upsert to storage, and publish trade events.
- Bitstamp connector: accounts, balances, payments, and trades ingestion with capabilities registered.
- Coinbase Prime connector: accounts, balances, payments, and wallet-to-wallet transfers using the Prime SDK.
- Dummypay: trades fetch added; capabilities updated to allow Formance trade creation.
Dependencies
- Added github.com/coinbase-samples/prime-sdk-go v0.5.3 and github.com/shopspring/decimal v1.4.0.
- Updated go.uber.org/mock to v0.4.0 and regenerated mocks.
- docker-compose: set GOCACHE and GOMODCACHE for gateway and workers to speed local builds.

^{Written for commit 789222e. Summary will update automatically on new commits.}

feat: adding a connector development server with direct access to connectors functions Signed-off-by: Clément Salaün <[email protected]>

Signed-off-by: Clément Salaün <[email protected]>

* Handle SQLSTATE 22P02 in storage as HTTP 400 * Disable profiler in e2e tests to remove unneeded overhead

…ling - Added detailed logging for account and balance fetching processes. - Refactored payment transaction handling to support multiple payment types from a single transaction. - Updated transaction struct to dynamically capture exchange rates. - Cleaned up unused comments and improved code readability across various files.

…validation for trade legs

…ors and tests

…ed capabilities

…ndling - Introduced FetchNextTrades method in the Bitstamp plugin to retrieve trades. - Updated fetchNextTrades to handle unique trades and filter out duplicates. - Modified transactionToTrade to include account reference and adjusted logging. - Removed legacy payment handling for exchange transactions, now managed by FetchTrades. - Added new capability for fetching trades in the Bitstamp capabilities list.

coderabbitai · 2025-11-28T10:33:10Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch feat/trades-and-bitstamp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

laouji · 2025-11-28T12:32:14Z

internal/connectors/plugins/public/coinbaseprime/client/_poc.go

@@ -0,0 +1,185 @@
+package main


Do we still need this file for something?

laouji · 2025-11-28T12:38:33Z

internal/storage/trades.go

+			return e("failed to convert trade to model", err)
+		}
+
+		payload := internalEvents.TradeMessagePayload{


Does the consumer of the event need the entire payload? It might be an option to send fewer fields and allow the consumer to query payments if they want additional information.

Yes we could indeed! What would be the gain of doing that though?

Some queue systems have somewhat limited payload size so if you have big nested structures you might run into the upper limits and not be able to send the event at all.

But also, just in general I'd be interested to know what the business use-case of these notifications would be.
What value does it bring to clients to know about new trades? What will they typically do with that info?

Fabrice recently did a big refactor of the notifications to reduce the operational costs, so it should be cheaper for us to send notifications now, but I also wanted to check if you added this function to send notifications because there was a particular business use-case in mind, or if you just copied the pattern we had for payments?

Oh I see, thanks for the hindsight. I know some users like to create a kind of backup of what's happening in our system by using the events, so by default I made it this way. But please keep in mind that I'm not yet as familiar as I'd have wanted with payment, and again with the recent event refactor.

I would recommend limiting the payload to the most essential fields that consumers are likely to care about and if possible remove the nested rawJson so that we have better control over the payload size and contents.

It's pretty common practice that notifications such as these don't contain the entire object, but rather only enough information so that the consumer of the payload can make a business decision about the entity. If they want a full backup they can fetch the full-object from our API.

Another thing to consider, particularly when the fields are being populated from an external source, we don't have control over what kind of private data is in the payload, so limiting duplication of that data within our system is typically better for security and privacy.

laouji · 2025-11-28T12:52:50Z

internal/connectors/engine/engine.go

 	CreateFormancePayment(ctx context.Context, payment models.Payment) error
+	// Create a Formance trade, no call to the plugin, just a creation
+	// of a trade in the database related to the provided connector id.
+	CreateFormanceTrade(ctx context.Context, trade models.Trade) error


What is the purpose of creating a trade on Formance only? If it doesn't connect to the PSP then what is it used for?

For payments we have a create payment endpoint which is used primarily by people using the generic connector that need a way to backfill data that for one reason or another their endpoint won't return via the connector.

But we have two separate entities:

payment_initiation (created via formance API)

payment (imported via the connector)

the former is the intent to create a payment, whereas the latter is a representation of something that exists on the PSP. The payment_initiation can be forwarded to the PSP - this will result in a corresponding payment being linked.

Here you have a mixed trade object that may or may not represent a real trade known by the PSP. You also allow (local) updates of the trade metadata which will never be reflected on the PSP.
Would it not be better to keep two different entities that can be linked, like we do for payments?

I see what you mean, I did not have that nuance when I built the object.
I know that some payment providers will be able to create trades objects directly, but some others (like bitstamp) treat all transactions (trades, wallet top-up, payouts, etc) as simple transactions, and in this case we create the trade object ourselves. Do you think it would make more sense to have both objects for both use-cases?

Maybe? I think you'd have to walk me through the two different use-cases for me to be sure.

In the case of the generic connector, it is usually connected to the client's own API so they have a lot of control in terms of adding the correct metadata to link an object that exists in their system already and a new one they create in payments.

In the case where the PSP is a 3rd party like Bitstamp that is completely unaware of Formance, there will be no real way to link together the entity on the PSP with the one they created on Formance without them managing their own mapping.

For Bitstamp is there a likelihood that their API won't return historical information and therefore they need to backfill a bunch of data that we otherwise won't be able to get from their API?

laouji · 2025-11-28T12:59:01Z

internal/connectors/engine/workflow/fetch_trades.go

+		}
+
+		if len(nextTasks) > 0 {
+			// Logic for next tasks if needed


You're probably not seeing any issues with this remaining unimplemented because your connector put fetch trades as the last element in the plugin workflow. Other connectors might not have trades being the last element, which will result in the rest of the workflow never executing. Let's handle this properly.

laouji · 2025-11-28T13:03:57Z

internal/connectors/engine/activities/send_events.go

+	// Safely get logger - it may be nil in unit tests
+	var logger interface{ Info(string, ...interface{}); Error(string, ...interface{}) }
+	// Try to get activity logger, but don't panic if not in activity context
+	func() {
+		defer func() {
+			if r := recover(); r != nil {
+				// Not an activity context, logger stays nil
+				logger = nil
+			}
+		}()
+		logger = activity.GetLogger(ctx)
+	}()
+
+	if logger != nil {
+		logger.Info("SendEvents activity started",
+			"idempotency_key", req.IdempotencyKey,
+			"has_payment", req.Payment != nil)
+	}
+


You can actually just use a.logger which can be set in unit tests as well. Then we don't need all these if statements checking if the logger is nil

laouji · 2025-11-28T13:23:41Z

internal/connectors/plugins/public/bitstamp/accounts.go

+	// TODO: accountsState will be used to know at what point we're at when
+	// fetching the PSP accounts.
+	// This struct will be stored as a raw json, you're free to put whatever
+	// you want.
+	// Example:
+	// LastPage int `json:"lastPage"`
+	// LastIDCreated int64 `json:"lastIDCreated"`


Pagination is unimplemented?

laouji · 2025-11-28T13:25:35Z

internal/connectors/plugins/public/bitstamp/client/balances.go

+
+	if resp.StatusCode != http.StatusOK {
+		// Try to unmarshal error response
+		body, _ := io.ReadAll(resp.Body)


Please handle the error.

laouji · 2025-11-28T13:36:56Z

internal/connectors/plugins/public/bitstamp/client/client.go

+func newNonce() string {
+	const charset = "abcdefghijklmnopqrstuvwxyz0123456789"
+	b := make([]byte, 36)
+	rand.Read(b)
+	for i := range b {
+		b[i] = charset[b[i]%byte(len(charset))]
+	}
+	return string(b)
+}


Since this is only pseudorandom, there might be collisions if a client has multiple payments pods running. Probably not a critical issue right now, but probably using the current time as part of the random generation would decrease chances of collision.

laouji · 2025-11-28T13:40:02Z

internal/models/trade_validator.go

+}
+
+// ExpectedLegAmounts calculates expected BASE and QUOTE payment amounts
+func ExpectedLegAmounts(trade Trade) (base decimal.Decimal, quote decimal.Decimal) {


Any reason you're using this particular decimal library instead of handling decimals the way we do for amounts in payments / balance?

I figured that decimal was the better library for the computation we had to do for asset conversion, but this is not necessarily a final choice, if you think it is not needed please let me know.

I think it would be better to stick to one middleware within the service for consistency / maintenance. I'm not particularly invested in one over the other, but if you have particular reasons for wanting to introduce a new library I'd like to hear them.
What limitations did you face?

The current approach for transaction amounts and balances is to persist a precision marker alongside an int value to avoid loss of precision.

Floating point numbers are generally not considered suitable for representing monetary values (see here), so often the best approach is not to use floating point values at all in the code.

Since the trade data is coming in via json, you can treat those as strings instead of as floating point values to avoid loss of precision when converting the json representation to Go (hence the use of big.Int to store cents).

If you have other types of decimals that represent something other than money you might have some additional considerations that aren't handled by our library, but I'd like to know about them if that's the case.

jdupas22 and others added 25 commits October 1, 2025 18:25

bistamp connector init commit

8edbb26

feat: added a connector dev

34a7526

feat: adding a connector development server with direct access to connectors functions Signed-off-by: Clément Salaün <[email protected]>

wip: scaffolding

88317ab

Signed-off-by: Clément Salaün <[email protected]>

feat: coinbase connector main capabilities

7ea2946

feat: add wallet-to-wallet transfers

f89128a

fix(API): Handle SQLSTATE 22P02 in storage as HTTP 400 (#556)

46a5bee

* Handle SQLSTATE 22P02 in storage as HTTP 400 * Disable profiler in e2e tests to remove unneeded overhead

payment list

fa397c3

fix payment list

d834d13

finished bitstamp

373017e

fix config

9228de1

openapi

c92cae5

first implemetation of trades object

221e692

fixed some tests for some reasons

ac48505

regenerated some mocks

e7f0b22

updated trades objects

26381cc

chore: add github.com/shopspring/decimal v1.4.0 dependency

8690c96

add openapi specs for trades

93a4b2b

feat(trades): implement trade creation with payment handling and add …

94fdc3c

…validation for trade legs

chore: update dependencies and regenerate generated files for connect…

6e42cf5

…ors and tests

feat(trades): add support for fetching trades and enhance trade-relat…

49afa6b

…ed capabilities

Merge feat/bitstamp into combined branch

0d1cba1

Merge remote-tracking branch 'origin/main' into feat/trades-and-bitstamp

7d8b615

refactor(tests): generated tests again

d47e1c3

laouji reviewed Nov 28, 2025

View reviewed changes

jdupas22 added 2 commits December 3, 2025 17:14

fixed first round of pr comments

63fef6d

fix tests

789222e

Feat/trades and bitstamp #606

Are you sure you want to change the base?

Feat/trades and bitstamp #606

Uh oh!

Conversation

jdupas22 commented Nov 28, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

coderabbitai bot commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jdupas22 commented Nov 28, 2025 •

edited by cubic-dev-ai bot

Loading

coderabbitai bot commented Nov 28, 2025 •

edited

Loading