[Part 3|*] Implement Second Part for SQL Backend functions #10291

ziggie1984 · 2025-10-13T19:12:26Z

Depends on #10287

This PR implements most of the Insert and Delete methods we need for the unit tests in the following PR.

ziggie1984 · 2025-10-17T07:54:25Z

/gemini review

gemini-code-assist

Code Review

This pull request implements a significant portion of the insert and delete methods for the new SQL backend for the payments database. The changes are extensive and include new SQL queries, data conversion functions, and the implementation of several methods on the SQLStore.

The use of ExecuteBatchQuery and ExecutePaginatedQuery is great for performance. The new PaymentAndIntent interface is a good abstraction.

I've found a few areas for improvement:

Adherence to the function commenting style guide for exported functions.
Consistent use of context.Context propagation instead of context.TODO().
Opportunities for performance improvements by using batch inserts and optimizing queries to avoid extra fetches.
A potential code smell related to handling of nil slices from the database.

Overall, this is a solid step forward in the SQL backend implementation. Addressing these points will improve the code's maintainability, robustness, and performance.

payments/db/sql_store.go

gemini-code-assist · 2025-10-17T07:56:20Z

payments/db/sql_store.go

+				// TODO(ziggie): Can we get rid of this?
+				// This has to be in place otherwise the
+				// comparison will not match.
+				if record.Value == nil {
+					record.Value = []byte{}
+				}


The TODO here highlights a potential code smell. This workaround of converting a nil slice to an empty slice might be hiding an issue or could be simplified. It would be beneficial to investigate whether the downstream code that consumes these records can be made to handle nil slices gracefully. This would remove the need for this conversion and make the data handling more consistent.

payments/db/sql_store.go

ellemouton · 2025-10-24T08:18:00Z

payments/db/sql_store.go

+		completePayment, err := s.fetchPaymentWithCompleteData(
+			ctx, db, fetchPayment,
+		)
+		if err != nil {


i dont think we need to fetch the complete payment here?

yeah looks like overkill to get all the data, I think we only need completePayment.Status?

unfortunately we have to because it depends on the state of the HTLCs, so we need to always fetch the whole payment

se also func (m *MPPayment) setState() error {

payments/db/sql_store.go

yyforyongyu

Cool and would appreciate if the CIs can pass first before requesting reviews, especially the linter.

yyforyongyu · 2025-10-28T12:35:21Z

payments/db/sql_store.go

+		completePayment, err := s.fetchPaymentWithCompleteData(
+			ctx, db, fetchPayment,
+		)
+		if err != nil {


yeah looks like overkill to get all the data, I think we only need completePayment.Status?

payments/db/sql_store.go

sqldb/sqlc/queries/payments.sql

payments/db/sql_store.go

sqldb/sqlc/queries/payments.sql

payments/db/sql_store.go

lightninglabs-deploy · 2025-11-04T13:12:48Z

@ziggie1984, remember to re-request review from reviewers when ready

ziggie1984

Addressed all the feedback, I think the batch inserting I will do in a followup PR.

payments/db/sql_store.go

ziggie1984 · 2025-11-10T19:38:16Z

payments/db/sql_store.go

+		completePayment, err := s.fetchPaymentWithCompleteData(
+			ctx, db, fetchPayment,
+		)
+		if err != nil {


unfortunately we have to because it depends on the state of the HTLCs, so we need to always fetch the whole payment

se also func (m *MPPayment) setState() error {

payments/db/sql_store.go

ziggie1984 · 2025-11-10T19:48:51Z

payments/db/sql_store.go

+			completePayment, err := s.fetchPaymentWithCompleteData(
+				ctx, db, existingPayment,
+			)


same as above we need to make sure no HTLCs are INFLIGHT and none exist or all are failed basically.

payments/db/sql_store.go

sqldb/sqlc/queries/payments.sql

payments/db/sql_store.go

ziggie1984 · 2025-11-11T10:49:44Z

The itest for native_sql backends will not pass in this PR as well, they will start to pass from the next PR one (Part 4)

ellemouton

looking good!

sqldb/sqlc/queries/payments.sql

ellemouton · 2025-11-19T09:20:06Z

payments/db/sql_store.go

 		return nil
 	}

+	ctx := context.TODO()


keep at top of file?

nit: guess I don't understand this change - why moving this ctx definition?

I am rebasing all these PRs on top of the complete serious and sometimes I think because later in the series these context get dropped and from some reason some merge conflict probably led to this.

ellemouton · 2025-11-19T09:54:00Z

payments/db/sql_store.go

+	//
+	// TODO(ziggie): Refactor to not mix application logic with database
+	// logic. This decision should be made in the application layer.
+	if s.keepFailedPaymentAttempts {


i think what YY is saying it that this is changing the behaviour as it currently stands. For KVStore: if DeletePayment is called from (r *rpcServer) DeletePayment, there is no check on keepFailedPaymentAttempts since it calls (p *KVStore) DeletePayment which does not check that. So if we want to keep the behaviour the same for (s *SQLStore) DeletePayment, then we should not check it there either.

sqldb/sqlc/queries/payments.sql

ellemouton · 2025-11-19T09:58:50Z

payments/db/sql_store.go

+			FirstHopCustomRecords
+
+		for key, value := range firstHopCustomRecords {
+			err = db.InsertPaymentFirstHopCustomRecord(


yeah i think there isnt right now an easy way to do bulk inserts

payments/db/payment.go

payments/db/errors.go

ellemouton · 2025-11-19T10:07:48Z

sqldb/sqlc/migrations/000009_payments.up.sql

+    CONSTRAINT idx_payments_payment_identifier_unique
    UNIQUE (payment_identifier)


do we not want to instead name these?

I think they are named and the name in this case is idx_payments_payment_identifier_unique , Can you confirm @gemini-code-assist

Yes, you are correct. The UNIQUE constraint is explicitly named idx_payments_payment_identifier_unique using the CONSTRAINT keyword. This is the standard way to provide a name for a constraint in SQL.

yyforyongyu

Getting close!

yyforyongyu · 2025-11-19T11:55:38Z

sqldb/sqlc/migrations/000009_payments.up.sql

 -- ─────────────────────────────────────────────

-- ─────────────────────────────────────────────
-- Payment Intents Table


nit: for future commits, it'd be easier for the reviewers if we don't move the code yet add some changes in one commit, which can be challenging for the reviewers to do a line-by-line comparison.

payments/db/sql_store.go

sqldb/sqlc/queries/payments.sql

yyforyongyu · 2025-11-19T12:03:37Z

payments/db/sql_store.go

 		return nil
 	}

+	ctx := context.TODO()


nit: guess I don't understand this change - why moving this ctx definition?

yyforyongyu · 2025-11-19T12:08:10Z

payments/db/sql_store.go

+	//
+	// TODO(ziggie): Refactor to not mix application logic with database
+	// logic. This decision should be made in the application layer.
+	if s.keepFailedPaymentAttempts {


Yeah keepFailedPaymentAttempts is a config value, which means the users want to delete the failed attempts automatically after a payment is settled, this shouldn't affect how the DeletePayment works. However with the current change it will introduce a bug.

yyforyongyu · 2025-11-19T12:45:12Z

payments/db/sql_store.go

+			return fmt.Errorf("failed to fetch payment: %w", err)
+		}
+
+		mpPayment, err = s.fetchPaymentWithCompleteData(


Are there upcoming performance tests?

yyforyongyu · 2025-11-19T12:50:13Z

payments/db/sql_store.go

+
+		// Add the attempt to the payment without fetching it from the
+		// DB again.
+		mpPayment.HTLCs = append(mpPayment.HTLCs, HTLCAttempt{


Designing a safe DB layer is what we are trying to do here, yet we delegate that responsibility to the callers. The most proper way to handle this is via SELECT FOR UPDATE, but unfortunately due to our multi-sql schema support, this cannot be easily done at the moment. We sort of rely on the RegisterAttempt implemented in ControlTower, which holds a lock before calling this method, more like a mediation than solution. And here we already know this in-memory append will create a split brain issue with the data on disk, so I don't understand why we stick to this approach? @gemini-code-assist

If we are to go with this approach, we should document in details about the possible race, and how the mpPayment.HTLCs can be different from what's stored in db.

In this commit we add all queries which we will need to insert payment related data into the db.

ziggie1984

Thank you both for the thorough reviews 🙏

ziggie1984 · 2025-11-19T20:13:58Z

sqldb/sqlc/migrations/000009_payments.up.sql

+    CONSTRAINT idx_payments_payment_identifier_unique
    UNIQUE (payment_identifier)


I think they are named and the name in this case is idx_payments_payment_identifier_unique , Can you confirm @gemini-code-assist

sqldb/sqlc/queries/payments.sql

ziggie1984 · 2025-11-19T22:03:33Z

payments/db/sql_store.go

 		return nil
 	}

+	ctx := context.TODO()


I am rebasing all these PRs on top of the complete serious and sometimes I think because later in the series these context get dropped and from some reason some merge conflict probably led to this.

ziggie1984 · 2025-11-19T22:08:13Z

payments/db/sql_store.go

 		return nil
 	}

+	ctx := context.TODO()


ziggie1984 · 2025-11-19T22:09:17Z

payments/db/sql_store.go

+	//
+	// TODO(ziggie): Refactor to not mix application logic with database
+	// logic. This decision should be made in the application layer.
+	if s.keepFailedPaymentAttempts {


now I see it, weird that I had it in. Fixed. Nice catch!

ziggie1984 · 2025-11-19T22:18:08Z

payments/db/sql_store.go

+			return fmt.Errorf("failed to fetch payment: %w", err)
+		}
+
+		mpPayment, err = s.fetchPaymentWithCompleteData(


yes after migration we will focus on proper performance testing

ziggie1984 · 2025-11-19T22:25:32Z

payments/db/sql_store.go

+
+		// Add the attempt to the payment without fetching it from the
+		// DB again.
+		mpPayment.HTLCs = append(mpPayment.HTLCs, HTLCAttempt{


let's fetch the whole payment instead of doing the HTLC append in memory logic.

ziggie1984 · 2025-11-19T22:29:18Z

payments/db/sql_store.go

+
+		// Add the attempt to the payment without fetching it from the
+		// DB again.
+		mpPayment.HTLCs = append(mpPayment.HTLCs, HTLCAttempt{


nice catch and inputs 🙏

payments/db/payment.go

yyforyongyu

LGTM💟

yyforyongyu · 2025-11-20T08:17:57Z

payments/db/sql_store.go

+
+		// Insert blinded route data if present. Every hop in the
+		// blinded path must have an encrypted data record.
+		if hop.EncryptedData != nil {


weird...I remembered this was changed before, but now changed back?

ellemouton

✅

ziggie1984 changed the base branch from master to elle-payment-sql-series-new October 13, 2025 19:13

ziggie1984 self-assigned this Oct 13, 2025

ziggie1984 added payments Related to invoices/payments sql labels Oct 13, 2025

ziggie1984 added this to v0.21 Oct 13, 2025

ziggie1984 added this to the v0.21.0 milestone Oct 13, 2025

saubyk moved this to In progress in v0.21 Oct 14, 2025

ziggie1984 changed the title ~~[Part 3|*] Implement FetchPayment for SQL Backend~~ [Part 3|*] Implement Second Part for SQL Backend functions Oct 15, 2025

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch 2 times, most recently from 80de0dd to c0134ed Compare October 15, 2025 15:12

ziggie1984 marked this pull request as ready for review October 15, 2025 15:13

ziggie1984 mentioned this pull request Oct 15, 2025

[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

Merged

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch 3 times, most recently from f0e3302 to 99ddf07 Compare October 17, 2025 07:53

gemini-code-assist bot reviewed Oct 17, 2025

View reviewed changes

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from 99ddf07 to 6145682 Compare October 17, 2025 08:43

ziggie1984 requested review from ellemouton and yyforyongyu October 17, 2025 08:50

ziggie1984 mentioned this pull request Oct 17, 2025

[feature]: Introduce Payment SQL storage #9863

Open

8 tasks

ziggie1984 moved this from In progress to In review in v0.21 Oct 18, 2025

ellemouton reviewed Oct 24, 2025

View reviewed changes

yyforyongyu requested changes Oct 28, 2025

View reviewed changes

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from 6145682 to 047e180 Compare November 10, 2025 17:56

ziggie1984 moved this from In review to In progress in v0.21 Nov 10, 2025

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from 047e180 to 5f2b14a Compare November 11, 2025 10:29

ziggie1984 commented Nov 11, 2025

View reviewed changes

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from 8d382f4 to 9fd7eba Compare November 13, 2025 10:31

ziggie1984 requested review from ellemouton and yyforyongyu November 13, 2025 13:44

ziggie1984 mentioned this pull request Nov 13, 2025

[Part 3b|*] Implement Third Part for SQL Backend functions #10368

Merged

ellemouton reviewed Nov 19, 2025

View reviewed changes

yyforyongyu requested changes Nov 19, 2025

View reviewed changes

ziggie1984 moved this from In review to In progress in v0.21 Nov 19, 2025

ziggie1984 added 7 commits November 19, 2025 23:01

sqldb: add queries for deleting a payment and attempts

1f9017c

paymentsdb: add query to only fetch resolution type for HTLCs

179506a

paymentsdb: implement DeleteFailedAttempts for sql backend

be9ac28

paymentsdb: implement DeletePayment for sql backend

a6610ad

sqldb+paymentsdb: add queries to insert all relavant data

ce121fb

In this commit we add all queries which we will need to insert payment related data into the db.

paymentsdb: implement InitPayment for sql backend

07467e9

paymentsdb: add note to RegisterAttempt

56de565

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from 9fd7eba to dbcd822 Compare November 19, 2025 22:39

ziggie1984 commented Nov 19, 2025

View reviewed changes

ziggie1984 requested review from ellemouton and yyforyongyu November 19, 2025 23:48

ziggie1984 moved this from In progress to In review in v0.21 Nov 19, 2025

yyforyongyu approved these changes Nov 20, 2025

View reviewed changes

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from dbcd822 to dc99a77 Compare November 20, 2025 08:53

ziggie1984 added 4 commits November 20, 2025 09:57

paymentsdb: implement RegisterAttempt for sql backend

da2a8a0

paymentsdb: verify total amount for last hop in the blinded path

005031d

paymentsdb: implement SettleAttempt for sql backend

6422190

docs: add release-notes

1101cc3

ziggie1984 force-pushed the introduce-sql-schema-payments-part-3 branch from dc99a77 to 1101cc3 Compare November 20, 2025 09:00

ellemouton approved these changes Nov 20, 2025

View reviewed changes

ziggie1984 merged commit a4756cd into lightningnetwork:elle-payment-sql-series-new Nov 20, 2025
33 of 38 checks passed

github-project-automation bot moved this from In review to Done in v0.21 Nov 20, 2025

		CONSTRAINT idx_payments_payment_identifier_unique
		UNIQUE (payment_identifier)

[Part 3|*] Implement Second Part for SQL Backend functions #10291

[Part 3|*] Implement Second Part for SQL Backend functions #10291

Uh oh!

Conversation

ziggie1984 commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziggie1984 commented Oct 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

gemini-code-assist bot Oct 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yyforyongyu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lightninglabs-deploy commented Nov 4, 2025

Uh oh!

ziggie1984 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ziggie1984 commented Nov 11, 2025

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ziggie1984 commented Oct 13, 2025 •

edited

Loading