[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

ziggie1984 · 2025-10-14T09:34:08Z

This PR adds the DB related unit tests for backend functions which were introduced in #10287,#10291 and #10368
and finalizes the implementation for the SQL backend.

This PR now lets the unit tests and the itest pass for the SQL payments backend.

ziggie1984 · 2025-10-17T07:54:34Z

/gemini review

gemini-code-assist

Code Review

This pull request adds comprehensive unit tests for the newly introduced SQL backend for the payments database. The changes include refactoring existing tests to be database-agnostic, which is a great improvement for maintainability. The new SQL queries and the Go store implementation are well-structured and cover a wide range of functionalities. My main feedback is on adhering to the testing style guide in the new TestQueryPayments function, where t.Fatalf and t.Errorf should be replaced with assertions from the require library as mandated by the repository's style guide.

gemini-code-assist · 2025-10-17T07:56:49Z

payments/db/payment_test.go

+				if err != nil {
+					t.Fatalf("unable to create test "+
+						"payment: %v", err)
+				}


The project's style guide specifies that unit tests must use the require library for assertions.¹ Please replace if err != nil { t.Fatalf(...) } with require.NoError(t, err, ...) for consistency and conciseness. This also applies to other t.Fatalf and t.Errorf calls in this test function, which should be replaced with appropriate require assertions (e.g., require.Equal, require.Len).

require.NoError(t, err, "unable to create test payment")

Style Guide References

Footnotes

The style guide mandates the use of the require library for assertions in unit tests to ensure consistency and readability. ↩

ziggie1984

I think Part 4 is also ready to land 🫡

ziggie1984 · 2025-11-21T17:54:39Z

payments/db/payment_test.go

+func genInfo(t *testing.T) (*PaymentCreationInfo, lntypes.Preimage, error) {
+
+	preimage, _, err := genPreimageAndHash(t)
+	if err != nil {


this will be done in Part 7 to avoid rebase collisions because of the whole stacked up series of payment PRs

yyforyongyu

Q: I just realized these flags are not added to the make file, and these are not running in the CI?

I ran this locally to get an understanding of the test coverage,

go test -v -tags=test_db_sqlite -coverprofile=cover.out ./payments/db && go tool cover -func=cover.out | grep "sql_store.go"

ok      github.com/lightningnetwork/lnd/payments/db     8.059s  coverage: 33.7% of statements
github.com/lightningnetwork/lnd/payments/db/sql_store.go:119:                   NewSQLStore                             83.3%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:146:                   fetchPaymentWithCompleteData            80.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:172:                   batchLoadPayments                       0.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:204:                   batchLoadPaymentsCoreData               0.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:269:                   batchLoadPaymentCustomRecords           100.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:303:                   batchLoadHtlcAttempts                   100.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:336:                   batchLoadHopsForAttempts                100.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:370:                   batchLoadHopCustomRecords               100.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:404:                   batchLoadRouteCustomRecords             42.9%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:443:                   batchLoadPaymentResolutions             81.8%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:491:                   loadPaymentResolutions                  75.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:508:                   computePaymentStatusFromResolutions     100.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:550:                   batchLoadPaymentsRelatedData            72.7%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:630:                   buildPaymentFromBatchData               92.3%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:729:                   QueryPayments                           92.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:901:                   fetchPaymentByHash                      83.3%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:925:                   FetchPayment                            92.3%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:975:                   FetchInFlightPayments                   0.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1111:                  DeleteFailedAttempts                    93.8%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1157:                  computePaymentStatusFromDB              75.0%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1209:                  DeletePayment                           87.5%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1272:                  InitPayment                             79.3%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1381:                  insertRouteHops                         46.9%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1505:                  RegisterAttempt                         76.5%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1632:                  SettleAttempt                           81.8%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1707:                  FailAttempt                             66.7%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1796:                  Fail                                    81.8%
github.com/lightningnetwork/lnd/payments/db/sql_store.go:1881:                  DeletePayments                          91.2%

looks like FetchInFlightPayments is not tested.

ziggie1984 · 2025-11-24T10:05:52Z

Ohh did not see that one, will add a proper unit test for inflight payments.

We do run the CI for the tests however:

we do not check the coverage however for the different backends

Now that every method of the interface was implemented we can remove the embedded reference we put into place for the sql store implementation so that the interface would succeed. This is now removed.

Since now the sql backend is more strict in using the same session key we refactor the helper so that we can easily change the session key for every new attempt.

We make the QueryPayments test db agnostic and also keep a small test for querying the duplicate payments case in the kv world.

We are now not supporting the LegacyPayload for the onion packet anymore. All payments and their onion payload need to be tlv encoded. The sql backend assumes tlv so we have to always set the in memory presentation of a hop where the legacy parameter is still available but deprecated to false, otherwise the hops will not be equal and unit tests for the sql backend will fail when switched on in the next commits.

In commit adds the harness which will be used to run db agnostic tests against the kv and sql backend. We have adopted all the unit tests so far so that with this commit all the payment tests not specifically put into the kv_store_test.go should all pass for all backends.

The design of the sql and kv db are a bit different. A harness interface is introduced which allows us to unit most of the test and keep the backend specific tests at a minimum.

ziggie1984 · 2025-11-24T19:19:32Z

Will add some additional unit test in PR 7, because I already refactored the helper methods as proposed by yy in this PR.

ziggie1984 · 2025-11-24T19:37:41Z

Final coverage when all tests are being merged see PR 7:

#10373 (comment)

yyforyongyu

LGTM

ellemouton

lgtm!

ziggie1984 changed the base branch from master to elle-payment-sql-series-new October 14, 2025 09:34

saubyk assigned ziggie1984 Oct 14, 2025

saubyk added this to v0.21 Oct 14, 2025

saubyk moved this to In progress in v0.21 Oct 14, 2025

ziggie1984 changed the title ~~[Part 4|*] Implement FetchInFlightPayments for SQL Backend~~ [Part 4|*] Add unit tests for the previous introduced SQL Backend methods Oct 15, 2025

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch 3 times, most recently from 415144d to 439b619 Compare October 15, 2025 16:28

ziggie1984 marked this pull request as ready for review October 15, 2025 16:29

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch from 439b619 to c922bc3 Compare October 15, 2025 17:02

ziggie1984 added this to the v0.21.0 milestone Oct 15, 2025

ziggie1984 added payments Related to invoices/payments sql labels Oct 15, 2025

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch 7 times, most recently from 3648e20 to ae6c366 Compare October 17, 2025 07:52

gemini-code-assist bot reviewed Oct 17, 2025

View reviewed changes

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch from ae6c366 to b011ea6 Compare October 17, 2025 08:43

ziggie1984 requested review from ellemouton and yyforyongyu October 17, 2025 08:50

ziggie1984 mentioned this pull request Oct 17, 2025

[feature]: Introduce Payment SQL storage #9863

Open

8 tasks

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch 2 times, most recently from b515aaa to bdbcc54 Compare October 17, 2025 11:12

ziggie1984 moved this from In progress to In review in v0.21 Oct 18, 2025

ziggie1984 mentioned this pull request Nov 10, 2025

[Part 2|*] Implement First Part for SQL Backend functions #10287

Merged

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch from c486d3d to 363b01b Compare November 21, 2025 17:57

ziggie1984 commented Nov 21, 2025

View reviewed changes

ziggie1984 requested review from ellemouton and yyforyongyu November 22, 2025 09:56

yyforyongyu reviewed Nov 24, 2025

View reviewed changes

ziggie1984 moved this from In review to In progress in v0.21 Nov 24, 2025

ziggie1984 added 11 commits November 24, 2025 19:44

paymentsdb: remove kvstore from sql db implementation

5a30446

Now that every method of the interface was implemented we can remove the embedded reference we put into place for the sql store implementation so that the interface would succeed. This is now removed.

paymentsdb: refactor test helpers

7b504ca

Since now the sql backend is more strict in using the same session key we refactor the helper so that we can easily change the session key for every new attempt.

paymentsdb: make QueryPayments test db agnostic

967f028

We make the QueryPayments test db agnostic and also keep a small test for querying the duplicate payments case in the kv world.

paymentsdb: introduce a harness interface

e754aac

The design of the sql and kv db are a bit different. A harness interface is introduced which allows us to unit most of the test and keep the backend specific tests at a minimum.

paymentsdb: make specific kv store tests only available via build tag

2700dce

itest: fix list_payments accuracy edge case

1fb7c60

lnrpc: fix linter

59edd24

paymentsdb: add more comments

339c001

paymentsdb: add firstcustom records to unit tests

5c5856b

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch from 363b01b to d7d635c Compare November 24, 2025 19:08

ziggie1984 moved this from In progress to In review in v0.21 Nov 24, 2025

ziggie1984 added 2 commits November 24, 2025 20:16

paymentsdb: add unit test for FetchInflightPayments method

577990c

docs: add release-notes

cfc0dc0

ziggie1984 force-pushed the introduce-sql-schema-payments-part-4 branch from d7d635c to cfc0dc0 Compare November 24, 2025 19:17

ziggie1984 requested a review from yyforyongyu November 24, 2025 20:38

yyforyongyu approved these changes Nov 25, 2025

View reviewed changes

ellemouton approved these changes Nov 25, 2025

View reviewed changes

ziggie1984 merged commit 4a6c7ab into lightningnetwork:elle-payment-sql-series-new Nov 25, 2025
39 checks passed

github-project-automation bot moved this from In review to Done in v0.21 Nov 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

Uh oh!

ziggie1984 commented Oct 14, 2025 •

edited

Loading

Uh oh!

ziggie1984 commented Oct 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Oct 17, 2025

Uh oh!

ziggie1984 left a comment

Uh oh!

ziggie1984 Nov 21, 2025

Uh oh!

yyforyongyu left a comment

Uh oh!

ziggie1984 commented Nov 24, 2025 •

edited

Loading

Uh oh!

ziggie1984 commented Nov 24, 2025

Uh oh!

ziggie1984 commented Nov 24, 2025

Uh oh!

yyforyongyu left a comment

Uh oh!

ellemouton left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

[Part 4|*] Add unit tests for the previous introduced SQL Backend methods #10292

Uh oh!

Conversation

ziggie1984 commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziggie1984 commented Oct 17, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Oct 17, 2025

Choose a reason for hiding this comment

Style Guide References

Footnotes

Uh oh!

ziggie1984 left a comment

Choose a reason for hiding this comment

Uh oh!

ziggie1984 Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

yyforyongyu left a comment

Choose a reason for hiding this comment

Uh oh!

ziggie1984 commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ziggie1984 commented Nov 24, 2025

Uh oh!

ziggie1984 commented Nov 24, 2025

Uh oh!

yyforyongyu left a comment

Choose a reason for hiding this comment

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ziggie1984 commented Oct 14, 2025 •

edited

Loading

ziggie1984 commented Nov 24, 2025 •

edited

Loading