[sql-44] firewalldb: add migration code for privacy mapper from kvdb to SQL #1092

ViktorTigerstrom · 2025-06-17T16:07:02Z

Based on the branch of #1079, to only show the relevant commits.

This PR introduces the migration logic for transitioning the privacy mapper from kvdb to SQL.

Note that as of this PR, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

I will rebase this on the new version of #1079, once the current feedback has been addressed after we agree on the approach there.

As this PR is heavily dependent on #1079, this is blocked on the dependent PR until it has been merged. This is therefore not ready for review.

Part of #917

ellemouton · 2025-06-18T10:30:46Z

@ViktorTigerstrom - feel free to push a base branch to upstream & then target that so that this pr can just contain the commits that are relevant

As the firewalldb migration will include more than just the migration of the kvstore data, we rename the migration tests that only migrate the kvstore data to make it clearer which tests only focus on migrating kv entries.

Currently, the migration tests for firewalldb only migrates the kv stores. In future commits, we will also migrate the privacy mapper and the actions in the firewalldb,. Before this commit, the expected results of the migrations tests could only be kv records, which will not be the case when we also migrate the privacy mapper and the actions. Therefore, we prepare the migration tests to expect more than just kv records. This commit introduces a new type of `expectedResult` type which the prep of the migration tests will use, which can specify more than just one type of expected result.

This commit introduces the migration logic for transitioning the privacy mapper store from kvdb to SQL. Note that as of this commit, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

ViktorTigerstrom · 2025-07-11T22:47:33Z

Rebased this on the latest version of #1079, as that PR feels good to go, and this PR is therefore close to not being blocked by that PR (and therefore ready to review).

ViktorTigerstrom · 2025-07-11T22:53:37Z

Requested a review by both of you @ellemouton & @bitromortac, even though #1079 isn't merged yet, just so you know that it is ready for review once that's been merged.

ellemouton

really really great work. Very clean 🔥

ellemouton · 2025-07-16T08:43:39Z

firewalldb/sql_migration_test.go

@@ -34,6 +34,11 @@ var (
 	testEntryValue = []byte{1, 2, 3}
 )

+// expectedResult represents the expected result of a migration test.
+type expectedResult struct {


very nice 🙏

ellemouton · 2025-07-16T08:45:45Z

firewalldb/sql_migration.go

+	privPairs, err := collectPrivacyPairs(ctx, kvStore, sqlTx)
+	if err != nil {
+		return fmt.Errorf("error migrating privacy mapper store: %w",
+			err)
+	}
+
+	// 2) Insert all collected privacy pairs into the SQL database.
+	err = insertPrivacyPairs(ctx, sqlTx, privPairs)
+	if err != nil {


we may want to tests the performance of this one as i think unlike the kv-store db, this one could be quite populated.

So for this one it might make sense to migrate on the fly instead of collecting everything in memory first.

ellemouton · 2025-07-16T08:49:10Z

firewalldb/sql_migration.go

+		_, err := sqlTx.GetPseudoForReal(
+			ctx, sqlc.GetPseudoForRealParams{
+				GroupID: groupID,
+				RealVal: realVal,
+			},
+		)
+		if err == nil {
+			return fmt.Errorf("duplicate privacy pair %s:%s: %w",
+				realVal, pseudoVal, ErrDuplicatePseudoValue)
+		} else if !errors.Is(err, sql.ErrNoRows) {
+			return err
+		}
+
+		_, err = sqlTx.GetRealForPseudo(
+			ctx, sqlc.GetRealForPseudoParams{
+				GroupID:   groupID,
+				PseudoVal: pseudoVal,
+			},
+		)
+		if err == nil {
+			return fmt.Errorf("duplicate privacy pair %s:%s: %w",
+				realVal, pseudoVal, ErrDuplicatePseudoValue)
+		} else if !errors.Is(err, sql.ErrNoRows) {
+			return err
+		}
+
+		err = sqlTx.InsertPrivacyPair(
+			ctx, sqlc.InsertPrivacyPairParams{
+				GroupID:   groupID,
+				RealVal:   realVal,
+				PseudoVal: pseudoVal,
+			},
+		)
+		if err != nil {


can we not potentially do this in one go with a single query that has a conflict rule?

ellemouton · 2025-07-16T08:51:19Z

firewalldb/sql_migration.go

+	if realBkt := bkt.Bucket(realToPseudoKey); realBkt != nil {
+		realToPseudoRes, err = collectPairs(realBkt)


just checking: do we have a test somewhere that makes sure that things dont fail on an empty kvdb?

ellemouton · 2025-07-16T08:53:25Z

firewalldb/sql_migration_test.go

 		kvEntries: fn.Some(insertedEntries),
+		// No privacy pairs are inserted in this test.
+		privPairs: fn.None[privacyPairs](),


we should have a test that includes both

bitromortac

Looks great! Only have a couple of nits 🚀

bitromortac · 2025-07-17T11:46:58Z

firewalldb/sql_migration_test.go

@@ -34,6 +34,11 @@ var (
 	testEntryValue = []byte{1, 2, 3}
 )

+// expectedResult represents the expected result of a migration test.
+type expectedResult struct {
+	kvEntries fn.Option[[]*kvEntry]


is it necessary to have an option type here, as an empty []*kvEntry already signals that, not? (also for the pair data)

bitromortac · 2025-07-17T11:49:12Z

firewalldb/sql_migration_test.go

@@ -213,6 +218,20 @@ func TestFirewallDBMigration(t *testing.T) {
 		}
 	}

+	// The assertMigrationResults function will currently assert that


nit: I'd rather not put comments of what will happen in the future, because it may add confusion for somebody who's not involved in the process and it also may never happen, or we forget to update the comment

bitromortac · 2025-07-17T11:53:56Z

firewalldb/sql_migration.go

@@ -87,10 +92,14 @@ func MigrateFirewallDBToSQL(ctx context.Context, kvStore *bbolt.DB,
 		return err


// Note that this migration currently only migrates the kvstores, but will be // extended in the future to also migrate the privacy mapper and action stores.

this is also an example of a "future" comment, where the comment wasn't updated

bitromortac · 2025-07-17T11:59:33Z

firewalldb/sql_migration.go

+}
+
+// collectPrivacyPairs collects all privacy pairs from the KV store and
+// returns them as the privacyPairs type alias.


nitty: and returns them as the privacyPairs type alias. I'd leave that away, as it's obvious from the function signature

bitromortac · 2025-07-17T12:18:50Z

firewalldb/sql_migration.go

+		_, err := sqlTx.GetPseudoForReal(
+			ctx, sqlc.GetPseudoForRealParams{
+				GroupID: groupID,
+				RealVal: realVal,
+			},
+		)
+		if err == nil {
+			return fmt.Errorf("duplicate privacy pair %s:%s: %w",
+				realVal, pseudoVal, ErrDuplicatePseudoValue)
+		} else if !errors.Is(err, sql.ErrNoRows) {
+			return err
+		}
+
+		_, err = sqlTx.GetRealForPseudo(
+			ctx, sqlc.GetRealForPseudoParams{
+				GroupID:   groupID,
+				PseudoVal: pseudoVal,
+			},
+		)
+		if err == nil {
+			return fmt.Errorf("duplicate privacy pair %s:%s: %w",
+				realVal, pseudoVal, ErrDuplicatePseudoValue)
+		} else if !errors.Is(err, sql.ErrNoRows) {
+			return err
+		}


are those checks necessary as they look to be covered by collectGroupPairs?

bitromortac · 2025-07-17T12:33:31Z

firewalldb/sql_migration_test.go

+
+// multiSessionsOnePrivPairs inserts 1 session with 1 privacy pair into the
+// boltDB.
+func oneSessionAndPrivPair(t *testing.T, ctx context.Context,


nit: we could use createPrivacyPairs directly

bitromortac · 2025-07-17T12:36:14Z

firewalldb/sql_migration_test.go

+			realKey := fmt.Sprintf("real-%d-%d", i, j)
+			pseudoKey := fmt.Sprintf("pseudo-%d-%d", i, j)


would it add anything if we'd let real/pseudo keys be the same for the sessions, to leave i away?

ViktorTigerstrom added the no-changelog This PR is does not require a release notes entry label Jun 17, 2025

ViktorTigerstrom self-assigned this Jun 17, 2025

ViktorTigerstrom added 3 commits July 12, 2025 00:42

firewalldb: rename kv stores migration tests

47a81e4

As the firewalldb migration will include more than just the migration of the kvstore data, we rename the migration tests that only migrate the kvstore data to make it clearer which tests only focus on migrating kv entries.

firewalldb: add privacy mapper SQL migration

9715313

This commit introduces the migration logic for transitioning the privacy mapper store from kvdb to SQL. Note that as of this commit, the migration is not yet triggered by any production code, i.e. only tests execute the migration logic.

ViktorTigerstrom force-pushed the 2025-06-migrate-privacy-mapper branch from bbe2b90 to 9715313 Compare July 11, 2025 22:45

ViktorTigerstrom changed the base branch from master to kvstores-migration-PR July 11, 2025 22:50

ViktorTigerstrom requested review from ellemouton and bitromortac July 11, 2025 22:52

ellemouton reviewed Jul 16, 2025

View reviewed changes

bitromortac reviewed Jul 17, 2025

View reviewed changes

		if realBkt := bkt.Bucket(realToPseudoKey); realBkt != nil {
		realToPseudoRes, err = collectPairs(realBkt)

		@@ -87,10 +92,14 @@ func MigrateFirewallDBToSQL(ctx context.Context, kvStore *bbolt.DB,
		return err

		realKey := fmt.Sprintf("real-%d-%d", i, j)
		pseudoKey := fmt.Sprintf("pseudo-%d-%d", i, j)

[sql-44] firewalldb: add migration code for privacy mapper from kvdb to SQL #1092

Are you sure you want to change the base?

[sql-44] firewalldb: add migration code for privacy mapper from kvdb to SQL #1092

Conversation

ViktorTigerstrom commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ellemouton commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViktorTigerstrom commented Jul 11, 2025

Uh oh!

ViktorTigerstrom commented Jul 11, 2025

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bitromortac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ViktorTigerstrom commented Jun 17, 2025 •

edited

Loading

ellemouton commented Jun 18, 2025 •

edited

Loading