Support SQLCipher using package traits

[marcprux]

This is a less ambitious alternative to #1701 that enables adding a SQLCipher source dependency to the Package.swift based on the GRDBCIPHER environment variable. Tests can be run with the dependency with a command like:

GRDBCIPHER="https://github.com/skiptools/swift-sqlcipher.git#1.2.1" swift test

Running it without the environment variable will leave things exactly as they are. The majority of changes in this PR are simply re-ordering the import header checks for GRDBCIPHER and SWIFT_PACKAGE, since we would need the former to take precedence of the latter.

An advantage of doing it this way is that it facilitates creating a GRDB-SQLCipher fork that can follow the releases of the upstream. The only change in such a fork would be to hardwire the GRDBCIPHER setting in Package.swift to some SQLCipher source repository (either mine, or one that you maintain). When a client package wants to switch between GRDB and GRDB-SQLCipher, they would just need to swap their dependency URL. (this is no longer needed when using package traits; see comment below)

This PR also adds in support building and testing against Android, mostly by stubbing out the unit tests that can't compile due to missing NSFileCoordinator, Combine, and the like. Android tests can be run with skip:

GRDBCIPHER="https://github.com/skiptools/swift-sqlcipher.git#1.2.1" skip android test

Most of the tests pass, but there are a few that need to be investigated. I'll look into them next.

Closes #1701

[marcprux]

Addendum: out of curiosity, I ran some comparisons between the built-in SQLite and the SQLCipher build with the following script (and adding a GRDB_PERFORMANCE_TESTS check to enable GRDBPerformanceTests in Package.swift).

The results over a dozen runs are a fairly consistent slowdown of around 10% when using SQLCipher. It'd be interesting to tweak the SQLite build flags in swift-sqlcipher to see what might increase/decrease the performance.

echo "With GRDBCIPHER"; rm -r .build; GRDB_PERFORMANCE_TESTS=1 GRDBCIPHER="https://github.com/skiptools/swift-sqlcipher.git#1.2.1" swift test --filter GRDBPerformanceTests 2>&1 | grep --after-context=2 "Test Suite 'GRDBPackageTests.xctest' passed"
echo "Without GRDBCIPHER"; rm -r .build; GRDB_PERFORMANCE_TESTS=1 swift test --filter GRDBPerformanceTests 2>&1 | grep --after-context=2 "Test Suite 'GRDBPackageTests.xctest' passed"

With GRDBCIPHER
Test Suite 'GRDBPackageTests.xctest' passed at 2025-01-26 22:33:13.477.
	 Executed 23 tests, with 0 failures (0 unexpected) in 353.144 (353.146) seconds
Test Suite 'Selected tests' passed at 2025-01-26 22:33:13.477.
Without GRDBCIPHER
Test Suite 'GRDBPackageTests.xctest' passed at 2025-01-26 22:38:59.398.
	 Executed 23 tests, with 0 failures (0 unexpected) in 314.684 (314.686) seconds

Update: this was simply because I wasn't building in release mode. See comment below, where the SQLCipher build is actually 10% faster.

[Joannis]

@marcprux the Android NDK doesn't seem to bundle sqlite3.h, whereas GRDB and your PR do rely on that being available. How are you providing that right now when building and testing GRDB?

[marcprux]

@marcprux the Android NDK doesn't seem to bundle sqlite3.h, whereas GRDB and your PR do rely on that being available. How are you providing that right now when building and testing GRDB?

It is included with the SQLCipher source module: https://github.com/skiptools/swift-sqlcipher/tree/main/Sources/SQLCipher/sqlite

[groue]

Hello @marcprux,

I like that very much. This pull request is a great gift 🎁! I'll shortly play with the new targets.

I made a first review and preliminary comments. I hope we agree on the broad outlines!

[groue]

The results over a dozen runs are a fairly consistent slowdown of around 10% when using SQLCipher. It'd be interesting to tweak the SQLite build flags in swift-sqlcipher to see what might increase/decrease the performance.

Isn't it just the cipher? (EDIT: I'm wrong because performance tests don't run on encrypted databases, see comment below.)

SQLCipher can also be extremely slow when it opens a connection. Some people have reported up to 0.5 seconds (😱). To the point I had to ship #1350.

[groue]

I ran the same performance tests, but this time in release configuration:

echo "With GRDBCIPHER"; rm -rf .build; GRDB_PERFORMANCE_TESTS=1 GRDBCIPHER="https://github.com/skiptools/swift-sqlcipher.git#1.2.1" swift test --configuration release --filter GRDBPerformanceTests 2>&1 | grep --after-context=2 "Test Suite 'GRDBPackageTests.xctest' passed"
echo "Without GRDBCIPHER"; rm -rf .build; GRDB_PERFORMANCE_TESTS=1 swift test --configuration release --filter GRDBPerformanceTests 2>&1 | grep --after-context=2 "Test Suite 'GRDBPackageTests.xctest' passed"

With GRDBCIPHER
Test Suite 'GRDBPackageTests.xctest' passed at 2025-02-01 13:53:37.700.
	 Executed 23 tests, with 0 failures (0 unexpected) in 72.258 (72.260) seconds
Test Suite 'Selected tests' passed at 2025-02-01 13:53:37.700.
Without GRDBCIPHER
Test Suite 'GRDBPackageTests.xctest' passed at 2025-02-01 13:57:26.264.
	 Executed 23 tests, with 0 failures (0 unexpected) in 80.139 (80.141) seconds
Test Suite 'Selected tests' passed at 2025-02-01 13:57:26.264.

This time, SQLCipher tests are faster.

[marcprux]

SQLCipher can also be extremely slow when it opens a connection

Agreed, but the performance hit should only occur when encryption is actually enabled on the database, which shouldn't affect any of the perf tests.

This time, SQLCipher tests are faster.

Of course! Foolish of me to not do that. Otherwise, it is comparing the debug build of the sqlcipher/sqlite source build against the release build of the vendored sqlite.

It is heartening to see that superior performance can be squeezed out of a source build. I wonder how much more might be managed with strategic performance-related flags (e.g., sbooth/CSQLite#59 (comment)).

[marcprux]

Apologies for the long delay in addressing the feedback (I was traveling).

I believe I've handled each of the requested changes. Let me know if I can do anything else to help get this PR over the finish line – it has grown quite large, so I'm nervous that rebasing is going to be difficult if it starts to become stale.

[Joannis]

We've been using this branch successfully.

[marcprux]

I'll follow up on the list of needs at https://mastodon.social/@[email protected]/113990450491505387 here, just to keep it in the PR record.

• Provides built-in SQLCipher support that works for 80+% of people.

This should work out of the box for anyone that just changes:

    dependencies: [
        .package(url: "https://github.com/groue/GRDB.swift.git", from: "7.0.0"),
    ]

to the (theoretical) locally-maintained fork:

    dependencies: [
        .package(url: "https://github.com/groue/GRDB-SQLCipher.swift.git", from: "7.0.0"),
    ]

• No extra dependency (we must control the SQLCipher compilation options).

You are welcome to relocate https://github.com/skiptools/swift-sqlcipher.git into the groue/ org and use that – then you would have control over releases and flags.

• No binaries (they rot).

swift-sqlcipher.git is a pure source build. There are no binaries.

• A clear documentation that tells users what to do to get GRDB+SQLCipher with SPM, written for people who are not build experts.

This isn't part of this PR, but it would only be a couple of lines in the README.

• Testable in GitHub CI.

This is part of the PR. The new SPMSQLCipher job just does:

GRDBCIPHER="https://github.com/skiptools/swift-sqlcipher.git#1.2.1" swift test

• Runs the full GRDB test suite with unencrypted databases, AND with encrypted databases (we already do that for SQLCipher with CocoaPods).

The tests do run against SQLCipher, but against an unencrypted database. I didn't see how this is being done with SQLCipher for CocoaPods, but I suppose we could just do the same setup here (perhaps just based on checking the GRDBCIPHER environment variable)

• Requires minimum efforts for debugging (one can debug tests from Xcode).

This shouldn't be an issue. Just activate (i.e., un-comment) the GRDBCIPHER line in the Package.swift and run the test cases in Xcode.

• Requires minimum efforts for maintenance (i.e. bumping SQLCipher version and adjusting compile options when needed)

For the SQLCipher dependency, you just need to make a new amalgamated sqlite.c from sqlcipher, commit it, and then tag a release. This is done with the script: https://github.com/skiptools/swift-sqlcipher/blob/main/scripts/build_sqlcipher.sh

An example of this sort of update commit is the bump to sqlite 3.46.1 and sqlcipher 4.6.1: skiptools/swift-sqlcipher@5ea2f8b

For GRDB itself, my proposal is that you maintain a fork of GRDB.swift.git at https://github.com/groue/GRDB-SQLCipher.swift.git. The fork is pristine, with the one exception that the GRDBCIPHER line in the Package.swift is un-commented. Whenever you make a release of GRDB.swift.git, you just update the GRDB-SQLCipher.swift.git fork and sync the tags.

It is a little bit of overhead, but it could be added as another step in any scripts you might already be using to maintain releases.

• A technique thats enables SQLCipher support in companion packages as well (GRDBQuery, GRDBSnapshotTesting, RxGRDB).

This is certainly a shortcoming of this method. I unfortunately don't see any solution other than to also have a GRDBQuery-SQLCipher that depends on the GRDB-SQLCipher.swift.git fork.

This last point is indeed painful. However, I think this PR gets us a lot, with minimal intrusion on the existing GRDB structure. A more "correct" solution would be along the lines of #1701, which requires a large refactoring and had performance implications, and so was rejected.

We can have a working GRDB-SQLCipher implementation (plus Android support!) today with this PR, while still continuing to design a more ideal future implementation.

Update: the fork is no longer necessary when using package traits (see comment below)

[groue]

I'm sorry I did not answer yet. I had to cope with the problems created with Xcode 16.3 beta. I'll get back to you soon.

[Alex293]

This PR saved me today while integrating a closed source framework which links SQLCipher while we were already extensively using grdb.

[marcprux]

I've updated this PR to use the new package traits in Swift 6.1, which seem designed for a scenario exactly like this. Now, rather than having to create and maintain a custom GRDB-SQLCipher fork that enables GRDBCIPHER, a consumer of this package can simply enable the GRDBCIPHER trait when adding a dependency on GRDB:

    dependencies: [
        .package(url: "https://github.com/groue/GRDB.swift", from: "7.5.0", traits: ["GRDBCIPHER"])
    ]

Adding the trait to the dependency will activate the GRDBCIPHER conditionals as well as adding a conditional dependency on the swift-sqlcipher repository, whereas leaving the trait empty (the default) will result in GRDB continuing to use the vendored SQLite3.

For the purpose of testing and CI, the "GRDBCIPHER" environment variable is still checked, but only to see whether to enable the trait by default in the Package.swift, so you can run:

# test with SQLCipher
GRDBCIPHER=1 swift test 
# test with SQLite3
GRDBCIPHER=0 swift test 

The performance test environment can also still be used, which demonstrates that the custom SQLCipher build continues to be ~10% faster than the vendored SQLite3, at least on my machine:

# SQLCipher performance tests
GRDBCIPHER=1 GRDB_PERFORMANCE_TESTS=1 swift test -c release --filter GRDBPerformanceTests 
# Executed 23 tests, with 0 failures (0 unexpected) in 73.005 (73.009) seconds

# SQLite3 performance tests
GRDBCIPHER=0 GRDB_PERFORMANCE_TESTS=1 swift test -c release --filter GRDBPerformanceTests 
# Executed 23 tests, with 0 failures (0 unexpected) in 81.398 (81.401) seconds

The only drawback is that since traits are simple booleans, you can no longer specify a custom repository with the "GRDBCIPHER" environment variable, so you need to depend on a specific repository. But an advantage is that the swift-sqlcipher.git repository could now publish its own traits that will control various SQLite build flags, which means that consumers of the package could potentially have fine-grained control over the SQLite build itself.

[marcprux]

@groue Hoping you can take another look at this PR with the new Package traits dependency. I think it really improves on the original idea.