PoC (Query): RowBinaryWithNamesAndTypes for enchanced type safety

[slvrtrn]

Summary

Warning

This is a work in progress implementation and may change significantly.
It implements RBWNAT for Query only; Insert should be a new PR.

First of all, let's abbreviate RowBinaryWithNamesAndTypes format as RBWNAT, and the regular RowBinary as just RB for simplicity.

There is a significant amount of issues created in the repo regarding schema incompatibility or obscure error messages in the repository (full list TBD). The reason is that the deserialization is effectively implemented in a "data-driven" way, where the user structures dictate the way the stream in RB should be (de)serialized, so it is possible to have a hiccup where two UInt32 may be deserialized as a single UInt64, which in worst case scenario may lead to corrupted data. For example:

This test will deserialize a wrong value on the main branch, cause DateTime64 is streamed as 8 bytes (Int64), and 2x(U)Int32 are also streamed as 8 bytes in total. It correctly throws an error on this branch now with enabled validation mode.

#[tokio::test]
#[cfg(feature = "time")]
async fn test_serde_with() {
    #[derive(Debug, Row, Serialize, Deserialize, PartialEq)]
    struct Data {
        #[serde(with = "clickhouse::serde::time::datetime64::millis")]
        n1: OffsetDateTime, // underlying is still Int64; should not compose it from two (U)Int32
    }

    let client = prepare_database!().with_struct_validation_mode(StructValidationMode::EachRow);
    let result = client
        .query("SELECT 42 :: UInt32 AS n1, 144 :: Int32 AS n2")
        .fetch_one::<Data>()
        .await;

    assert!(result.is_err());
    assert!(matches!(
        result.unwrap_err(),
        Error::InvalidColumnDataType { .. }
    ));
}

This PR introduces:

• Optional RBWNAT format usage instead of RB, which allows for stronger type safety guarantess. This is regulated by the StructValidationMode client option, which has three possible modes:
  • Disabled - like current main branch, uses RB without validation
  • FirstRow - new mode, uses RBWNAT and checks the types for the first row only, so it retains most of the performance compared to the Disabled mode, while still providing significantly stronger guarantees.
  • EachRow - every single row is validated. It is the slowest out of the three.
• New rowbinary internal crate that contains utils to deal with RBWNAT and Native data types strings parsing into a proper AST. Rustified from https://github.com/ClickHouse/clickhouse-js/blob/main/packages/client-common/src/parse/column_types.ts, but not entirely. The most important part is the correctness and the tests, the actual implementation detail can be adjusted in the follow-up.
• An ability to conveniently deserialize map as a HashMap<K, V>, and not only as Vec<(K, V)>, which was confusing.
• Clearer error messages when using RBWNAT.
• A lot of tests and more to come, especially with difficult corner cases (nested nullable, multi-dimensional mixed arrays/maps/tuples/enum, etc).
• Benchmarks to highlight the difference in performance between validation modes are WIP and will be added to this PR.

Likely possible to implement:

• Stricter enum values validation, e.g. that the deserialized Int repr matches exactly
• Support for "shuffled" structure definition, where the order of the fields does not match the DB, but the names and types are correct; it should be possible by leveraging (perhaps optionally) visit_map API allowed for deserialize_struct instead of current visit_seq, which processes a struct as a tuple.

Source files to look at:

• data_types.rs - RBWNAT/Native data types strings parsed as proper AST. See the tests for more output examples, like Variant data type parsing.
• rbwnat.rs - main integration tests WIP.
• validation.rs - validating Serde calls against the data types provided in the RBWNAT format header.

[slvrtrn]

Added a few comments regarding the intermediate implementation.

[slvrtrn]

Perhaps it is more of clickhouse-data-types-utils, cause this logic is shared across RBWNAT and Native, and the produced AST can be used in libraries such as ch2rs.

[slvrtrn]

for easy testing

[slvrtrn]

more or less the same as the implementation in the deserializer. Perhaps as a follow-up, all the reader logic can be extracted in similar functions with #[inline(always)]?

[slvrtrn]

Needs revising.

[slvrtrn]

used for tests only

[slvrtrn]

This is currently the main drawback of the validation implementation if we want to disable it after the first N rows for better performance. If these first rows are all NULLs, then we do not properly validate the inner type.

[slvrtrn]

adds HashMap support with proper validation, including nested types

[slvrtrn]

moved to utils. Perhaps it should be just moved to rowbinary crate.