Skip to content

Introduce validation to the ColumnArray constructor #428

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

IyeOnline
Copy link

@IyeOnline IyeOnline commented Jun 3, 2025

The ColumnArray constructor would just accept incompatible value- and
offset-arrays and the client would then happily send them, causing
issues in the ClickHouse server.

This introduces validation of the arrays.

Fixes #426

The ColumnArray constructor would just accept incompatible value- and
offset-arrays and the client would then happily send them, causing
issues in the ClickHouse server.

This introduces validation of the arrays.
@CLAassistant
Copy link

CLAassistant commented Jun 3, 2025

CLA assistant check
All committers have signed the CLA.

@IyeOnline
Copy link
Author

Code that triggers the issue and would fail without the change
#include <clickhouse/client.h>

using namespace clickhouse;

/// Produces the offset array for a column of `n_rows`, which each row being an
/// array of 2 elements
auto make_offsets(const size_t n_rows) -> std::shared_ptr<ColumnUInt64>
{
    auto column_offsets = std::make_shared<ColumnUInt64>();
    for ( size_t i=0; i < n_rows; ++i) {
      column_offsets->Append(i*2);
    }
    return column_offsets;
}

/// Produces non-monotonic offsets that still satisfy the protocol
auto make_broken_offsets(const size_t n_rows) -> std::shared_ptr<ColumnUInt64>
{
    auto column_offsets = std::make_shared<ColumnUInt64>();
    for ( size_t i=1; i < n_rows+1; ++i) {
      column_offsets->Append(i%2 ? i : i*2);
    }
    return column_offsets;
}


/// Produces the values array for a column of `n_rows`, which each row being an
/// array of 2 elements
auto make_values(const size_t n_rows) -> std::shared_ptr<ColumnString>
{
    auto column_values = std::make_shared<ColumnString>();
    for ( size_t i=0; i < n_rows; ++i) {
      column_values->Append("some_long_string "+std::to_string(2*i));
      column_values->Append("some_long_string "+std::to_string(2*i+1));
    }
    return column_values;
}

/// Simply sends the column array as a block
void send(Client &client, std::shared_ptr<ColumnString> values, std::shared_ptr<ColumnUInt64> offsets)
{
    Block block;
    auto column = std::make_shared<ColumnArray>(std::move(values), std::move(offsets));
    block.AppendColumn("col", std::move(column));
    client.Insert("crash_showcase", block);
}


int main()
{
    Client client(ClientOptions().SetHost("localhost"));
    client.Execute("CREATE TABLE IF NOT EXISTS crash_showcase (col Array(String)) ENGINE = Memory");

    /// Missing elements in the value array causes a timeout in the clickhouse server
    /// Solutions:
    /// * The client should at least do some rudimentary checks of offset array vs
    ///    the data array
    /// * The server should presumably do some verification of the data it gets
    send( client, make_values(5), make_offsets(10));


    /// Having too many elements in the data array causes the server to just start
    /// interpreting the excess values as new messages, which is really bad
    /// Solutions:
    /// * The client should at least do some rudimentary checks of offset array vs
    ///    the data array
    /// * The server should absolutely verify that there is not more elements than
    ///   expected
    send( client, make_values(6000),  make_offsets(100));


    /// The protocol at the very least specifies that the last offset is the column count:
    /// https://clickhouse.com/docs/native-protocol/columns#array
    /// While non-monotonic offsets *seem to work* (at least no error happens when sending), I am not sure about that.
    send( client, make_values(10),  make_broken_offsets(10));
}

@mshustov mshustov requested a review from Copilot June 5, 2025 08:24
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds validation logic in the ColumnArray constructor to ensure that the provided value and offset arrays are compatible, preventing inconsistent data from being sent to the ClickHouse server.

  • Added a check to ensure that the offsets array is monotonically increasing.
  • Validated that the size of the data array matches the expected number of values derived from the offsets.

@mshustov
Copy link
Member

mshustov commented Jun 9, 2025

@IyeOnline could you check the failing tests, please?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Invalid request sent from clickhouse-cpp
3 participants