The current bulk ingestion mechanism in the MySQL ADBC driver relies on batched INSERT statements. This method can be slow when compared to using LOAD DATA statement which according to the MySQL documentation is usually 20 times faster than using INSERT statements.
We should implement a high-performance ingestion path using MySQL's LOAD DATA LOCAL INFILE statement. This command allows the database to read a stream of raw data directly, bypassing much of the standard SQL processing overhead.
References:
- https://dev.mysql.com/doc/refman/8.4/en/insert-optimization.html
- https://dev.mysql.com/doc/refman/8.4/en/load-data.html
The current bulk ingestion mechanism in the MySQL ADBC driver relies on batched INSERT statements. This method can be slow when compared to using LOAD DATA statement which according to the MySQL documentation is usually 20 times faster than using INSERT statements.
We should implement a high-performance ingestion path using MySQL's LOAD DATA LOCAL INFILE statement. This command allows the database to read a stream of raw data directly, bypassing much of the standard SQL processing overhead.
References: