[C++] Arrow test 'arrow-utility-test' contains heap-buffer-overflow error

[PhoebeHui]

Describe the bug, including details regarding any error messages, version, and platform.

When I run arrow test with VS2022 msvc x64 configuration, the test 'arrow-utility-test' failed with heap-buffer-overflow error, could you please take a look?

Steps to reproduce:

1. Open VS2022 x86 Native Command Prompt tools
2. Clone repo and checkout 56e3836
3. set _CL_=/fsanitize=address /GS- /wd5072 & set _LINK_=/InferASanLibs /incremental:no /debug
4. cd to build folder: cd /d C:\gitP\apache\arrow\cpp\build_amd64
5. cmake -G "Visual Studio 17 2022" -A x64 -DCMAKE_SYSTEM_VERSION=10.0.26100.0  -DARROW_BUILD_TESTS=ON -DARROW_ACERO=ON -DARROW_FILESYSTEM=ON -DARROW_PARQUET=OFF -DARROW_SUBSTRAIT=ON -DARROW_DATASET=ON
6. msbuild /m /p:Platform=x64 /p:Configuration=Release arrow.sln /t:Rebuild
7.set ARROW_TEST_DATA=C:\gitP\apache\arrow\testing\data
8.ctest -C Release --output-on-failure -R "arrow-utility-test"

Memory safety issue reported by Address Sanitizer:

=31288==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x123a93731969 at pc 0x7ffb31957c67 bp 0x00705a96dd40 sp 0x00705a96d4c8
READ of size 1007 at 0x123a93731969 thread T0
    #0 0x7ffb31957c66 in MemcmpInterceptorCommon(void *, int (__cdecl *)(void const *, void const *, unsigned __int64), void const *, void const *, unsigned __int64) C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:850
    #1 0x7ffb31960390 in memcmp C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\sanitizer_common\sanitizer_common_interceptors.inc:882
    #2 0x7ff7c3faf7d0 in std::basic_string_view<char, struct std::char_traits<char>>::_Equal(class std::basic_string_view<char, struct std::char_traits<char>>) const (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14054f7d0)
    #3 0x7ff7c3fd8a4b in testing::internal::CmpHelperEQ<class std::basic_string_view<char, struct std::char_traits<char>>, class std::basic_string_view<char, struct std::char_traits<char>>>(char const *, char const *, class std::basic_string_view<char, struct std::char_traits<char>> const &, class std::basic_string_view<char, struct std::char_traits<char>> const &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140578a4b)
    #4 0x7ff7c3fdc5a9 in arrow::util::test::IsSecurelyCleared(class std::basic_string_view<char, struct std::char_traits<char>> const &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14057c5a9)
    #5 0x7ff7c3fdc426 in arrow::util::test::IsSecurelyCleared(class std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>> const &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14057c426)
    #6 0x7ff7c3fc5c0f in arrow::util::test::TestSecureString_AssertSecurelyCleared_Test::TestBody(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140565c0f)
    #7 0x7ffb92418676 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180008676)
    #8 0x7ffb924181cb in testing::internal::HandleExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800081cb)
    #9 0x7ffb9248674c in testing::Test::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x18007674c)
    #10 0x7ffb92486a2a in testing::TestInfo::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076a2a)
    #11 0x7ffb92486e83 in testing::TestSuite::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076e83)
    #12 0x7ffb92487cfd in testing::internal::UnitTestImpl::RunAllTests(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180077cfd)
    #13 0x7ffb924186d6 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800086d6)
    #14 0x7ffb924185c9 in testing::internal::HandleExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800085c9)
    #15 0x7ffb924872db in testing::UnitTest::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800772db)
    #16 0x7ff7c3f435de in main (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1404e35de)
    #17 0x7ff7c42b2287 in invoke_main C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
    #18 0x7ff7c42b2287 in __scrt_common_main_seh C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #19 0x7ffbf7c5259c  (C:\Windows\System32\KERNEL32.DLL+0x18001259c)
    #20 0x7ffbf850af77  (C:\Windows\SYSTEM32\ntdll.dll+0x18005af77)

0x123a93731970 is located 0 bytes after 1008-byte region [0x123a93731580,0x123a93731970)
allocated by thread T0 here:
    #0 0x7ff7c42b1285 in operator new(unsigned __int64) C:\repos\msvc\src\vctools\asan\llvm\compiler-rt\lib\asan\asan_win_new_scalar_thunk.cpp:40
    #1 0x7ff7c3adf531 in std::_Allocate<16, struct std::_Default_allocate_traits>(unsigned __int64) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14007f531)
    #2 0x7ff7c3adf5c9 in std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>::_Allocate_for_capacity<0>(class std::allocator<char> &, unsigned __int64 &) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14007f5c9)
    #3 0x7ff7c3ca6446 in std::basic_string<char, struct std::char_traits<char>, class std::allocator<char>>::_Construct<0, char>(char, unsigned __int64) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140246446)
    #4 0x7ff7c3fc5b62 in arrow::util::test::TestSecureString_AssertSecurelyCleared_Test::TestBody(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x140565b62)
    #5 0x7ffb92418676 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180008676)
    #6 0x7ffb924181cb in testing::internal::HandleExceptionsInMethodIfSupported<class testing::Test, void>(class testing::Test *, void (__cdecl testing::Test::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800081cb)
    #7 0x7ffb9248674c in testing::Test::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x18007674c)
    #8 0x7ffb92486a2a in testing::TestInfo::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076a2a)
    #9 0x7ffb92486e83 in testing::TestSuite::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180076e83)
    #10 0x7ffb92487cfd in testing::internal::UnitTestImpl::RunAllTests(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x180077cfd)
    #11 0x7ffb924186d6 in testing::internal::HandleSehExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800086d6)
    #12 0x7ffb924185c9 in testing::internal::HandleExceptionsInMethodIfSupported<class testing::internal::UnitTestImpl, bool>(class testing::internal::UnitTestImpl *, bool (__cdecl testing::internal::UnitTestImpl::*)(void), char const *) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800085c9)
    #13 0x7ffb924872db in testing::UnitTest::Run(void) (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow_gmock_main.dll+0x1800772db)
    #14 0x7ff7c3f435de in main (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x1404e35de)
    #15 0x7ff7c42b2287 in invoke_main C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
    #16 0x7ff7c42b2287 in __scrt_common_main_seh C:\repos\msvc\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
    #17 0x7ffbf7c5259c  (C:\Windows\System32\KERNEL32.DLL+0x18001259c)
    #18 0x7ffbf850af77  (C:\Windows\SYSTEM32\ntdll.dll+0x18005af77)

SUMMARY: AddressSanitizer: heap-buffer-overflow (C:\gitP\apache\arrow\cpp\build_amd64\release\Release\arrow-utility-test.exe+0x14054f7d0) in std::basic_string_view<char, struct std::char_traits<char>>::_Equal(class std::basic_string_view<char, struct std::char_traits<char>>) const
Shadow bytes around the buggy address:
  0x123a93731680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x123a93731900: 00 00 00 00 00 00 00 00 00 00 00 00 00[01]fa fa
  0x123a93731980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x123a93731a00: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x123a93731a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x123a93731b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==31288==ABORTING

Component(s)

C++

[ArnavBalyan]

Looks like this is due to MSVC specific behaviour of smaller heap than capacity, patch will be raised thanks for reporting

[pitrou]

Issue resolved by pull request 49906
#49906