Cvising "PASS_BUG_INFO"

[aytey]

I've been cvising some C++14 code recently, and ended-up with quite a few cvise_bug_* folders (38 currently).

Is there a way for me to cvise these failures?

Basically, what I'm asking is this: if I see

ClangPass::remove-nested-function has encountered a bug:
pass error
state: 5

in PASS_BUG_INFO.TXT, is it possible for me to:

1. manually run the clang pass
2. know if it "failed"

? If so, then I can cvise the inputs to provide reduced failures.

Sadly, I can't share the original files as-is, because they're proprietary code; if I could cvise then, I can probably make files I can share.

[marxin]

Yes, I do the same. In your case you should reproduce it with ./clang_delta --transformation=remove-nested-function --counter=5 foo.c and then you can check for the corresponding crash return value.
Hope it helps?

[aytey]

Perfect! It was the clang_delta invocation I needed; I'll try to reduce my issues and file what I have left!

Maybe we should add this to the project readme (if it isn't already there)?

[marxin]

Maybe we should add this to the project readme (if it isn't already there)?

Definitely. Are you willing to send a pull request for it?

[aytey]

Can do! Do you think we should put it into PASS_BUG_INFO.TXT? We could even automatically generate a "template" interestingness test?

[marxin]

Can do! Do you think we should put it into PASS_BUG_INFO.TXT? We could even automatically generate a "template" interestingness test?

Yes and yes.
We can generate cvise command line where -c is filled with clang_delta invocation.

[marxin]

Any update on this, please?

[aytey]

Slowly working its way up the list to get done -- do you want it done sooner rather than later?

[marxin]

Don't hurry, I just wanted to remind it. It's good you're slowly, but surely getting to it.

[aytey]

Hope to look at this tonight (famous last words!).

[aytey]

Is there an easy way to reproduce this:

IncludesPass has encountered a bug:
pass got stuck
state: 50001

?

The passes that start with ClangPass:: are fine to generate the interestingness test for; but I'm not sure about the cvise passes.

EDIT what I mean is: given a file, is there a way to reproduce the cvise crash like you can do with clang_delta crashes?

[aytey]

Okay, I have things sketched out in a small Python script:

• Given a directory root, find all PASS_BUG_INFO.TXT files
• For all of these files, find the smallest file for each PASS_BUG_INFO.TXT
• Using what's in PASS_BUG_INFO.TXT, reconstruct the clang_delta command
• Re-run this command, check it still fails, and capture the output
• Sort these outputs for uniqueness, and select the smallest unit again
• Create a folder for each input file/output pair and automatically generate an interestingness script
• The interestingness captures the original message, and has placeholders to allow the user to enter the path to clang_delta, their compiler and their cflags.

There's more here than what's necessary for cvise, but the last give steps (on a per example basis), will be useful to allow someone to cvise their file before submitting it.

An example, auto-generated interestingness test looks like this:

#!/bin/bash

set -eu

clang_delta="/home/avj/clones/cvise/master/build/clang_delta/clang_delta"
pass_name="lift-assignment-expr"
pass_no="1"
cc="g++-7"
cflags="-I. -m64 -std=c++14 -c -Wall -Wextra -Werror -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-variable -Wno-literal-suffix -Wno-noexcept-type"
file="unit.cpp"

$cc $cflags $file 1>/dev/null 2>&1

ret=0
{ ${clang_delta} --transformation=${pass_name} --counter=${pass_no} ${file} 2>&1; } 1>log.txt 2>&1 || ret=$?

if [ $ret -eq 0 ]; then
    exit 1
fi

message='clang_delta: /usr/include/llvm/Support/Casting.h:104: static bool llvm::isa_impl_cl<To, const From*>::doit(const From*) [with To = clang::AttributedStmt; From = clang::Stmt]: Assertion `Val && "isa<> used on a null pointer"'"'"' failed.'

fgrep "${message}" log.txt 1>/dev/null 2>&1

exit 0

# EOF

The use of a sub-shell is very useful when clang_delta SIGSEGVs, as the sub-shell captures the string "segmentation violation":

#!/bin/bash

set -eu

clang_delta="/home/avj/clones/cvise/master/build/clang_delta/clang_delta"
pass_name="aggregate-to-scalar"
pass_no="5"
cc="g++-7"
cflags="-I. -m64 -std=c++14 -c -Wall -Wextra -Werror -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-variable -Wno-literal-suffix -Wno-noexcept-type"
file="unit.cpp"

$cc $cflags $file 1>/dev/null 2>&1

ret=0
{ ${clang_delta} --transformation=${pass_name} --counter=${pass_no} ${file} 2>&1; } 1>log.txt 2>&1 || ret=$?

if [ $ret -ne 139 ]; then
    exit 1
fi

fgrep "Segmentation fault" log.txt 1>/dev/null 2>&1

exit 0

# EOF

I'll try to get this wrapped-up in a PR soon -- I'm now testing it on my 11 "unique" clang_delta crashes!

[aytey]

... and then, you start to reduce your "bad" files, and you find more cvise bugs while cvising the original ones 🙈

[aytey]

I changed the auto-generated interestingness test so it now looks like this:

#!/bin/bash

set -eu

clang_delta="/home/avj/clones/cvise/master/build/clang_delta/clang_delta"
pass_name="lift-assignment-expr"
max_passes="1"
cc="g++-7"
cflags="-I. -m64 -std=c++14 -c -Wall -Wextra -Werror -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-variable -Wno-literal-suffix -Wno-noexcept-type"
file="unit.cpp"

$cc $cflags $file 1>/dev/null 2>&1

found=1

message='clang_delta: /usr/include/llvm/Support/Casting.h:104: static bool llvm::isa_impl_cl<To, const From*>::doit(const From*) [with To = clang::AttributedStmt; From = clang::Stmt]: Assertion `Val && "isa<> used on a null pointer"'"'"' failed.'

for pass_no in $(seq 1 ${max_passes}); do

    ret=0
    { ${clang_delta} --transformation=${pass_name} --counter=${pass_no} ${file} 2>&1; } 1>log.txt 2>&1 || ret=$?

    if [ $ret -eq 0 ]; then
        continue
    fi

    ret=0
    fgrep "${message}" log.txt 1>/dev/null 2>&1 || ret=$?

    if [ $ret -ne 0 ]; then
        continue
    fi

    found=0

    break

done

exit $found

# EOF

The reason for this change is that I wanted the pass count to potentially "get lower", rather than being fixed at what it was originally.

The new template hunts for the bug at any pass number (limited to the original pass number); I've used a hard-coded the max passes (vs. using clang_delta's ability to tell the number of instances) because I doubt we want the pass count to grow and the loop will exit early if we found it at a lower pass number.

[marxin]

Thank you very much for the testing effort, I would expect quite some cvise crashes for various inputs.
Luckily, one is not blocked by a single crash during a reduction, and the reduction typically finishes.

[aytey]

Apologies that I was quite noisy on this thread, one question at a time ... how can I manually re-run IncludesPass (which is a cvise pass and not a clang_delta pass)?

[marxin]

Sorry. Well, it's not so easy, but you can call cvise --start-with-pass IncludesPass and then somehow identify with --verbose that it's not doing progress. Do you have a reproducer I can take a look at?

[marxin]

And you can also lower GIVEUP_CONSTANT = 50000 in cvise/utils/testing.py and then grep for the message.

[aytey]

Right now, I'm unable to share my example for IncludesPass because it contains proprietary code of my employer :(

Let me see what I can do about reducing it ...