Truffle gives different error messages for Ganache and for GETH

[barakman]

Issue

Upon 'revert' (as a result of require(false)) or 'invalid opcode' (as a result of assert(false)), Truffle gives different error messages when working with Ganache and when working with GETH.

When working with Ganache, the errors are:

• "VM Exception while processing transaction: revert"
• "VM Exception while processing transaction: invalid opcode"

When working with GETH, the errors (copied from /node_modules/truffle/build/cli.bundled.js) are:

• "Transaction: " + tx + " exited with an error (status 0) after consuming all gas.\n" + "Please check that the transaction:\n" + " - satisfies all conditions set by Solidity assert statements.\n" + " - has enough gas to execute the full transaction.\n" + " - does not trigger an invalid opcode by other means (ex: accessing an array out of bounds).";
• "Transaction: " + tx + " exited with an error (status 0).\n" + "Please check that the transaction:\n" + " - satisfies all conditions set by Solidity require statements.\n" + " - does not trigger a Solidity revert statement.\n";

This makes it hard on deploying a common infrastructure for testing contract behavior, namely, ensuring that an erroneous transaction is aborted and/or reverted.

Of course, since the specific phrases "revert" and "invalid opcode" are present in both cases (when working with Ganache and when working with GETH), it is feasible to have a common testing infrastructure.
The files assertRevert.js and assertJump.js by OpenZeppelin are a good example of it.

However, these two phrases ("revert" and "invalid opcode") can theoretically appear in a whole bunch of other error messages (in particularly, messages added by the user).

Therefore, IMO, larger phrases would be highly preferable.

For example, up until now, working with Ganache, I've been testing that the error message starts with:

• "VM Exception while processing transaction: revert"
• "VM Exception while processing transaction: invalid opcode"

Now, in order to keep my code compatible with GETH, I need to test that the error message includes:

• "revert"
• "invalid opcode"

More generally, a common set of error messages, regardless of the EVM used, would be ideal.

Thanks

Environment

• Operating System: Agnostic
• Truffle version: 4.1.3

[cgewecke]

@barakman Sorry for the slow response, missed this.

More generally, a common set of error messages, regardless of the EVM used, would be ideal.

Agree. This difference goes back to the pre-byzantium chain which did not return any information about whether a transaction was successful or not. In order to make testing possible, ganache had a policy of appending errors to the response so that web3 would trigger a helpful message.

Post-byzantium, the mainnet clients append a status field to the receipt indicating whether or not a tx was successful. Additionally, ganache-cli now has a mode that emulates them. You should see the same error pattern as geth if you run it like this:

ganache-cli --noVMErrorsOnRPCResponse

The error object also contains the receipt, so you be able to catch the error and do this:

catch(err){
  parseInt(err.receipt.status) === 0 // Failure
  parseInt(err.receipt.status) === 1  // Success
} 

Truffle continues to add a message that includes key phrases like revert in order to maintain backwards compatibility with testing libraries like Zeppelin's and to provide useful information about what happened.