No mutants killed

[nfurfaro]

@JoranHonig Hello. Great work on this project. I've just started playing with it a bit on a fairly large codebase. I ran a full test (--sample-ratio = 1.0) last night, and the results were 0/247 mutants killed. I was suprised by this, so I picked 1 mutation at random and manually tested it by switching the operator and running some tests targeting this bit of code. I quickly had 5 failing tests, which makes me question the results from this.

I'm wondering if there's any way to tweak settings to produce more accurate results?
I've tried running a smaller sample-ratio (0.1) which just produced

Mutation testing report:
Number of mutations:    24
Killed:                 0 / 24

Not sure if I'm doing something wrong here? If so, please feel free to close this issue.
For reference, here's the command & options I passed:

vertigo run --network vertigo_test_network_1 --network vertigo_test_network_2 --truffle-location /Users/overlord/DAPPS/unlock/smart-contracts/node_modules/.bin/truffle --output ./vertigo_output.txt --sample-ratio 0.1

[JoranHonig]

Hi @nfurfaro, thanks for opening an issue!

I think there might be an issue in Vertigo here, I'll triage!

[nfurfaro]

No problem. One thing that may be of help to you:
In our project we use truffle to compile and test. Often, after making changes to a contract and running truffle test, truffle will fail to pick up the changes made to contracts, and run the tests against the old artifacts instead of re-compiling. The workaround for us is to delete the build directory and recompile.

With vertigo, it may be the case that simply forcing recompilation between test runs could solve our issue.
Anyways, happy to help if I can. Please reach out if you need more info.

[JoranHonig]

@nfurfaro Thanks for the tip.

There we're actually two small issues I found.

1. Truffle doesn't like the pre-existing build directory (like you mentioned, thanks!)
2. With the feature addition of trivial compiler equivalence I accidentally put a statement in an else branch that I shouldn't have.

Anyway, it should work now! Version 1.1.3 has the fix.
I hope you can now use it on your project, tell me how that goes!

[nfurfaro]

@JoranHonig Thanks for working on this so quickly! I'm unable to get it working atm, but it's a different issue.

 Artifacts written to /var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/test-2020120-92945-1b8b0jo.mjd1
> Compiled successfully using:
   - solc: 0.5.16+commit.9c3226ce.Emscripten.clang

[-] Encountered an error while running the framework's test command:
Error: Contract Unlock not found in path /private/var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/tmpbot0c63s/build/contracts/Unlock.json

I want to triple-check and make sure I'm following the right steps to get it up and running. I'm using this as my reference:
https://medium.com/swlh/mutation-testing-for-smart-contracts-a-step-by-step-guide-68c838ca2094

Update:
So I've looked at this a bit more. Here's what I see happening in /private/var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/:

• First, the directory tmp9filh3p4 is created, including the .json artifacts.
• Next, the file test-2020121-24838-94sgf2.7s7dl appears briefly.
• It then disappears (deleted or moved?).
• Then the first file(tmp9filh3p4) is deleted or moved as well
• The process exits with:

[-] Encountered an error while running the framework's test command:
Error: Contract Unlock not found in path /private/var/folders/3_/f3d20vzs5z71mqh32lgkkz2m0000gq/T/tmp9filh3p4/build/contracts/Unlock.json

[moose-code]

I think I am getting the exact same issue.

`Error: Contract Staker not found in path /private/var/folders/c4/vx4lqxtd5v11mxtr_md_6qdh0000gn/T/tmp4upmx947/build/contracts/Staker.json

Truffle v5.1.54 (core: 5.1.54)
Node v14.15.1`

When simply running vertigo run --network development

Have tried deleting the build folder and running again