<div dir="ltr"><div><div>Hellos,<br><br></div><div>I have a few points to make, and a couple of questions:<br><ol><li>Complete correctness isn't needed for all types of software development.</li><li>Runtime of tests, and supplying asynchronous results matter.</li><li>Can fuzz testing use analysis of types to generate the preconditions?<br></li></ol></div><div><br>1) Complete correctness of a solution is not always needed. Especially when prototyping, in artistic practice, and with many other types of software development or experimentation. If you can get a prototype running, then it can make errors in specification and requirements more apparent quicker than fully testing a solution which may have errors in specification.<br></div><div><br></div><div>Anything that gets in the way of iteration speed can be harmful to the outcomes (for some types of software development). Automatically testing your code is a really nice thing though, and I think can even help provide better results in the cases I mentioned above.<br></div></div><div><div><br></div><div>For really low effort testing, I think Fuzz testing can be used successfully. For development which needs to be very fast, but also fairly correct. Defining invariants which are more general is better practice, and says more about a program than not. (e.g. writing "all additions are commutative" is better than writing "1 + 2 == 2 + 1". It says more.<br></div><div><br></div><div>Just like how some people say doc tests (ipython note books) are for lazy developers, I see a similar misconception about Fuzz tests. Both provide different types of value for different amounts of resources. One accusation against writing doc tests is that they do not test many corner cases, and if you do the tests make the documentation unreadable. With Fuzz testing (with eg. Hypothesis) you can state your invariants in a small amount of space and their intention is quite clear. [[<i>this is python, we should be optimizing for reading our code right?!?</i>]]<br><br><br><br><br><br>2) The runtime of tests definitely impacts how the tests are used. If the
tests can run in under the time it takes to save a file, and write a
commit message then you won't impact the experience of the developer in
any bad ways. If you do it asynchronously, without distraction, and you
don't block up the development pipeline, then all the better.<br><br></div>By making the tests fast to run, and easy to use, Fuzz testing will become more useful for these styles/stages of development. If it takes an extra hour or two before test results are returned because fuzzing takes so long to run, then keeping Fuzzing tests as a separate job can help. Don't bundle the Fuzz tests up with your fast running unit tests. At one place we put our functional tests that took over 30 minutes to run into a separate test job. So then the quick running tests would return earlier (in less than a minute after commit).<br><br>In reality developers are not going to wait 30+ minutes for their tests to run before pushing code so they can collaborate. Running those tests in CI though does catch errors. So run the slow tests on CI, and not by default locally.<br></div><div><div><br></div><div>I know some shops have commit hooks which run ALL tests before the commit is allowed. In these places adding 30 minutes of tests won't make your work mates happy!<br></div><div><br><br></div><br><div><br><br></div><div>Ok... question time...<br></div><div><br><br><br><br><br></div><div>3) Can static type analysis be used for automated fuzz testing?<br><br></div><div>(as collected by inference, or collected by traces when running unit tests etc).<br></div><div><br>In this first example from hypothesis ( <a href="http://hypothesis.readthedocs.org/en/latest/quickstart.html" target="_blank">http://hypothesis.readthedocs.org/en/latest/quickstart.html</a> ), although the inverse operation invariant of encode(str)/decode(list) may not be deduced with static type analysis, it could definitely find out that the inputs and outputs are strings.<br><br>So for the first bug mentioned (where an empty string causes an exception in encode(str)), the automatically trying to stuff strings into the function would have found the error.<br></div><div><br></div><div>Likewise, static analysis could find that the input type for decode(list) is [[str, int]]<br></div><div><br></div><div>It seems finding invariants in existing tests would also be possible by inspecting existing assertions, and the types used in those assertions.<br><br>Say this assertion test already existed.<br><div style="margin-left:40px"><span style="font-family:monospace,monospace">our_input = 'WBWBBBW'</span><br><span style="font-family:monospace,monospace">encoded = encode(our_input)</span><br><span style="font-family:monospace,monospace"></span></div></div><div style="margin-left:40px"><span style="font-family:monospace,monospace">assert our_input == decode(encoded)<br></span></div><div><br></div><div>It could trying '<i>parametrising</i>' decode with strings, because "our_input" tells us the input is a string.<br><br></div><div>Now if the inverse property was not a real invariant, and the test provides a failing case, then you have also provided a result. The result is that the invariant does not exist. This case would require feedback from the programmer, as the test would be incorrect. But at least now the invariant would be clear. We found an inconsistency in the specification... ya!<br></div><div><br><br></div><div>This commutative assertion on addition:<br></div><div><pre style="margin-left:40px"><span style="font-family:monospace,monospace"><span><span style="font-family:monospace,monospace"><span>def</span> <span>test_ints_are_commutative</span><span>(</span><span></span><span>):</span>
<span>assert</span> <span>1</span> <span>+</span> <span>2</span> <span>==</span> <span>2</span> <span>+</span> <span>1<br></span></span></span></span></pre><pre><span style="font-family:arial,helvetica,sans-serif"><span><span>could be transformed into this:</span></span></span><br></pre><pre style="margin-left:40px"><span style="font-family:monospace,monospace"><span><span style="font-family:monospace,monospace"><span></span></span>@given</span><span>(</span><span>st</span><span>.</span><span>integers</span><span>(),</span> <span>st</span><span>.</span><span>integers</span><span>())</span>
<span>def</span> <span>test_ints_are_commutative</span><span>(</span><span>x</span><span>,</span> <span>y</span><span>):</span>
<span>assert</span> <span>x</span> <span>+</span> <span>y</span> <span>==</span> <span>y</span> <span>+</span> <span>x</span></span></pre>By finding the types in the original expression, and '<i>parametrising</i>' them.<br></div><div><br><br></div><div>But I guess in many cases there would be lots of false positives, and perhaps this wouldn't be worth doing. But maybe it will be a case of the developer just redefining the model until all the ambiguities are gone, and all the assumptions are listed.<br><br></div>But perhaps there are some cases which can be completely automated with the help of type analysis that don't give false positives? Or perhaps there are Fuzzing tools which already do this?<br><br><br><br></div><br><div><div>cheerio,<br></div><div><div><br></div><div><br></div></div></div></div>