https://www.hillelwayne.com/tags/testing/ Recent content in Testing on Hillel Wayne Hugo -- gohugo.io en-us Mon, 28 Jun 2021 00:00:00 +0000 https://www.hillelwayne.com/post/cross-branch-testing/ Mon, 28 Jun 2021 00:00:00 +0000 https://www.hillelwayne.com/post/cross-branch-testing/ This was originally a newsletter post I wrote back in December, but I kept coming back to the idea and wanted to make it easier to find. I’ve made some small edits for flow. There’s a certain class of problems that’s hard to test: The output isn’t obviously inferable from the input. The code isn’t just solving a human-tractable problem really quickly, it’s doing something where we don’t know the answer without running the program. https://www.hillelwayne.com/post/property-testing-complex-inputs/ Fri, 19 Jun 2020 00:00:00 +0000 https://www.hillelwayne.com/post/property-testing-complex-inputs/ This was originally written as a tutorial for Hypothesis and will eventually be reproduced there, but it might still be useful for people using other property-testing libraries. This essay assumes some familiarity with Hypothesis. If you haven’t used it, a better introduction is here. Once you learn the basics, there are two hard parts to writing property-based tests: What are good properties to test? How do I generate complex inputs? https://www.hillelwayne.com/post/feature-interaction/ Wed, 05 Feb 2020 00:00:00 +0000 https://www.hillelwayne.com/post/feature-interaction/ In most testing frameworks, you’re expected to write a lot of low-level tests and few high-level tests. The reasoning is that end-to-end tests are slow and expensive and only cover a tiny amount of the program’s state space. It’s better to focus on testing all the parts in isolation versus testing that they all work together. In practice, global correctness does not follow from local correctness. It’s possible for every part to be individually rock-solid but the system to be broken as a whole. https://www.hillelwayne.com/post/contract-examples/ Mon, 08 Apr 2019 00:00:00 +0000 https://www.hillelwayne.com/post/contract-examples/ A while back I was ranting about APLs and included this python code to get the mode of a list: def mode(l): max = None count = {} for x in l: if x not in count: count[x] = 0 count[x] += 1 if not max or count[x] > count[max]: max = x return max There’s a bug in it. Do you see it? If not, try running it on the list [0, 0, 1]: https://www.hillelwayne.com/post/metamorphic-testing/ Mon, 25 Mar 2019 00:00:00 +0000 https://www.hillelwayne.com/post/metamorphic-testing/ Confession: I read the ACM Magazine. This makes me a dweeb even in programming circles. One of the things I found in it is “Metamorphic Testing”. I’ve never heard of it, and nobody I knew heard about it either. But the academic literature was shockingly impressive: many incredibly successful case studies in wildly different fields. So why haven’t we heard of it before? There’s only one article anywhere targeted at people outside academia. https://www.hillelwayne.com/post/a-bunch-of-tests/ Mon, 02 Apr 2018 00:00:00 +0000 https://www.hillelwayne.com/post/a-bunch-of-tests/ I’ve been working on a bunch of longform obligation pieces and while they’re a lot of fun, they’re also steadily driving me insane. So I took a day off to write about all of the kinds of automated testing I know about. I’m defining tests here to be “an independent verification program that, as part of verification, executes the code we want to verify.” This means types are not tests, as they don’t involve execution of the code, and contracts are not tests, because they’re not executed as an independent program. https://www.hillelwayne.com/post/pbt-contracts/ Sun, 17 Dec 2017 00:00:00 +0000 https://www.hillelwayne.com/post/pbt-contracts/ I’m a pretty big fan of contracts and property-based testing. While they’re both powerful, they do have tradeoffs: Contracts are simple and allow for complex conditionals, but you need to test them over a wide space to confirm them. Property-Based Testing can test your code over a very wide search space, but you have to come up with good generators and invariants. Maybe they complement each other: “your code doesn’t violate any contracts” counts as a PBT invariant. https://www.hillelwayne.com/post/why-tdd-isnt-crap/ Mon, 30 Oct 2017 00:00:00 +0000 https://www.hillelwayne.com/post/why-tdd-isnt-crap/ After my recent vitriol about unit testing, a couple of people sent me Why TDD is Crap as a thorough debunking of TDD and unit testing. As someone very interested in software correctness, I ended up writing a debunking of his debunking. Transcriptions will be in quotes, my responses below. Some important notes: From what can tell, neither of us is using TDD in the “purest” possible sense of “write the bare minimum that makes the smallest unit test pass”. https://www.hillelwayne.com/post/unit-tests-are-not-tests/ Thu, 26 Oct 2017 00:00:00 +0000 https://www.hillelwayne.com/post/unit-tests-are-not-tests/ This is an absolutely insane idea and I’m delighted by its audacity so I’m sharing it with all y’all. There’s like a bazillion things wrong with it but saying dumb shit has never stopped me before so let’s do some grambling! The way I see it, unit tests act as a frame around your coding. The most common practice is you write the tests to cover the return values and calls, write your code, and enforce that it passes the unit tests. https://www.hillelwayne.com/post/uncle-bob/ Thu, 05 Oct 2017 00:00:00 +0000 https://www.hillelwayne.com/post/uncle-bob/ A while back I wrote that Robert Martin was ruining software by being too good at programming. That was supposed to be a joke. Since then he’s done his damndest to actually ruin software by telling people they’re doing it wrong. His most recent response where he yells at software correctness was the breaking point for me, so I’m going to go ahead and say what many of us have been thinking: https://www.hillelwayne.com/post/how-do-we-trust-science-code/ Mon, 14 Aug 2017 00:00:00 +0000 https://www.hillelwayne.com/post/how-do-we-trust-science-code/ In 2010 Carmen Reinhart and Kenneth Rogoff published Growth in a Time of Debt. It’s arguably one of the most influential economics papers of the decade, convincing the IMF to push austerity measures in the European debt crisis. It was a very, very big deal. In 2013 they shared their code with another team, who quickly found a bug. Once corrected, the results disappeared. Greece took on austerity because of a software bug. https://www.hillelwayne.com/post/hypothesis-oracles/ Fri, 21 Jul 2017 09:00:00 -0500 https://www.hillelwayne.com/post/hypothesis-oracles/ This post is about the Hypothesis testing library. If you’re unfamiliar with it, check it out! Property tests are a fantastic addition to your testing suite. All examples use pytest. Imagine we’ve written a bubblesort: def bubblesort(l: List[int]) -> List[int]: # blah blah blah What are some of the properties we could test? We could check that the input has the same length as the output: from hypothesis import given from hypothesis. https://www.hillelwayne.com/post/python-doctests/ Thu, 16 Feb 2017 00:00:00 +0000 https://www.hillelwayne.com/post/python-doctests/ Doctests are one of the most fascinating things in Python. Not necessarily because it’s particularly elegant or useful, but because it’s unique: I haven’t found another language that has a similar kind of feature. Here’s how it works. Imagine I was writing an adder: def add(a,b): return a + b There’s two kinds of ways we can test it. The first is with unit tests, which everybody’s already used to.