I hear this question frequently, especially from students and developers just starting their careers.
Let’s start answering this question by raising the stakes, and say you’re working on software embedded in a medical device. A medical professional using the device can enter various settings using a touch screen or remotely, using their tablet computer. Let’s say this device administers life-saving medication to a patient, and if the dosage delivered is not correct, one of two outcomes could occur:
- Too small a dose, and the patient’s condition remains the same or worsens. The result could ultimately be fatal.
- Too large a dose, and the patient could die or suffer some long-term disability. The result could ultimately be fatal.
Are these stakes high enough? I sure hope so.
Now, let’s say there is a function in this software which is documented to permit a specific range of dosage values, and the software requirements documentation says that any value outside this range will be rejected and generate an error message to the user, and will not change the dosage delivered to the patient unless an in-range value is passed in.
Great. The documentation makes the function sound safe enough, and a medical expert has confirmed that the range of dosages the function accepts is the safe range. So, based on the premise of your question, let’s say we test only valid inputs to this function. In this case, we pass in the valid values at both ends of the range, and perhaps a few valid values in between those extremes. And that’s all we test for that parameter to that function…valid values. All the tests pass. The project is finished, the software is released, and the device is hooked up to patients. What could possibly go wrong?
Testing is about ensuring that the software meets the requirements. If you don’t test the invalid inputs – the error paths – you’re not ensuring that the software meets the requirements.
I’ll tell you, in case you haven’t figured it out yet. Just because the documentation says that the function will accept values within a certain range doesn’t mean the software developer who implemented the function got it completely right. There are many many ways to get it wrong inside the function: using an incorrect relational operator, checking the wrong parameter, relying on constants defined elsewhere in the program which happen to have the wrong values, a race condition in shared memory that steps on a value in some situations, rejecting the value but not issuing the error message, and the list goes on and on and on.
Testing is about ensuring that the software meets the requirements. If you don’t test the invalid inputs – the error paths – you’re not ensuring that the software meets the requirements.
The most obvious case is delivering far too much medication, putting the patient’s life in danger. If the function doesn’t properly check the upper end of the valid range, it dutifully delivers too much medication, and the patient suffers the consequences. No error message is issued, so no one thinks anything is wrong. Let’s say there’s a bug in the communication mechanism for remotely changing the dosage via a tablet computer. The medical professional enters a valid value into the tablet, the value gets corrupted enroute to the device, the function doesn’t detect that the value is out of range, the patient gets too high a dose, and the display on the tablet (the only thing the medical professional sees in front of them) sees the correct dose they entered.
Consider this scenario. A doctor sees that the patient is not improving, so they increase the dosage, but they tap on the keyboard an extra time, which increases the dosage beyond the valid range. The function receives the invalid value, and correctly rejects it by not changing the actual dosage, but it fails to report the invalid entry to the doctor, so the doctor walks away thinking the dosage has been increased. The patient’s condition continues to deteriorate, because they didn’t get the needed increased dosage, and no error message came up to alert the doctor that the wrong value was entered. By the time the staff realizes the dosage wasn’t changed when they thought it had been, the patient’s condition is grave.
So, testing the invalid inputs is not just about ensuring that invalid values are ignored, but that all the right things happen to alert the user that something went wrong, allowing them to see the problem and correct it.
If life-sustaining medical equipment isn’t high-stakes enough for you, think about the software embedded in avionics systems, that keeps planes flying, keeping passengers, crew, and folks on the ground safe. I once discussed the opportunity to work on a terrain-following radar system, and one of the requirements for all software developers on the project was to be on board the aircraft while their software was being tested. You can bet that invalid input cases were included in those test plans.
Now, if you think the software you work on today is not that mission-critical – no lives or property will be lost if an untested invalid input causes some unexpected behavior – I understand your point of view. But I believe it is definitely important enough to test invalid inputs, no matter what it is you’re working on. Let’s say it’s a little game that your company sells. No harm to anyone, right? Now, let’s say that, when the user swipes some striped candies diagonally across a troll’s den (an invalid input), the game crashes and locks up the user’s device. They have to reboot the device, and they have lost whatever data they were working on in other windows/apps. You’ll have some pretty disgruntled customers, and perhaps even a few lawsuits over the data loss.
Will your company survive? Will anyone trust your company’s software again? Will people lose their jobs? Will the local economy suffer from your company’s woes? Will stockholders lose their retirement funds? Will you?
No matter how low you think the stakes might be, invalid input testing is a requirement, if you care at all about meeting the requirements set out for the software. The consequences are different, depending on the type of software you’re working on, but there will be consequences if you don’t do this type of testing.