I came across an amazing presentation years ago (which of course I can't find) that listed a bunch of kinds of failures for remote services that people usually don't test for.
In addition to timeout, 4xx, 5xx, etc, it listed things like:
- connection closes after 10 bytes of data
- returns contents of www.google.com
- returns contents of /dev/random
- returns contents of /etc/passwd
- returns correctly-formatted unicode chinese text
- returns ansi color control characters
- returns an incorrect content-type, labeled correctly (You requested Content-Type: application/json, I send back Content-Type: application/jpeg)
- returns one byte of data every 29 seconds
What are some types of "out-of-band failures" you've encountered that developers don't usually (but should) test for?
(extra bonus points if you can find the original presentation)
2 Answers
Answers 1
The ones you listed are great; I'd love to see the original presentation if you dig it up! A couple other favorites:
- A "valid" response with a couple bits flipped
- A "valid" response with extra data you weren't expecting (
{"result": 123, "extraStuff": {...}}) to simulate upgrades to the remote side - A syntactically-valid response that never ends (
{"results":["lol", "lol", "lol", ..., or just a bunch of whitespace)
Answers 2
Low-frequency failures. In other words, test that some response is correct not just once, but every time out of a thousand tries. You'll get random Internet breakage if you're going over a network, but you might expose some process is stochastic when you thought it was fixed.
0 comments:
Post a Comment