Netflix has built a tool they call a Chaos Monkey. Its job is to randomly kill instances and services. This forces the developers to make sure their stem can fail smoothly and safely.