Decision making in black box scenarios

Making technology decisions on well-documented and stable systems can already be challenging, but how about opaque ones – when we deal with a black box? As technology leaders, we almost always operate within constraints – but a lack of general understanding of a system is perhaps the most challenging (especially under time pressure and if the system in question is mission-critical, such as payments).

– Legacy software: When we inherit software written by other people, we rely solely on adequate documentation, which is unfortunately rare.

– Senior engineering talent leaving: Even if they wrote good documentation, understanding complex systems takes time.

– Poorly written systems.

Of course, we can understand any black box if unlimited time is available, but that’s seldom the case. Yet, there are several things that we can do:

– Measure outputs and inputs: While the systems can be opaque, what goes into and out of them is not. Analyzing those points can yield valuable insights into the system’s inner workings.

– Adopt a scientific approach: Conduct experiments, test hypotheses, and document the results. Observe the behavior of the system patiently over time.

– Set up feedback loops: While running the system, prod it and observe any changes in behavior and performance. Isolate subsystems: Break down the black box into separate components and measure their inputs and outputs instead of the system as a whole.

– Replicate: Replace different components piece by piece, culminating in a complete copy.

Those solutions will help engineering leaders guide their teams around the frustrations and dangers of working with black box systems.