Introduction #
This post deals with my thoughts on experimentation as it relates to myself and tech teams. It is the second in this series as I seek to more accurately understand the concepts of maintenance and experimentation as they apply to both myself, and the wider world.
Experimentation #
Experiment:
- a scientific procedure undertaken to make a discovery, test a hypothesis, or demonstrate a known fact.
Do we experiment? What is the true meaning of this word? Experimentation in a strict sense is a scientific undertaking that adheres to rigorous methods and processes so results can be cast into academic journals. However, to some degree I think we all can experiment and this is where the meaning of this word can go astray as we do not closely examine what the intent of our experimentation is.
To clarify my already vague meaning of experimentation: given a certain set of circumstances, when I perform X following a specific set of methods, I expect (hypothesise) that Y will occur. Other important things to note: 1) Y may not indeed occur, so it means I need to check my process and importantly there is something else to learn, 2) as much as possible the experiment should yield the same result given the same process.
Most Start-Up companies I think pretend to experiment by testing markets with new products; and you may experiment with buying a particular crypto currency to see if you can make money; or seeing if using SaaS application X improves employee retention. I would contend that all these scenarios are not experiments as they have a goal or a desired state and do not inherently seek discovery as their main undertaking. An experiment should be something where first and foremost it is the truth of a contended hypothesis that is at stake. This means you are fallible and must accept when outcomes do not fit your hypothesis - one must need to be able to say “I was wrong.” Finding the time and energy to perform experiments is quite rare in today’s busy world, but I think you should do it.
For example, I really like Mexican cooking right now. It is one of my passions and it gets me into the kitchen making new things. I have read lots of recipes and watched a lot of videos to help me improve this culinary style. By cooking the same recipes repeatedly I allow myself to make small experiments where I test if, for example, refried beans taste better if I leave them to soak, or just start cooking them. In this way I ascribe some controls to my experiment and test the outcome where the observed outcome is the goal. I do not care if the result doesn’t hold to my previously held intuitions. I observe and think of the next thing to test. And this is very similar to how most businesses operate today with regards to experimentation….
Except they do not.
Outside of an application teams potentially testing whether customers like the new shade of blue or teal on their buttons for two test groups when was the last time a company you worked for or managed performed some controlled test purely to discover an outcome. Here, you could make the argument that proof of concept or PoCs as they are known are experiments, except that in most cases they are the thin end of the wedge for further adoption of a product. If business goals were no obstacle my inner mad scientist would have development teams using different sets of tools to see if one team performed more software releases or had less downtime in their systems. By testing things in a complex setting you could make observations and draw some conclusions as to what works best given your current circumstances.
Businesses could certainly make the argument that such experiments would be costly and potentially not inline with the overall mission of their objectives. A banking company that uses technology doesn’t necessarily need to be at the forefront of testing new technologies or software development processes. But then again, I am not a chef but I see the value in experimenting with food to find methods that make the food taste better or allow me to be more organised in my assembly of a meal. My current thoughts are experimentation is a metaphysical muscle people and truth seeking collective possess that improves with use and atrophies with neglect. In appreciating that inherent capability in ourselves we learn to think differently about our own reality through the process of testing, observation and being happy with being wrong, rather than merely following what other “smart” people are doing or recommending so we appear smart and not wrong.
You must look… [2]
I also think there is an interesting emergent characteristic of experimentation where it is more valuable as a skill the more complex the system is that you are trying to observe and specific the hypothesis is to you. This is where I don’t understand Computer Science as a science in terms of the process of using experiments, controlling for variables X and Y and observing the outcome. Most research in Comp Sci as I understand it is deriving algorithms from mathematical proof rather than through observation in some scenarios. Here, you could say that the system setup to test the algorithm is so controlled that observation in some real world scenario is unnecessary; reality has been abstracted away. Conversely, when there is a need to observe the outcome of sugary foods on a large population, as an example of a highly complex system, the value of considering and setting up well thought through experiment is important.
Regarding the value of experimentation as it relates to your own questions: as I previously mentioned with my Mexican food example, experimentation is a great way to help you discern little truths of the world around you in small but meaningful ways. No one else is going to care or be affected if my experiment to make better beans by soaking them for x many hours, but to me this small test matters a lot, as I am a lot more sure that it made no difference, and perhaps I need to test if adding more bacon will make it taste better.
The graph above is the focal point of this post and like the figure in my previous posts the units of measurement are arbitrary. The main goal is to describe a relationship between proximity, complexity and perceived benefit. Proximity in this case is a score that seeks to measure how close you are to the construction and observation of outcome of the experiment. An experiment involving ice weasels in Iceland that you hear about in a newspaper would have very low proximity, and at the other end experimenting with your own cooking would have very high proximity. Complexity is a measure of the amount of variables or environment factors that can affect the results. Benefit is the subjective measure of truth from the experiment that is realized to you, and I think it derives from the results being able to add to your understanding of your own situation and allows for further fact finding.
Here, I would also add that experiments are best when they are most valuable to your truth seeking unit. To explain, it would be unfair or unethical to experiment on a large segment of the population that does not possess the correct understanding in order to give valid consent and when the outcome of that experiment only really matters to a handful of scientists. In my previous example of having two development teams test two different sets of tools the whole team would be informed about the experiment and the observable outcome would be potentially valuable to each member of the team so that it furthers their understanding of the question at hand. This cohort of developers could be thought of as the truth seeking unit and I think it is important to establish experiments with this in mind as history has many examples of experiments being both dangerous or immoral.
Without the ability to perform experiments I feel like most of our operational understanding of our world will be like that of an Large Language Model (LLM). The next best thing to gain information will be to talk to an AI to get seemingly plausible answers. Our reality will be as constrained as all the online information on the internet, rather than a potentially infinite understanding being experienced by all individuals and groups in whatever part of the world we exist in. Cynically I think that it is plausible that companies will brandish the intuitive guesses of a well trained, and spoken AI over the facts they see in front of them. But the truth of observation and the reasoning that comes with that is powerful, as when someone insists they have done something due to a well observed set of phenomena you take notice, and less and less notice is given to ideas purely generated from books or search results, as well credentialed as they may be.
Conclusion #
I discussed how my approach to experimentation is not an overly formulaic one, but the utility you can get in a less academic setting is the same - to discover information. Businesses do not often experiment in my view, but should as it would allow them to make their very own discoveries. Experimentation is a process to make sense of the world we find in front of us that is not just theorized and that process is made easier simply by doing it often like exercise. Proximity to experimentation is an important consideration, those who participate should understand and receive a benefit in the results.
References #
[1] - Mr Bean - Art Scene