The thermometer is not the temperature
Introduction
Measuring things is essential for trade, construction and many other activities. However, for most of human history, accurate tools for the measurement of simple things such as length were difficult to craft and not widely available. It was only with the advent of the industrial and scientific revolution that mass-produced and truly accurate tools were developed. This inevitably led to the god-like precision that humans wield today to create such things as nano meters wide transistors available for billions of people. Being able to measure things easily and precisely then is probably one of the most fundamental abilities humanity has acquired.
With time, this ability has grown to encompass less concrete realities. While a meter is certainly a clear and undebatable concept, the cost of an apple is certainly more vague. Contrary to a meter, the price of an apple is a dynamic value that depends on the supply and demand. It is a much more volatile concept in the sense that the worth of something is not exactly an objective truth but depends on the agents acting in the market. Inevitably, while measuring a distance requires the simple action of using a ruler, measuring the value of something requires the continuous process of many interacting people engaged in transactions with each other. Hence, the price of an apple changes continuously, rising and falling constantly. The cost of things is definitely not a simple and clear concept.
Moreover, economists now start thinking about the environmental cost of commodities as well. Indeed, the environment also pays a price as non-renewable energy and pesticides are used to produce apples. However, we did not think of this cost before because we assumed our measurement of the price was complete. However, in practice, measures of complex and abstract things are often incomplete and the consequences can be problematic.
The thermometer is not the temperature
As we apply quantitative measurements to more and more things, we separate ourselves from the thing we measure until everything becomes mere numbers. At this point, we run the risk of forgetting there was even a thing behind the numbers in the first place. We thought the price of an apple was simply the money used in the transaction, now we realize that on a large enough scale, environmental costs matter as well. The problem then is that when one confuses the measure with the thing being measured, one risks ignoring the reality behind the measurement.
This problem can be seen in many places. Bubbles in the stock market are good examples where the price of something becomes unrelated to the actual thing. Inevitably however, reality comes back to haunt us, the bubble explodes and everything comes back to normal. But this can cause enormous damages as we have seen in the 2008 crisis. Another example is grades. While grades roughly correspond to the knowledge of a student. It is wiser to focus on learning rather than getting good grades. This is because while learning may get you good grades, getting good grades does not mean you have learned. They are obviously many examples like IQ : while IQ is a measure of intelligence it should be confused with intelligence itself.
Hence, while the measurement of a distance corresponds to the reality of the distance, grades for example inevitably simplifies the reality they measure. This is because the skills and knowledge acquired for a course are complex multifaceted concepts that cannot be reduced to a mere number. Yet we constantly reduce complex realities into simple numbers and forget about the hidden complexities and assumptions.
My personal experience
As a researcher, I strongly believe that we are not skeptical enough of measurements. We too often create measures and then forget what is behind them. For example, let’s say two artificial intelligence (AI) have learned to separate images of cats and images of birds. They both have the same accuracy of 70%. Does that mean they are equivalent?
A deeper investigation might reveal that one AI accurately learned relevant differences between cats and birds (such as eyes, limbs, ears, etc) while the other AI learned that photos of birds often include branches, leaves and the sky and photos of cats often include couches and interior decorations.
In other words, one AI actually learned the difference between the two animals while the other simply learned the specifics of the images given to it. However in practice, given new images in different environments, only one AI will have learned to correctly classify the images. Hence, if we only look at the accuracy of an AI on a standard dataset, we cannot always tell if an AI has learned correctly. In real life applications this can cause a myriad of issues down the line as real life is always more complex than the constrained scientific environment.
In practice there exist methods to alleviate these issues and verify what is behind the performance of an AI. For example, we can use error analysis to understand what kind of error an AI makes or attention based analysis to understand what an AI pays attention to to make a decision. However, these methods are imperfect, incomplete and time-consuming and as a consequence rarely used.
Conclusion
I hope that through this article I have convinced you that while simple measures of complex realities are useful they should be used with skepticism. The thermometer is not the temperature and one should investigate if the measure they are using accurately represents what they are measuring or if this measure isn’t hiding the truth from you.
Is your company looking to integrate AI solutions, analyze data, or strengthen its back-end development? Contact me!
Mail : pro.judicael.poumay@gmail.com
Portfolio : www.portfolio.thethoughtprocess.xyz
Spinoff project : www.aiashi.be
- The Transformer Revolution: How AI was democratized - 21 February 2024
- AI Model Distillation : how to compress a brain? - 15 January 2024
- Prompt Engineering: How to Talk to AIs like ChatGPT? - 22 November 2023
