The Complexity and Entropy in a Complex Problem

How people evaluate the complexity of a problem

Evaluating the complexity of a problem involves understanding several interrelated factors that can influence an individual’s perception and experience. These factors include knowledge, measurement, predictability, and available resources.

Knowledge refers to the background information and expertise a person possesses, which can significantly shape their ability to navigate complex tasks.

Measurement pertains to how we quantify or assess the intricacies of a problem, often relying on metrics or criteria that may vary across different contexts.

Predictability relates to how foreseeable the outcomes of a problem are; more predictable problems tend to be perceived as simpler, while those with uncertain outcomes may be viewed as more complex.

Lastly, resources encompass the tools, time, and support available to an individual, which can either facilitate or hinder their ability to tackle complexity effectively.

By considering these elements together, we can gain deeper insights into how complexity is evaluated and managed in various situations.

Disorder and order

As the 2nd Law states, in a closed system the level of disorder tends to increase over time, leading to greater entropy, until the system reaches a state of equilibrium.

In our world, we observe that almost all systems tend to evolve toward greater disorder if not managed by external forces. You can sense that if you attempt to accomplish something on your own without any guidance or structure, things will eventually become chaotic and unmanageable. Ultimately, you may find that tasks become overwhelming and cannot be completed.

We also apply the concept of entropy to information systems — for example, when we use “cross-entropy” to train large models. This connection illustrates how uncertainty and disorder in data can affect the performance of algorithms. In machine learning, higher entropy often indicates more disorder in the model’s prediction distribution. Minimizing cross-entropy during training helps models learn to make accurate predictions, and this is precisely what we aim to achieve.

Intuitively, disorder and order are intricately linked, as increased disorder often results in heightened complexity. Making things well-ordered is essential to finding the balance and structure that achieve simplicity in problem-solving.

More disordered things are more complex. This is an instinctive feeling in our daily lives.

Complexity and simplicity

It is not easy to evaluate whether a work is complex or simple with an objective measurement. Complexity can often be subjective: what one person finds intricate and challenging, another may perceive as straightforward and manageable.

Factors such as prior knowledge, experience, and the specific context in which a task is approached can significantly influence this judgment.

Additionally, the tools and resources available to an individual can either simplify or complicate their perception of a task’s complexity. Thus, understanding complexity requires not only an analysis of the task itself but also an awareness of the individual’s unique perspective and circumstances.

Complexity is also hard to define. One informal definition is: complexity is the minimal amount of resources (information, energy, computation, or description length) required to specify, predict, or generate a system or problem.

The relationship between disorder and complexity

To explore this relationship further, we can consider that complexity often arises from the interactions and dependencies within a system. In many cases, as disorder increases, the number of possible states or configurations of a system also increases, leading to greater complexity.

For instance, in information theory a highly disordered dataset may require more sophisticated algorithms to extract meaningful patterns or insights, as the noise can obscure relevant signals.

Similarly, in physical systems, increased entropy can lead to more complex behaviors that are difficult to predict or control.

Moreover, in computational contexts, algorithms designed to handle complex problems often need to account for various sources of disorder, such as noise in data or variability in user behavior. This necessitates the development of robust models that can adapt to changing conditions while still providing accurate outputs.

In essence, while disorder can contribute to complexity by introducing uncertainty and variability, it is also essential to recognize that not all complexity is inherently negative. Some degree of complexity is necessary for systems to function effectively and adaptively. Therefore, understanding the balance between order and disorder is crucial for managing complexity in any domain — technology, biology, or social systems. Ultimately, this interplay shapes our approach to problem-solving and innovation.

Problem setup

Let’s set up a problem and take some observations from it.

You have a bottle of Coke. You open it, put some ice cubes into it, and close the bottle again. The main observations from this environment are:

• At the start: a clear boundary — solid cubes in liquid.
• In the middle: intricate, lace-like melt fronts, convection plumes, and bubbly paths — rich, multiscale geometry.
• In the end: temperature equalizes; the drink is uniformly cold and still. Few features remain.

Explanation by the 2nd law of thermodynamics

According to the second law of thermodynamics, the total entropy of an isolated system can never decrease over time.

Initially, when the solid ice cubes are placed in the liquid Coke, there is a clear distinction between the solid and liquid phases — a state of low entropy.

As the ice melts, energy is transferred from the warmer liquid to the colder ice, increasing disorder as the molecules in both phases begin to mix. This process produces intricate patterns and convection currents as heat flows and phase changes occur. Entropy increases.

Eventually, as equilibrium is reached, the system’s entropy maximizes, resulting in a uniform temperature throughout the drink with minimal features remaining. This progression illustrates how systems evolve towards greater disorder over time, aligning with the principles outlined by the second law.

Thinking in a simple mathematical way

The 1st law of thermodynamics tells us:

$\Delta U = Q - W$

where $U$ represents the total energy contained within the system, including both the kinetic and potential energy of its particles. $Q$ is the heat added to the system, and $W$ is the work done.

Let the energies of the Coke, the ice, and the melted ice be $U_{coke}$, $U_{ice}$, and $U_{melt\ ice}$, with $U_{total}$ the final iced Coke’s energy. Because energy is conserved,

$\Delta U_{\text{coke}} + \Delta U_{\text{ice}} + \Delta U_{melt\ ice} = 0$

From the entropy side, we can define the entropy of this system as $S_{sys}$. No matter what happened as the Coke cooled or the ice cubes melted, the entropy of this closed system increases monotonically until equilibrium, so

$\Delta S_{sys} \ge 0$

How about the complexity?

Complexity in this context can be understood as a measure of how the interactions between different components of a system evolve over time. Initially, when the ice cubes are added to the Coke, we observe a simple structure with clear boundaries between solid and liquid. This initial state represents low complexity because of the distinct phases.

However, as the ice melts and energy is exchanged, we witness an increase in complexity characterized by intricate patterns and behaviors — such as convection currents and bubbly paths — emerging from the interactions of molecules.

This transition from a simple to a complex state illustrates how systems can exhibit rich behaviors as they move towards equilibrium. The melting ice introduces variability and disorder, which in turn leads to more complex interactions among particles. As entropy increases, so does complexity; this relationship highlights that complexity is not merely about having many components, but also about how those components interact dynamically.

Moreover, once equilibrium is reached and the system stabilizes at a uniform temperature, we see a reduction in observable complexity. The intricate patterns dissolve into a homogeneous state where fewer features are present.

This suggests that while complexity can arise from disorder and interactions, it can also diminish as systems reach stable states.

Conclusion

In a closed system, we observe that entropy increases monotonically, but complexity does not. As the system becomes disordered, its complexity rises and reaches a maximum before decreasing.

The Kolmogorov complexity

Kolmogorov complexity gives us a way to describe the complexity of a problem from the information-theory side.

Kolmogorov complexity (KC) quantifies the complexity of a string or dataset by determining the length of the shortest possible description — or program — that can generate that string using a fixed computational model.

In essence, it measures how much information is required to represent an object or problem.

Explaining the iced Coke

Can we use KC to explain the iced-Coke experiment?

Yes — in a coarse-grained way, we can apply Kolmogorov complexity by analyzing how the complexity of the system evolves as ice melts in Coke.

Initially, when ice cubes are added, we have a relatively simple state characterized by distinct phases — solid ice and liquid Coke. The Kolmogorov complexity at this stage is low because we can describe the state with a straightforward representation: “ice cubes in Coke.”

As the ice begins to melt, energy transfer occurs, leading to increased molecular interactions and disorder. This transition introduces more intricate patterns and behaviors, such as convection currents and bubbles forming within the liquid. The Kolmogorov complexity increases during this phase because a more detailed description is required to capture these dynamic interactions and emergent phenomena. We might need to describe not just the presence of ice and Coke, but also how they interact over time — temperature changes, molecular movement, and energy transfer.

Eventually, when equilibrium is reached and the system stabilizes at a uniform temperature, we observe a reduction in observable complexity. At this point, while entropy remains high due to disorder, the Kolmogorov complexity may decrease again, because the system can now be described simply as “uniform iced Coke.” This illustrates that while entropy tends to increase monotonically in a closed system, complexity can fluctuate based on how components interact dynamically throughout the different phases of the process.

The complexity and the KC

We can explain this phenomenon using Kolmogorov complexity, but KC is just a theoretical construct that does not account for all aspects of real-world complexity.

In practical applications, KC may overlook factors such as environmental influences, external interactions, and the inherent unpredictability of complex systems. It also does not account for the temporal dynamics and contextual variations that can significantly affect how complexity manifests in real-world scenarios. Thus, while KC offers valuable insights into the nature of complexity, it should be complemented with other approaches to fully understand the multifaceted behaviors observed in dynamic systems like the iced-Coke experiment.

As a human feeling, the system’s complexity rises rapidly — but KC increases only at a logarithmic speed, which is deeply unintuitive. There should be some other latent essence that shapes our perception of complexity.

---

Reference: Scott Aaronson — The First Law of Complexodynamics