As Testers, How Do We Know What We Know?

One of the central questions—if not the central question—of software testing is "What do we know about this piece of software?" When we know enough about the software under test, we can stop testing it. But, how do we acquire knowledge? Given that our perceptions are sometimes unreliable, can we actually be certain of learning anything through the senses? How about through intuition or feelings? Are there some things we can't know? Epistemology is the branch of philosophy that studies knowledge and attempts to answer questions such as these.

Building Blocks: Knowledge and Truth

For a concept so fundamental, philosophers surprisingly don't agree on a single definition of “knowledge.” That said, the most widely known (though, again, not universally accepted) formulation is that knowledge is justified true belief.

Let’s take these one at a time. For some proposition to count as knowledge, it requires:

• Belief: You have to be aware of the proposition and, further, think it to be true.
• Truth: It has to actually be true.
• Justification: There has to be some reason for believing it to be true.

Theories of Perception

Because the ways in which we can gain knowledge about the universe are fundamentally dependent on how we believe the universe is constructed, epistemology builds on and is closely related to metaphysics.

We perceive the world through our senses and our mind, but these perceptions can be mistaken. In dreams we perceive things that do not exist. We may hear a noise but find out other people in the room have not heard it. Two people may witness an event yet not agree on what actually happened. It seems that we cannot rely on our senses for knowledge. How, then, are we to know anything about the world?

As one might expect, the answer to this question depends heavily on how the philosopher answering it constructs the world. Plato, an early Greek philosopher, asserted that true reality does not consist of the physical world around us, but of forms—perfect, abstract, immutable concepts such as beauty, harmony, and blue. We can't perceive the forms directly; instead, we see them manifested (albeit imperfectly) in things like beautiful landscapes, harmonious music, and blue skies.

Plato believed that we are all born with intrinsic knowledge of the forms but that this knowledge is latent. In his view, true knowledge comes from becoming acquainted with the forms, not by carefully studying their imperfect manifestations in the physical world. Since the forms are perfect and our knowledge of them is innate, perfect knowledge of some area of expertise is something we can hope to attain.

Idealism and Realism

Broadly speaking, later philosophers tended to take one of two different approaches to the problem of perceiving reality: that of Idealism or Realism. Idealism answers that our perceptions are reality. Objects exist only as bundles of sensory data in the minds of those who perceive them. Our perceptions are all we can be truly certain of, and they are the measure of all things.

Realism, by contrast, declares that reality exists independent of perceivers or perceptions. Direct realism says that we can directly perceive the world through our senses and thereby form an accurate impression of it. However, we already know this position to be problematic, because our senses are unreliable. Indirect realism says that, despite the fact that an external world exists, we cannot perceive it directly. We only know what our senses tell us about the world, and those perceptions can be valid, incorrect, biased, and so forth.

Both Realism and Idealism seem at least somewhat plausible. The problem is that neither leaves us with a satisfactory explanation for how we can know anything! Our conventional conception of knowledge is that anything that we can know should be the same to everyone, and we should be able to know it with certainty. Idealism grants knowledge that is certain for you but not everyone. Realism yields knowledge that can be shared amongst everyone, but since we cannot be certain that our perceptions accord with reality, our knowledge can never be certain.

Rationalism and Empiricism

Given this problem of perception, then, by what means can we acquire knowledge? Rationalism answers that we can gain knowledge solely through the use of reason. For instance, without making any observations of the world, we can formulate the idea of a triangle and describe its characteristics without having to resort to the use of sensory input. We can make logical concepts defined in our own minds agree with each other without reference to any external phenomena.

Empiricism, on the other hand, argues that we must rely on our perceptions (imperfect though they may be) to acquire knowledge. We come into the world as a blank slate, without any ideas on which to build our knowledge, and must experience things such as triangles before we can begin to reason about their qualities.

Logical positivism—also known as logical empiricism—holds that these two viewpoints are not necessarily mutually exclusive. Reason alone can demonstrate the truth of abstract constructs such as logic and mathematics, while controlled observation of the external world leads to knowledge through being able to verify one's theories about it.

The scientific method is based upon logical positivism, amongst other philosophical perspectives. To learn about the world, we must assume that it behaves by consistent rules and that we can observe the results of testing these rules. We think of knowledge and the truth of theories in terms of reproducibility and demonstrability. A theory is "true" to the extent that it lets us predict the observed result of some test or set of actions. As testers, we form theories about software and perform experiments to validate or refute these theories.

Takeaway Points for Software Testing

How does this inform our testing strategy? Historically, the more we've studied epistemology, the less sure we've become of our knowledge and our ability to acquire it! We started off with at least the hope of attaining perfect knowledge and ended up with the realization that we can never know anything about the external world with complete certainty. What benefit, then, have we drawn from several millennia of contemplating epistemology?

First, we can draw a distinction between two categories of knowledge: the realm of logical constructs, about which we can reason without resorting to the senses world, and all other knowledge, which requires observation of the world to obtain.

Second, our perceptions can be mistaken, so any knowledge that depends on our perceptions can be flawed. Imperfect perception is a reality of being human, so we shouldn't dwell on an occasional perceptual error.

Third, the truth of a theory about how the world works is measured via repeatability. A theory is true to the extent that it can predict reproducible results. A theory that we cannot test is one that we cannot obtain knowledge about.

Fourth, we can refine our knowledge by using the results of our experiments to refine our theories. We then test our revised theories, leading to a virtuous circle in which our knowledge becomes more accurate.

Finally, we can never be completely certain in our knowledge. We don't know with complete certainty that the sun will rise in the east tomorrow; we merely have an extremely high confidence that it will, based on our having observed it having done so and the fact that it seems to conform to our tested theories about how heavenly bodies work. Scientific theories often hold up for many decades or centuries but are later discredited in the face of more complete knowledge. Thus, we must never close our minds in thinking we know something, but always be open to new information that alters what yesterday we deemed well-nigh certain.

 Check out the other articles in Rick's series on philosophy and software testing:

• Philosophy and Software Testing
• The Case for Ethics in Testing
• Testing Metaphysics
• Logic in Software Testing
• Is It Beautiful? Aesthetics in Software Testing