Testing What, Exactly?

Artificial intelligence is an area of research that fascinates a lot of people. I’m not entirely sure why. It seems to me that promoting human intelligence would be a far more useful activity. But I digress.

In dissecting the article on David Cope (see previous blog entry), I made a passing reference to the Turing test. That got me curious. The Turing Test is supposed, by some people at least, to address the question, “Can machines think?” The way the Turing Test works is this: You’re having a conversation with an entity, and you don’t know whether the entity is human, or whether it’s a computer. The “conversation” is held by typing words into a computer, and reading the entity’s responses on the machine.

The thesis of the test is that if you can’t tell whether the entity with whom you’re conversing is human or a computer, the entity is exhibiting intelligence. It has passed the test. Several serious criticisms have been made of the Turing Test. (If you’re interested, read the Wikipedia article. That’s what I did.) Basically, it has been blown out of the water. It isn’t even worth discussing, except that people keep bringing it up because it’s simple and obvious and seems, at first glance, to be a meaningful benchmark.

I claim that I could beat the Turing Test with any software program ever devised. In less than 30 seconds, I could tell whether I was dealing with a human or with a machine. I would type the following message: “Hey, I just won 2.6 million dollars in the lottery!!! I’d like to share the wealth. I’ll write you a check for $10,000 today, but only if you’ll meet me for lunch. Where would you like to go for lunch?”

My suspicion is that if I typed that text, the experimenter would stop the test. They would say, “Hey, that’s not fair. You can’t test the entity by inviting it to leave the testing environment. You’re supposed to restrict your conversation to X, Y, and Z, and only typing is allowed.” In other words, you’re not allowed to interact with the other entity as if it were a real human being.

That’s another reason, one that has apparently been missed by other critics, why the Turing Test is a fraud. An artificial set of conditions is set up, and the test can produce results (supposedly, reliable ones) only within the artificial parameters of the test.

This is a bit like testing whether humans have depth perception by forcing them to view test objects out of one eye at a time.

It’s also a dandy example of reductionist thinking. Scientific research has thrived on reductionism for hundreds of years, but we now seem to be reaching a point where reductionism no longer works. The simple questions, the ones that can be answered using simple artificial tests (Galileo rolling balls down an inclined ramp, for instance) have all been answered. What we’re now grappling with are questions so complex that they can only be answered by looking at the test subject in its native environment. To move forward, we need to look at all of the factors that may be coming into play.

The design of the scientific test itself is always one of the factors. It can’t be left out of the analysis.

This is why things like testing new drugs before releasing them have become so difficult. To test a drug, you have to give it to human subjects and then observe what happens. But once a chemical enters the human body, it can affect any of the thousands of metabolic processes that are taking place in the body. And the effects may not show up for months, or years. One possible outcome: Thalidomide babies. Not only that, but the drug can interact with any other drug that the person is taking, and also with any chemical that the person comes into contact with in their environment.

Scientists have gotten pretty good at predicting some of these effects and interactions, but of course they make many mistakes. It’s not their fault. The problem is not inept scientists; the problem is that reductionist scientific testing methods just don’t answer complex real-world questions very reliably.

Holistic thinking got a sort of bad rap in the Seventies, because a lot of hippies were trying to apply it in simplistic ways. But I think we need to go back there and have another look.

One of the huge problems today in what passes for political discourse in the United States is that simple solutions are proposed (mainly by right-wing knuckle-draggers) for complex problems. This is reductionist thinking in full flower. But that’s a topic for another time.

I wonder if kids today are being taught in high school how to think about complex problems. I hope they are, because they’re going to have a bunch of them.