Welcome to the A.L.I.C.E. AI Foundation

Promoting the adoption and development of Alicebot and AIML free software.

Notes on the Loebner Contest Grand Prize Rules

Dr. Richard S. Wallace
1 December 1997

Dr. Richard S. Wallace


This document was submitted as a position paper for the Human Computer Conversation workshop being held in conjunction with the January, 1998 Loebner Competition. Hugh Loebner, the founder of the Loebner Prize in Artificial Intelligence, has received much crticism for his decision to extend the "Turing Test" to audio-visual media. This essay suggests groundrules for this "Grand Prize" which attempt to pacify the critics by reviving the spirit of Turing's original Game.

The Turing Game and the Loebner Prize

The notion that the "Turing Test" may be expanded to audio-visual media rests on the mistaken, if widely held, assumption that there is such a thing as a "Turing Test" in the first place. In Turing's 1950 paper "Can Machines Think" he described an elaborate scenario that he called the "Imitation Game" involving a man, a woman and third person who may be of either sex. In his original game, the third person, who is called an interrogator, communicates with the couple through a restricted medium much like what we now call text-based chat. The object of the game is for the interregator to guess the gender of each of the other players. In order to confuse the interegator, Turing specified that the man should decieve or lie and the woman should tell the truth. Now the role of the computer, as Turing wrote, is to replace the man, i.e. the deceiver, in the game. When a machine can imitate the man in the game, so that the interregator identifies the computer as the "man" as often as he would the real man, then that computer is said to be thinking in a restricted sense.

The Imitation Game by Alan M. Turing, as it originally appeared in jounral Mind in 1950 (Volume LIX).
One wonders what history diluted the original "Imitation Game" into the caricature called the "Turing Test" in most engineering schools today [4]. Most authors (and teachers) mistakenly understand the "Test" as a chat-like scenario involving only two players, in which the experimenters attempt to replace one of them with a computer. This description of the Turing Test forms the basis of the Loebner competition rules today [7]. No attempt has yet been made to play a Turing Game as Turing originally described it.

The critics [1][2] of the Loebner Contest Grand Prize may have a point when they claim that the Audio Visual competition is far from what Turing imagined, but then so is the 2-player "Turing Test" itself. Are we to conclude as some have that the Loebner contest, and especially its AV extension, is misdirected, a diversion of attention from the "real issues" of AI?

An Audio Visual Turing Game

No argument will silence all the critics, but here I'd like to advance some suggestions for the groundrules of the Grand Prize [3] that preserve the excitement and novelty of the audio-visual experiment but also rehabilitates the "Imitation Game" as Turing imagined it.

Today we are immersed in the world of e-mail and chat-line communications and many people are familiar not only with the basic medium but also of its many pitfalls: among them that the person on the other end may not be the gender they claim to be. The extension of the Imitation Game I propose to audio-visual media looks slightly ahead of today's (mostly) text-based chat world to one where multimedia communications permits real-time transmission of at least low grade video imagery. One might be tempted to think that video-based chat would eliminate the pitfall of gender deception. But there are any number of video and graphic "tricks" available to deceive the receiver into thinking that the image of a particular face goes with a particular gender, when in fact it might not.

The audio-visual imitation game likewise involves three players: a man, a woman, and an interregator of either sex. The interregator is involved in a 3-way audio/video chat with the other two players. For the contest, we should consider limiting the frame rate and the resolution of the video image so that absolute synchronization of sound and video is not necessary. When the computer plays the role of the man, the interregator will see a video image of a man graphically manipulated as necessary to produce the illusion of the actual male participant. No restrictions need be placed on the content of the dialogue or of the video imagery.

Final Thoughts

Before concluding this essay I would like to add three more remarks regarding the demographics of the judges, the "no-telecommunications" rule and a final point about consciousness and AI.

Some controversy has surrounded the selection of the judging panel for past Loebner contests. Part of the problem is that deciding who is qualified to judge AI requires some assumptions about the nature of AI itself. Our experience with the ALICE program is that the client's experience varies widely according to his background, but because of the exposure of the program to clients on the web, we have amassed a remarkable corpus of what people actually try to say to a computer. Our strategy is to develop ALICE in response to this distribution of client inputs. As the Web clientele increasingly reflects the demographics of the general population, we expect that the distribution of client queries follows a popular conceptualization of what an AI should be expected to do. It follows that the ALICE program "works better" for the general population than for select subsets. Judges for the Imitation Game ought to reflect these general demographics, and for that reason it seems appropriate that a "jury of peers" include some young and elderly people, some with little formal education, and few if any academics.

The "no-telecommunications" rule ought to be abandoned. The emergence of the World Wide Web makes possible the development of elaborate client-server programs dispersed over a wide area [8]. In the case of ALICE for example, the telerobotic "eye" is located in a lab in Pennsylvania; the audio speech-generation is derived from a Bell Labs site in New Jersey [6], and the natural language server may reside anywhere at all on the worldwide Internet. Bringing these components together in one place is costly and perhaps prohibitively expensive to some would-be competitors. In place of the rule we ought to consider a contract requiring prize forfeiture if any "hanky-panky" is discovered. In practise it seems unlikely that an elaborate hoax would have much chance of success, especially for the Grand Prize.

Finally, this essay concludes with a brief remark about consciousness and AI. Many have argued [9] that in order to make progress AI must look beyond simple "Eliza-like" transformations and toward something else, usually not well defined, called "real intelligence". If anything the reductionist approach of ALICE, who has been called "Eliza's Big Sister" and "Super-Eliza", has shown that much more can yet be mined from this simplistic approach. ELIZA has been called a "hoax" and perhaps rightly so, but is the conclusion that she is not "real AI"? This author at least is now prepared to consider the inescapable alternative explanation: "Real consciousness" is itself a deception [5]. Turing had a glimmer of this notion when he envisioned the computer in his imitation game playing the role of the deceiver. The boundary between real intelligence and deception has never been less clear.


  • 1. [A HREF="http://www.diemme.it/~luigi/talk.html"] How to Pass the Turing Test by Cheating
    [/A] http://www.diemme.it/~luigi/talk.html
    Jason L. Hutchens
  • 2. Lessons from a Restricted Turing Test
    Stuart M. Shieber
    LANL Report-no: CRCT TR-19-92
  • 3. [A HREF="http://acm.org/~loebner/In-response.html"] In Response [/A]
    Hugh Loebner
  • 4. [A HREF="http://dragon.acadiau.ca/~001741c/project/turing.html"] The Turing Test [/A]
    Dr. Andre Trudel (course notes)
    [Note: this URL has a nice illustration of the "Turing Test" as it is commonly misconceived.]
  • 5. The Lying Game
    Richard S. Wallace
  • 6. Bell Labs Text to Speech Synthesis
    John Holmgren and Michael Tanenblatt
  • 7. Home Page of The Loebner Prize--"The First Turing Test"
    Hugh Loebner
  • 8. SETL for Data Processing on the Internet
    David Bacon
  • 9. Personal communication, David Powers