Abstract code representation

Many models of reaction time assume that performance can be understood in terms of the formation of mental codes. That is, the process of identifying the stimulus in a task is understood as a process of forming a mental representation of the stimulus, or a stimulus code.  Similarly, the process of selecting a response in a task is understood as a process for forming a mental representation of the desired response, or a response code.

Stimulus and response codes represent abstract properties of stimuli and responses in a task. For example, spatial stimuli are coded in terms of their location relative to the other possible locations in the stimulus set, rather than their absolute location in the visual field; similarly, spatial responses are coded in terms of their desired outcomes, relative to other outcomes in the response set, instead of in terms of specific actions or muscular movements.

Simon and his colleagues (Craft & Simon, 1970; Simon, Craft & Small, 1971; Simon, Small, Ziglar & Craft, 1970) demonstrated that the spatial S-R consistency effect in a Simon task depends on where you perceive the stimulus to be, and not on what ear, eye or visual hemi-field is actually stimulated. Umilta and colleagues (Umilta & Nicoletti, 1985, exp. 2 and 4; Umilta & Liotti, 1987, exp. 3) later showed that a spatial S-R consistency effect can also be found for relative stimulus positions: that is, when both stimulus positions appear on the left side of a display, but one stimulus position is farther left than the other. These experiments indicate that stimulus codes are formed based on your perception of the current irrelevant stimulus relative to other elements in the set, rather than on physical sensory stimulation or absolute stimulus position.

Other experiments have shown that response codes are also represented in terms of abstract properties, such as the desired outcome of the response. For example, when the subjects’ goal is to “press a key” in the Simon task, performance is determined by the relationship between the stimulus position and the response key position, even when subjects crossed their hands (Simon, Hinrichs & Craft, 1970; Wallace, 1971), cross their fingers (Riggio, Gawryszewski, & Umilta, 1986), or press keys using only one finger from one hand (Bauer & Miller, 1982), or using two fingers from the same hand (Heister, Ehrenstein & Schroeder-Heister, 1987). However, when pressing a key also lights up a light on the opposite side (i.e. pressing a left key lights up a light on the right side, pressing a right key lights up a light on the left side), and subjects are told to respond by “lighting up a light” rather than “pressing a key,” performance is determined by the relationship between the stimulus position and the light position. That is, when the stimulus and the light position are on the same side (but the response key is on the opposite side), subjects respond faster, while when the stimulus and the response light are on the opposite side (but the response key is on the same side as the stimulus), subjects respond more slowly (Hommel, 1993a).

The assumption of abstract code representation allows coding models to explain the appearance of consistency effects under a diverse set of stimulus and response conditions using a single explanatory mechanism (see Nicoletti & Umilta, 1984; Riggio, Gawryszewski, & Umilta, 1986; Umilta & Nicoletti, 1990). The coding model framework leaves open for debate the question of how mental codes are formed (e.g. Heister, Schroeder-Heister & Ehrenstein, 1990; Proctor, Reeve, & van Zanddt, 1992; Rubichi et al., 1997; Stoffer, 1991; Stoffer & Umilta, 1997; Weeks, Chua, & Hamblin, 1996); however, once it is taken as given that appropriate abstract mental codes are formed, these models then only need to specify how the relationship between different mental codes influences performance in order to explain the effects in all of these diverse task conditions.

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

Parallel identification processes

Most models of reaction time based on the formation of mental codes assume not only that separate mental codes are formed for relevant and irrelevant stimulus features, but that the codes are formed in parallel by separate processes. Whenever a stimulus is characterized by more than one dimension (i.e. whenever there is more than one stimulus set), each stimulus dimension can be understood as a separate functional stimulus (Miller, 1988), from the point of view of the perceptual system.

Thus, instead of there being a single stimulus identification process, each dimension of the stimulus is identified by its own separate process, or separate “channel.” These processes can operate completely concurrently, and they do not depend on one another for information. (There is debate, however, as to whether there is “cross-talk” between different stimulus identification processes that are going on at the same time; see, e.g., Egeth, 1977; Estes, 1972, 1982; Mordkoff, 1991; Morton, 1969).

The assumption of multiple identification processes was developed by Eriksen and colleagues (Eriksen, 1966; Eriksen & Lappin, 1965, 1967; Eriksen & Spencer, 1969) for displays with multiple elements, such as flanker stimuli. They suggested that display items that are presented in different spatial locations are identified through separate and independent processes that act in parallel. The idea that the stimulus dimensions of color and word are processed by separate “perceptual analyzers” has also been assumed in even the earliest accounts of performance in the Stroop task (e.g. Morton & Chambers, 1973; Posner & Snyder, 1975). These assumptions were brought together to form the general assumption of separate stimulus identification processes or “channels” (see Egeth, 1977; Miller, 1988): when a stimulus consists of multiple dimensions, they form separate functional stimuli, and are processed by separate stimulus identification processes.


 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

Stimulus codes form automatically

One of the assumptions shared by all coding models of reaction time is that mental codes for irrelevant stimuli are formed automatically, even though they are not necessary to carry out a task. Coding models explain consistency effects in terms of a match or a mismatch between this irrelevant stimulus code and one of the mental codes required to perform the task.

This kind of explanation is very general, because mental codes can be about anything. As a result, any kind of irrelevant stimulus can give rise to a consistency effect: letters, words, locations, colors, and so on.  What these models must specify is exactly when and how irrelevant stimulus codes influence the formation of one or more of the mental codes that are required to carry out a task.

Wallace (1971, 1972) first suggested that the S-R consistency effect in a Simon task appears because people automatically, involuntarily form a spatial code, even though the stimulus position is irrelevant. Eriksen and Eriksen (1974; see also Eriksen & Schultz, 1979) similarly suggested that flanker letters in a Flanker task are identified (forming their own letter codes) even though they are known to be irrelevant to the task. This view has been generalized since then to apply to any irrelevant stimulus characteristic, and is an assumption made by all coding models of consistency effects.

Irrelevant stimulus codes form automatically, and influence the formation of other mental codes.  Many coding models assume that mental codes form gradually, and that selective attention will eventually suppress the formation of the irrelevant stimulus code once it is identified as irrelevant. This mechanism of attention produces a rising-then-falling, or inverted U-shaped activation of the irrelevant stimulus code.

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

What are mental codes?

Consider what has to go on in your mind in order for you to carry out the instructions for a typical Choice Reaction Time task, such as: “press the left key when you see the color green and the right key when you see the color blue.”

When a stimulus appears, at least three things have to happen: 1) you have to figure out the color of the stimulus, 2) you have to decide which key to press, and 3) you have to actually press the key. These are usually considered the three most basic, broadly-defined processes involved in carrying out a task, and are usually called stimulus identification, response selection, and motor programming, respectively (although they can be broken down into more specific sub-processes, as well; see Sanders, 1980, 1990).

These processes can be described more concretely in terms of information and mental codes. Your senses give you signals that contain information about what is going on in the world around you. In order to understand and react to the world, you use that information to create a mental picture of your environment. In other words, you form a stimulus code: a mental representation of what properties are in the stimulus environment that produced the sensory signals that you received.

Stimulus identification can be thought of as the process of forming stimulus codes based on sensory information. Those stimulus codes, in turn, contain information that can be used to decide on a response. In order to act on the world, you form a response code: a mental representation of what actions you want to carry out. Response selection can be thought of as the process of forming response codes based on information from stimulus codes. Finally, motor programming is the process of using information from response codes t prepare specific muscular movements that carry out your response. This can be thought of as the formation of motor codes, which are the programs your muscles use to make a response.

The idea that thought and action in the world consists of mental codes (representations of stimulus properties and response actions) is called the information processing approach, and models of performance based on this framework are called information processing models (see Anderson 1995; Bower 1975; Miller, 1988). Performing a task requires transforming information from the world into a stimulus code, a response code, and then a motor code, through a sequence of mental processes.

These processes clearly depend on one another. In the example above, what key you press depends on what side (left or right) you decide is correct, and what side you decide is correct depends on what you think the color of the stimulus is. In the language of information processing models, the output of stimulus identification, which contains information about the stimulus code, is used as the input for response selection. Similarly, the output of response selection, which contains information about the response code, is used as input for motor execution. Information processing models use terms like “input” and “output” a lot, because they were originally motivated by the idea that mental processes are like computer programs, and mental codes are like computer data (see Newell, Rosenbleem, & Laird, 1989; Simon, 1981; Simon & Kaplan, 1989).

Psychologists want to know exactly what is going on in these processes; that is, how information is represented in these codes, and how they are actually formed. One way to approach this question is to measure people’s performance, their speed and accuracy when carrying out a task, under different kinds of task conditions. The amount of time it takes for you to make a response is related to how difficult each of these processes is: when something about the task makes your response faster or slower, it is because one (or more) of these processes has been helped or hindered. By examining how different kinds of task conditions influence performance, psychologists are able to get an idea about what is actually going on in the formation of these different mental codes. This approach is called mental chronometry (see Meyer et al., 1988; Sanders, 1993).

There are a number of specific questions one can ask about the formation of mental codes during choice reaction time tasks. Is input information compared to items in memory one by one, until a match is found? Or is the input information compared to all possible items in memory at once? Does input information for a process cause mental codes to form gradually, or do mental codes form in chunks, like “yes” and “no” decisions? Does incomplete information get used by later processes, or do they have to wait until the previous process is completed?

Even more questions can be asked about consistency effects in classification tasks. How does irrelevant information affect the formation of mental codes? Does it influence the formation of stimulus codes, response codes, or motor codes? Does irrelevant information always have the same kind of influence on mental codes, or does it depend on task conditions?

Today, most models of consistency effects share a few basic assumptions about mental codes and how they behave during classification tasks (see, e.g., Barber & O’Leary, 1997; Kornblum et al., 1990; O’Leary & Barber, 1993; Lu & Procter, 1995; Prinz, 1990; Proctor, Reeve, & van Zandt, 1992; Umilta & Nicoletti, 1990; Wallace, 1971). For example, they assume that irrelevant stimulus codes form automatically, that different stimulus features are formed by multiple parallel identification processes, that mental codes are abstract representations, and that mental codes form gradually over time.

However, they also disagree on a few very key assumptions about mental processing. For example, different models often disagree about where selective inhibition happens. They also can disagree about whether the formation of response codes from stimulus information is continuous or happens only in discrete stage-like chunks. Finally, they can disagree about whether irrelevant stimulus information influences the formation of stimulus codes, the formation of response codes, or both.

This last question is the key issue that differentiates the Dimensional Overlap Model from other models of consistency effects. Most models of consistency effects assume that irrelevant stimulus information influences the formation of response codes, whereas the Dimensional Overlap Model assumes that influence of the irrelevant stimulus depends on what the irrelevant stimulus dimension overlaps with.

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

The link from stimulus to response

In connectionist network models of reaction time, stimulus codes are transformed into response codes through links that are set up between stimulus and response units.  These connections are thought of as associative links between concepts, stored in memory.  There can be both long-term memory (LTM) associations, and short-term memory (STM) associations (Barber & O’Leary, 1997).

LTM associations form between two units from repeated exposure over an extended period of time, such as the link between the stimulus unit for a written word and the response unit for saying that word, or the link between a stimulus unit for a particular location and a response unit for acting towards that location (since we often respond toward stimuli in our environment).

STM associations are temporary links formed between stimulus and response units based on a particular task at hand.  For example, if you are told to press a left key when you see a blue stimulus and to press a right key when you see a green stimulus, then a STM association would form between the “blue” stimulus code and the “left” response code, and between the “green” stimulus code and the “right” response code.  In the language of connectionist network models, there would be a temporary link between the “blue” stimulus unit and the “left” response unit.  This means that activation in the “blue” stimulus unit would be used as input to the “left” response unit, causing activation in the “left” response unit to increase.

According to these models, both controlled (intentional, deliberate, conscious) and automatic (unintentional, reflexive, unconscious) translation of a stimulus code into a response code happens through the same mechanism: activation in a stimulus unit is transformed into output, passed along an associative link, and used as input to a response unit, causing that response unit to accumulate activation.  STM associations implement the controlled translation from stimulus to response that is determined by the specific instructions of the task at hand.  Because STM associations encode the instructions of the task, they always link a stimulus to the correct corresponding response.  LTM associations, on the other hand, are based on previous experience, rather than the task at hand.  As a result, STM associations have also been called “controlled lines,” while LTM associations have been called “automatic lines” (Kornblum et al., 1999).

It is easy to see how the combination of automatic and controlled lines can allow these models to account for the S-R consistency effect (regardless of whether one is implementing a dimensional overlap model or a response selection model, because the mechanism accounting for S-R consistency is the same in both).  Consider, for example,  how information processing might proceed in a typical Simon task: a blue stimulus appears on the left side, causing activation to accumulate in the blue relevant stimulus unit and the left irrelevant stimulus unit; if the instructions assign a left key-press to blue stimuli, then activation from the blue relevant stimulus is passed along a STM association to the left response unit; activation of the left irrelevant stimulus unit is passed along a LTM association to the left response unit; because the response unit is getting input from both stimulus units, it has a high input, and activation accumulates quickly, reaching the threshold for completion in a short amount of time.  On the other hand, if the blue stimulus had appeared on the right side, the right irrelevant stimulus unit would have activated the right response unit through the LTM association, and input to the (correct) left response unit would have been lower.  Lower input, of course, means activation accumulates more slowly, and the decision threshold is reached later.

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

Mental codes form gradually

One of the basic assumptions shared by all connectionist models of reaction time is that mental codes form gradually, through the  accumulation of evidence over time based on input information.  For example, when a blue stimulus appears, information from the sensory signals gradually causes evidence to accumulate in favor of the stimulus code representing the color blue.  In the language of connectionist networks, the activation of the stimulus unit for the color blue gradually increases over time. A mental code can be thought of as “completely formed” once activation in the appropriate unit has reached some criterion level.  Activation  in a stimulus unit can trigger the accumulation of activation in a response unit either right away, or after a decision threshold is met, depending on whether the model assumes stages or continuous processing. The ultimate speed of performance is determined by how long it takes for activation of the motor units to reach some “decision criterion,” indicating that the motor codes have been fully formed.

This idea evolved out of the combination of three ideas.  Signal detection theory (Green & Swets, 1966; Swets, 1964; Tanner & Swets, 1954) suggested that detecting a particular stimulus (i.e. forming a particular stimulus code) is a statistical decision: the signals that you get from your sensory system are subject to variability, so although a particular stimulus characteristic on average produces a particular sensory signal, there will be times when the signal appears without the stimulus being there, and there will be times when the signal fails to appear when the stimulus is there.  So, you have to use a decision threshold for how strong the signal has to be so that you are most likely to detect it when it is there, but least likely to think it is there when it is actually not.

Stimulus sampling theory (Estes, 1950, 1955) introduced the idea that perception involves repeated sampling of a stimulus over time, allowing signal detection theory to be extended over time (see, e.g., Pike, 1973).  The statistical decision tool called sequential sampling and optional stopping (Wald, 1947) is used whenever you do not want to sample more data than necessary, to determine the number of times a signal has to be sampled in order to make a confident decision about its value.  According to this method, each additional sample of information modifies your cumulative level of confidence, allowing you to continue sampling information until your confidence is high enough to meet some decision criterion, at which point you stop sampling.

This idea was immediately incorporated into a large number of psychological models of performance in CRT tasks (e.g. Audley, 1960; Audley & Pike, 1965; LaBerge, 1962; McGill, 1963, 1967; Stone, 1960; Vicker, 1970).  Although the details of these models differ, the basic premise is the same: evidence for each stimulus code accumulates over time, due to repeated sampling of sensory information, until a decision criterion of some sort is reached, indicating that the code has been fully formed and the stimulus has therefore been fully identified (see Luce, 1986, for more details).  Currently, two major types of models based on this premise are being pursued: diffusion models (Ratcliff, 1978, 1980, 1981, 1988) and the connectionist models discussed here.

Most connectionist models use the same equation to determine how activation changes over time, drawing on the first connectionist model of performance, McClelland’s (1979) Cascade model.  McClelland proposed that units be understood as first-order linear integrators, so that their activation at any given point in time is a time-averaging function of their input.  When units like this are given a constant input, their activation will asymptotically approach that input value according to a “loading curve”: approaching the input level at a rate proportional to how far away it is from the input.  This function actually first appeared in a psychological model proposed by Grice (1972, 1977; Grice, Nullmeyer, & Spiker, 1982), although he rarely gets credit for this contribution (see Luce, 1986).

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn