Cross-Modal Tasks

Cross-modal Tasks are a type of choice reaction time task that use structural overlap between stimulus dimensions of different sensory modalities. As a result, there is dimensional overlap between the irrelevant stimulus and the relevant stimulus, but no dimensional overlap with the responses. In the dimensional overlap taxonomy, these are considered to be Type 4 tasks. Other Type 4 tasks include the Flanker task and the Stroop-like task.

Robert Melara and Lawrence Marks (1990; Marks, 1987; Melara, 1989) are two of the first to study stimulus-stimulus consistency effects using cross-modal tasks. In a typical cross-modal task, subjects are told to press a left key or a right key depending on the brightness of a stimulus light. In conjunction with the stimulus light, subjects hear a tone of either a high or a low pitch, or a loud or a soft volume. The stimulus dimensions of brightness, pitch, and volume are not perceptually similar, and no specific level of brightness has an associative link with any particular pitch or loudness of sound. However, all of these dimensions can be rank-ordered from “low” to “high” along a continuum. As a result, the relationship between the elements of the relevant stimulus set (high and low brightness) and the relationship between the elements of the irrelevant stimulus set (high or low pitch or volume) allows them to be paired. Thus, although the tone is irrelevant to the task, it can be either consistent (e.g. a bright light paired with a loud sound) or inconsistent (e.g. a bright light paired with a soft sound) with the light. Responses are faster and more accurate for consistent stimuli than inconsistent stimuli.

Cross-Modal Task

This type of task validates the generality of the dimensional overlap theory. If the stimulus-stimulus consistency effect only appeared in Flanker tasks, a theory could be proposed that the effect was caused directly by a perceptual mechanism. If the stimulus-stimulus consistency effect only appeared in Stroop-like tasks, a theory could be put forth that was based on individual learned associations (such as the learned association between a color and its color word). The fact that the effect appears in cross-modal tasks, however, demonstrates that it is truly the dimensional overlap–in the most abstract sense–between the two stimulus dimensions that is key to the appearance of the consistency effect.

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn

What is Dimensional Overlap?

Dimensional Overlap (DO) is a property of sets.

A set is a collection of items.  When these items have one or more attributes in common they form a category (cf. Kornblum, Hasbroucq & Osman; Kornblum & Lee).  This category is identified by the common attribute which is also the dimension of that category.  When two or more sets are included in a larger set this larger set is called an ensemble.

Dimensional Overlap is the degree to which a stimulus set, and a response set, or two or more stimulus sets are perceptually, structurally or conceptually similar.  Similarity is a continuous variable, going from “almost identical” to “they have nothing to do with each other”, and different levels of  similarity are carried over to differences in degrees of DO. These differences are dealt with in the computational model as different strengths of the “automatic association lines” between different mental representation units. (More info: The Computational Model)

The best way to understand the ideas of perceptual, conceptual, and structural similarity is to consider some examples of tasks with dimensional overlap  based on such similarities.

A.  An experimental subject is sitting in front of a screen.  On the screen is the outline drawing of two outstretched hands.  The subject has his own outstretched hands positioned on eight response keys, and he is instructed to press the key under the finger that corresponds to the finger outline on the screen, when an “X” appears in the tip of that finger.  The stimulus set (i.e. the outline drawing of the hands with an X in any fingertip), and the response set (i.e. the set of potential key presses with one of the fingers) are perceptually similar, giving that ensemble dimensional overlap.

B.  In this task, the subject must press the left key when he hears a sound in the left ear, and the right key when he hears a sound in the right ear.  Laterality is the attribute, or dimension, that the stimulus and the response sets have in common, and which makes them perceptually similar, thus giving the ensemble dimensional overlap.

C.  In this task, the subject says the name of a color that is presented (e.g. when you see Blue you say “Blue”, and when you see Green, you say “Green”). Here, the attribute that the stimulus set (Blue and Green) and the response set (“Blue” and “Green”) have in common is COLOR.  These two sets are conceptually similar, thus giving this SR ensemble Dimensional Overlap.

D.  Take the set of symbols 1, 2, 3, 4, 5. They form a category called “integers”; it could also be called “digits”, or any of several other names.  Next, take the set of words “one”, “two”, “three”, “four”, “five”…They form a category called “digit name”.  Both of these sets are tokens for numerals, and are conceptually similar.  The ensemble comprising these two sets, therefore, has dimensional overlap.

In this case, as well as in the case of colors (in C above),  the correspondence between the elements of the stimulus set and the response set are so exact as to give their similarity level (hence the level of DO) an especially high value.

E.  The set of symbols B, C, D, F, G…are members of a category called “consonants”.  Likewise, the set of symbols A, E, I, O, U are members of a category called “vowels”.  Both of these sets are sets of “letters”.  Letters may, therefore, be viewed as the attribute that they have in common, on which they overlap, and makes them conceptually similar.  An ensemble with both of these sets has dimensional overlap.

F.  Now let us consider the set of integers (1, 2, 3, 4, 5…), and the set of letters (A, B, C, D, E,).  Both of these sets share the property of being the first consecutive elements of two sequential series (the alphabet and the integers).  Thus making them structurally similar, and giving them dimensional overlap.

G.  Finally, consider a set of weather reports: “hot, mild, cold, and freezing”.  To the extent that this set may also be viewed as consecutive items on an abstract sequential series of items (this would be true whether high and low temperatures are indicated by a numeric scale, or in some other way), this set of weather reports is similar to the set of digits, and the set of letters, and to the extent that all three sets share the same dimension, they are all structurally  similar to one another, and together would form an ensemble with dimensional overlap.  This particular pattern of DO is at the basis of what we have called “cross modal” tasks.  (More info: Cross-Modal Tasks)


Relevant and Irrelevant Stimuli

In all the tasks that we have used to illustrate different sorts of similarities, the stimulus set that overlaps with the response set has always been a set of relevant stimuli. (cf, “Why Study Compatibility”)   Relevant stimuli are those that the subject is instructed to attend to, and respond to.  Of course, any “stimulus” has many features that could, in principle, be selected to play that role.  However, in the mapping instructions for all tasks, specific features are selected for that precise purpose, while other features are explicitly excluded and are to be ignored by the subject.  Those “to be ignored” features are the irrelevant stimuli.  The correlation between the relevant stimuli and the response, is 1.0,  the correlation between the irrelevant stimuli and the response is zero.

Even though they are irrelevant to the task, and the subject is told to ignore them, irrelevant stimuli are sometimes difficult to ignore.  For example, let’s go back to the experimental set up we used in case B above, and introduce a small change: Instead of presenting the same tone to the left and the right ear, let the tone now be either a high or low pitch tone.  Again, it will be randomly presented to the left or the right ear, just as in B above..  However, this time, instead of mapping the side of the ear to the key, the subject is instructed to make a left or right key-press according to the pitch of the tone, and to ignore in which ear the tone was presented: “press the right key to the high pitch tone, the left key to the low pitch tone, and ignore the ear to which the tone is presented”.  With these instructions, and the experimental set up, pitch is the relevant stimulus, and ear is the irrelevant stimulus. When these irrelevant (i.e. “to be ignored”) stimuli have dimensional overlap with the response, or with some other aspects of the tasks, it results in significant changes in performance.

The eight combinations of the presence/absence of relevant/irrelevant, stimulus-response ensembles, and stimulus-stimulus ensembles are presented in the Taxonomy of DO Ensembles.  (More info: Taxonomy of DO Ensembles)


Consistency and Inconsistency

When the relevant and the irrelevant stimuli denote the same attribute, this gives rise to S-S Consistency / Inconsistency.  For example: if the color BLUE is the relevant stimulus, and the name of the color “BLUE” is the irrelevant stimulus, the dimension on which they overlap is COLOR, and the fact that they both represent the same feature (i.e. BLUE) makes this an S-S CONSISTENT  pair of stimuli.

If this match between the color and the name does not hold, then this is an S-S INCONSISTENT pair.

When the irrelevant stimulus and the response denote the same attribute, this gives rise to S-R Consistency / Inconsistency. For example, if the color of a light is the relevant stimulus, and a left/right key-press is the response, and the light can be turned on either on the left or on the right, then the location of the lights (left or right) and the position of the response keys (left or right) make this ensemble S-R CONSISTENT.  The dimension on which they overlap is “lateral position in space”.

If the match between the location of the lights and the position of the response key does not hold, then this is an S-R INCONSISTENT pair. (More Info: What is consistency?)


Dimensional Overlap is a GENERAL mental phenomenon

Some sets sometimes seem so physically different from other sets – for instance when the sensory channel for the one is vision and for the other is audition – that the similarity (the DO) between them seems far fetched.  Yet, they give rise to compatibility effects in performance, which verifies that there is some set-level similarity between the mental representations (cf. Robert Melara and Lawrence Marks (1990; Marks, 1987; Melara, 1989).

 Share on Facebook Share on Twitter Share on Reddit Share on LinkedIn