eJournals Kodikas/Code 39/1-2

Kodikas/Code
0171-0834
2941-0835
Narr Verlag Tübingen
Es handelt sich um einen Open-Access-Artikel der unter den Bedingungen der Lizenz CC by 4.0 veröffentlicht wurde.http://creativecommons.org/licenses/by/4.0/
After forty years of debate, the question of whether visual imagery is exclusively based on propositions, or if there are autonomous pictorial forms of imagery, still remains unsolved. Both approaches, though, fall short of taking into account the situated, embodied and experiential character of imagery. This paper proposes an experiential and body-based account of imagery in the light of Peirce's theory of signs.
2016
391-2

Imagery as Experience

2016
Sabine Marienberg
K O D I K A S / C O D E Volume 39 (2016) · No. 1 - 2 Gunter Narr Verlag Tübingen Imagery as Experience Sabine Marienberg (Berlin) After forty years of debate, the question of whether visual imagery is exclusively based on propositions, or if there are autonomous pictorial forms of imagery, still remains unsolved. Both approaches, though, fall short of taking into account the situated, embodied and experiential character of imagery. This paper proposes an experiential and body-based account of imagery in the light of Peirce ’ s theory of signs. 1 Mental Imagery Even though the term ‘ imagery ’ is etymologically rooted in the word ‘ image ’ , and hence in something we see, its coverage is not limited to visual phenomena. It can apply just as well to the sense of hearing and touch as it can to vision - and we can also have auditive, olfactory, gustatory or motor imagery. What all these different kinds of imagery have in common is that they allegedly evoke “ internal ” images representing something that is not actually present and perceived. They count as quasi-perceptions, occurring in mental operations, anticipations, imagination or memory. It is a much discussed question whether the contents of imagery have to have been previously perceived - in this case we are dealing with the sensory aspect of memory - or whether they are actively created, as may be the case in imagination and dreams, as well as in problem solving or other kinds of perceptual anticipation (and be it only by combining elements of former perceptions). To see both possibilities as excluding one another draws an artificial boundary between two activities - acting and perceiving - which, even though they can and should be theoretically distinguished, are inseparably linked to one another in experience. The so-called imagery debate - a debate in cognitive science that was most vigorously conducted in the 1970 s and ’ 80 s and has been going on ever since - does not relate to vivid quasi-perceptions but to the format of imagery. The dimension of experience, more than being only neglected, has often been explicitly excluded (Kosslyn 1994), narrowing the question to how mental content, wherever it derives from and under whatever circumstances it is formed, is generally processed. In this way, mental representations or cognitive states are tacitly conceived of as signs in the Aristotelian sense of aliquid stat pro aliquo. By adopting instead Charles Sanders Peirce ’ s much richer notion of a sign as “ something which stands to somebody for something in some respect or capacity ” (CP 2.228), one could avoid eliminating prospection and situatedness completely as well as letting the sign relation be a miraculous and universal one. 2 The Imagery Debate 2.1 In cognitive science, imagery is considered above all in respect to the role it plays in cognitive processes. The imagery debate focuses on two particular ways of dealing with imagery: a depictive one, and a linguistic or propositional one. The question is an epistemological one. To ask about the format of imagery is to ask about the cognitive role of pictures and descriptions respectively - as well as about the nature of cognition itself. So, more than being concerned with reasoning about pictures or language, the contentious issue is: Do we think, anticipate and remember in pictures or propositions - and if we do so in both, are they both indispensable vehicles of thought? In other words: Do we have to assume autonomous picture-like mental operations, or are they merely epiphenomenal in the sense that they can be reduced to language-like ones? The most prominent protagonists in this debate are Stephen Kosslyn - who opts for a pictorial account of imagery, referred to as ‘ pictorialism ’ - and Zenon Pylyshyn, who favors a propositional account, which is commonly termed ‘ propositionalism ’ or ‘ descriptivism ’ . 2.2 After the assumption of mental images had long been discredited as unscientific by the predominating behavioristic paradigm in psychology, the spatial cognition experiments of Roger Shepard and his colleagues led to renewed consideration of mental imagery (Shepard & Metzler 1971; Cooper & Shepard 1973; Shepard & Cooper 1982). In these experiments, people were shown pairs of twoor three-dimensional figures (such as letters, random shapes or asymmetrical stacks of cubes), one of which had been rotated to a different orientation, and were then asked to decide whether the two figures were identical or mirror inverted. The results showed that the rotation angle corresponded directly to the reaction time: The more the orientation had been altered, the more time the partipants required to come to a conclusion about the identity or non-identity of the figures in question. There was thus reason to conclude that people were actually mentally rotating one of the figures. These results were subsequently supported by a series of mental scanning experiments conducted by Stephen Kosslyn (Kosslyn 1973). In this case, the participants were asked to visualize a horse as seen from the side and then to imagine first its tail and then the ears, or first the middle of its back and then the ears, or to imagine the shortest route between two locations on a map which they had been shown beforehand (Kosslyn, Ball & Reiser 1978) - and again the results showed that the longer an imagined distance was, the longer it took to “ mentally scan ” it. These findings eventually led to the view that there are in fact pictorial mental representations, which moreover have spatial characteristics themselves. Properties such as distance, size and rotation angles are supposedly represented in the same way as external visual objects are seen; and mentally folding or turning an object, mentally bridging a spatial distance or striding across a landscape on a map is supposed to work analogously to visually perceiving these processes. However, this does not mean that mental images do possess the material characteristics of visual objects, but they are thought to preserve their spatial relations - it ’ s more of a functional analogy than an ontological one. Besides, pictorialists do not maintain that picture-like representations are the only ones to exist. But they consider them as a representational form in its own right that allows for cognitive processes which propositions cannot provide. In contrast, descriptionalists or Imagery as Experience 49 propositionalists make a far stronger claim by holding that the one and only format of imagery is a language-like one and that even though pictorial imagery may occur, it does not play an essential cognitive role. 2.3 The propositionalists ’ rejection of pictorialism is chiefly based on the following arguments: 1 Firstly, if there is any such thing as pictorial imagery, it is at best a way of thinking about how something looks; it touches on what a mental image is about, but not the way in which it is represented. The experience of mental imagery is not at all a cognitive form; it is what happens when the content of our imagery is itself a picture. Secondly, the so-called “ tacit knowledge argument ” (Pylyshyn 1973; 1981) amounts to the objection that when we supposedly mentally manipulate a visual object, we are actually investing what we know about real visual objects in what we are visualizing. So when, for example, people are asked to think of a blue circle and a yellow circle slowly moving towards each other till they begin to overlap, and then to state what color this overlapping area has, the answer could be green, or white, according to whether one is thinking of subtractive or additive color blending (Pylyshyn 2003); and when test subjects need more time to mentally rotate an object, it is because they know that it would take longer to physically rotate it. Thirdly, pictures are often cognitively impenetrable and therefore cannot count as forms of cognition. To be cognitively impenetrable means that how we see something is sometimes “ hard wired ” , so to speak, in that it is due to the natural architecture of visual perception and cannot be altered by knowledge or belief. The empirical evidence for this hypothesis is derived, for example, from optical illusions where what we think or believe does not affect our perception at all. One example is the Muller-Lyer Illusion, which consists of two straight lines of equal length that form arrow shafts with arrow fins pointing either inwards or outwards at both ends. Depending on whether the fins point inwards (forming two arrow heads) or outwards (forming two arrow tails), the line in between is perceived to be longer or shorter. And even if we measure the lines and are fully aware of the fact that there is no difference in length, they continue to “ look ” shorter or longer. They are what they are, no matter what our attitude towards them may be. For representations to be cognitively constitutive though, Pylyshyn argues, knowledge and belief should have an impact. And the fourth (and maybe the most important) criticism is that a pictorialist approach inevitably leads to a homunculus regress: If we claim that the mind ’ s eye works more or less like our physical eyes do - and that we really are moving internal images around - then we would need a mind ’ s eye ’ s brain (or a mind ’ s eye ’ s mind) to observe the mind ’ s eye ’ s images, and so forth ad infinitum. 2 Propositionalism holds that mental images represent the world in the form of descriptions - and, what is more, that we dispose of an innate inner language, in which, for example, a state of affairs like “ there is an orange in the pocket of my coat ” could be represented in the form of propositional lists of objects and spatial relations operators, for instance: IN (ORANGE, (IN (POCKET, COAT)). But they could just as well be represented in 1 Methodological discussions concerning experimental settings as well as neuroscientific investigations in later stages of the debate do not essentially alter the line of argumentation followed in this paper. 2 This objection is countered, if not ridiculed, by Kosslyn as taking the metaphor of an inner image all too literally (Kosslyn 1980; 1994). 50 Sabine Marienberg (Berlin) other symbolic forms, such as binary code or algorithms. The main point is that these language-like descriptions do not resemble their objects, since they do not share any characteristics of what they are describing. Or, in traditional terminology, they are radically arbitrary. 3 Conceptualizing Imagery 3.1 This is, of course, a poor and pale notion of language. The internal expressions that form propositional language are, just as the words of inner speech were for Augustinus, verba nullius linguae, or, in modern terms, they constitute the language of thought (Fodor 1975) or mentalese (Pinker 1994); they are expressions that do not belong to any spoken language, let alone pertain to individual utterances. It is thus a language with no material characteristics, a language that does not exhibit intonations or a plenitude of connotations and vaguenesses and, above all, a language that does not pose any obstacles to cognition. The pictorialists - even though they insist on the cognitive equivalence of imagery and visual perception and posit that mental images not only represent but also share certain spatial properties of visual objects - they too are also mainly interested in the deep structure of images in the form of types (and not tokens) of analogue syntactic relations between points in space - or, to put it another way, in a kind of spatial mentalese. 3.2 Pictorialism and propositionalism both try to provide a model of cognitive processes. In this respect, both approaches are rather more or less appropriate than simply true or wrong. The evaluation criterion can only be a pragmatic one: They should be judged by what they are able and not able to explain. Examining the competing models behind both positions, it is easy to see that, despite all their differences, pictorialist and propositionalist approaches have a lot of - highly problematic - traits in common: First of all, they both start out from the idea that there are such things as mental representations operating as intermediate entities between the world and us. Spoken or written language - as well as pictorial objects - thus appear to be mere outer expressions of cognitive states. In this way, in the “ outside ” world, we are confronted with two different kinds of objects, one being the external signs and the other being the objects they refer to - and the only way to solve the problem of how one object can ever be about another one is to take the traditional mentalist approach and oppose world and mind, conferring the noble duty of mediation to the latter. This gap could be bridged though by viewing our understanding of the world and our symbolic access to it as a correlated enterprise. Or, as Wittgenstein states in one of his lectures: “ In the process of thinking the thought does not appear first, to be translated subsequently by us into words or other symbols. There is not something which exists before it ’ s put into word or imagery ” (Wittgenstein 1980, Lecture C IX: 86). Furthermore, mental representations - be it in pictorial or propositional mentalese - are not considered with respect to the process of their formation, but as readily available. With an eye to the triadic structure of the sign, as elaborated by Peirce, one cannot fail to notice that its iconic and indexical dimensions are completely disregarded. There is no iconic quality that gives rise to a still undetermined feeling - a feeling of a quality in isolation, like the mere possibility of blue or something warm or soft, still unconnected to anything else Imagery as Experience 51 but opening up the possibility of multitudinous relations and interpretations. And no reference is made to the dramatic moment of experienced indexicality when one feeling is replaced by another. This “ sense of action and reaction ” , as Peirce calls it (Peirce 1894: § 1), manifests itself as a surprise or disappointment, a forcible disruption between what was before and what is now, or between what was expected and what was found. It is, in other words, the notion of an antagonism, not only between two feelings but also between ourselves and what confronts us. All we are offered by the protagonists of the imagery debate is the symbolic aspect of the Peircean sign 3 - the interpretation of something as something. However, when icons and indices are not taken into consideration at all, the dynamic structure of signs vanishes too, and it becomes difficult to understand interpretation as a learning process. Finally, the arguments of both pictorialists and propositionalists hinge upon the computational model of the mind: Imagery is chiefly viewed as a kind of information processing, be it in an analogue or amodal way. Instead of conceiving of cognition as an activity of a concerned human being, as an intertwinement of acting and suffering, it is misunderstood as a transformation process, and the outcome of such processes can be attributed a meaning only in a very metaphorical sense. And it is no wonder that the cognitive role of emotions is also neglected, not to mention cultural diversity (in the nonstatistical sense) and biographical contexts. In the framework of the imagery debate, imagery figures like a thin layer that can be investigated detached from social and cultural contexts and isolated from both learning histories and bodily actions. But imagery as a situated experience does not just take place in the mind (or brain), nor can it be reduced to syntactic operations against the backdrop of presupposed semantics. Computational investigations have therefore to be enriched by both semiotic and pragmatic ones, and to be complemented by phenomenological 4 or even narrative accounts. And it is precisely a narrative that I would like to take up as a playful countermodel in order to exemplify the substantial role of action and perception in dealing with imagery. 4 Shakespeare ’ s Memory In one of his last stories, entitled “ Shakespeare ’ s Memory ” , Jorge Luis Borges (1999) brilliantly stages a scenario demonstrating just how much imagery relies on enactive and embodied experience. The protagonist, the German Shakespeare scholar Hermann Sörgel, is introduced to a man called Daniel Thorpe by a friend of his at a Shakespeare conference. One evening in a London pub, the latter mentions the ring of King Solomon which, according to legend, gave 3 Since pictorial imagery is thought to be analogue, it might seem odd to understand it as symbolic. What is meant here is that the Peircean Icon, even though similarity is one of its features, does not map but engenders form by creating the possibility of a sign-relation: “ [. . .] its Object is whatever there may be which is like the Icon, and is its Object in the measure in which it is like the Icon ” (CP 2.314). Propositional and pictorial mental imagery are both symbolic in the sense that they are exclusively looked at in a moment where sign processes have come to rest (or have never started). 4 For a phenomenological critique of the imagery debate see Thompson 2007. 52 Sabine Marienberg (Berlin) him the ability to understand the language of the birds, but is now regarded as lost. Asking themselves what could have happened to it, they speculate that it might be possessed by someone who either lives in a birdless place or in an area where there are so many birds that it would be impossible to find out what they are talking about because they were all twittering at once. So while in the first case one would be equipped with a language device (or, why not, a language organ, as Pinker imagines) but not exposed to any external language to be processed, in the second case, the birds ’ talking would be naturalized, showing just its material outside. The anti-rationalist move Borges is making consists in demonstrating that if we assume a magic interpreting or decoding entity (a magic that effectively allows for circumventing the burden of learning), the relation between world and language can easily break because it is not an internal one - and we are left with meaningless objects. The ring is just a ring and the sounds of birds are not lógos, but at best phoné, if not simply noise. Later on, Thorpe accompanies Sörgel to his hotel, where he offers him something even more fantastic than Solomon ’ s ring. He offers him Shakespeare ’ s memory dating from his early boyhood to April 1616. The only condition for handing it over from one owner to the other is that it has to be offered and accepted aloud. First of all, this seems to be another cognitive trap of the declarative type, suggesting that the contents of someone ’ s memory are recorded in list form, as results that can be passed from one man to another, independently of the life in which they were formed. But, as Thorpe explains, this is not at all the case. Sörgel will not be able to take possession of it as he could do with an umbrella. Shakespeare ’ s memory cannot be handed over in one piece, and Sörgel will have to wait for it to arrive and unfold, to reveal itself at impredictable moments. Thorpe also warns him not to invent it. In the end, to acquire someone else ’ s memory does not essentially differ from acquiring anything else. It is obtained and understood in a dynamic entwinement of activity and passivity just as other experienced things are, including one ’ s own memory. The problem is: The latter is still there as well. Waiting for the miracle to happen, Sörgel expects the first bits of memory to be images, when all of a sudden he finds himself articulating a string of words he cannot identify. A colleague tells him that they are from Chaucer ’ s “ A. B. C. ” . Days after he hears himself whistling a melody he has never heard before. Obviously, Shakespeare ’ s memory is a rather auditory one. But the auditory events somehow present themselves in a monolithical form, their relations to other things being all but clear. Their first appearance is, of course, iconic, but immediately unfolds its disturbing indexical power. After a while, visual memories also start coming up in dreams of unknown places and faces, and they do so in the same isolated and puzzling form. Everything that appears has to be practised and related to be understood, and only by practising can memories be evoked. Sörgel begins to read Shakespeare ’ s verses aloud till he discovers - or better re-covers - the open vowels of the 16th century. Alva Noë would call this “ action in perception ” (Noë 2006). But the more he practises and the more he captures (or better: is captured by) Shakespeare ’ s memory, the more his capacity to orient himself in the present deserts him. The objects and sounds in his everyday surroundings fade into blurrily shaped materials and noises of unknown origin, and he feels that the other ’ s memory begins to take over his own. Afraid of losing his identity, he decides to free himself from the burden of the gift and starts to dial random telephone numbers, putting down the phone immediately Imagery as Experience 53 when he hears the voice of a woman or a child, till finally there is a man on the other end who - performing a genuine speech act - accepts to be the next owner. 5 Experience in Action There seems to be no need to take an either-or decision between images or language and between analogue or arbitrary concerning the way in which they allow access to Shakespeare ’ s world. Moreover, the problem is not a syntactic but a pragmatic one: Shakespeare ’ s memory is not something the protagonist has or processes, it is first of all something he actualizes and experiences - whistling, articulating, reading, or dreaming. It comes to him in the form of iconic fragments that lead to conflictual experiences and ask to be related and interpreted as something - or, in Peircean terminology, that bring about an habit change in the sense of being able to act, and understand, differently. 5 In the imagery debate, the issue of interpretational changes has been tackled in a series of reclassification tasks, beginning with the research of ambiguous figure reversal by Chambers and Reisberg. 6 In later experiments, subjects were briefly shown unfamiliar bistable images (for example, a snail/ seahorse or a bird/ cat figure) whose second, alternative interpretation was hard to infer in one orientation but became immediately obvious when the image was physically rotated by ninety degrees (Slezak 1992; 1995). As in the Chambers and Reisberg experiments, the presentation time was too short for the subjects to switch to the alternative interpretation of the image, but long enough for them to remember it. If a mental image of a bistable figure really is pictorial, so the argumentation, it should be possible to resolve its ambiguity by way of internal rotation. However, that was not at all the case, not even with the help of hints. Furthermore, even when the visual presentation time was increased to three minutes, barely ten percent of the subjects were able to accomplish the task. One line of explanation is that mental images are intrinsically encoded and symbolically stable so as to prevent further interpretation. In an additional task, to make sure that the quality of the image had not decreased and therefore lacked the required information for reconstruing it, subjects were asked to draw the figure in question from memory. The results showed that not only the constitutive characteristics of the original interpretation, but also contingent details that revealed the alternative one when rotating the drawing had been preserved. This paradigm, though, has only been used to eliminate the hypothesis that visual images are too abstract and reduced to offer alternative interpretations of what they represent. The idea that it is precisely the act of drawing itself that sets free the iconic and indexical force of the image and discloses a multitude of possible interpretations has not been developed. 7 “ A man ’ s memory ” , Borges writes, “ is not a summation, it is a chaos of vague possibilities ” (Borges 1999: 514). In the same vein, a man ’ s imagery that is conceived of as being neither 5 “ If there be a unity among our sensations which has no reference to how we shall act on a given occasion, as when we listen to a piece of music, why we do not call that thinking. To develop its meaning, we have, therefore, simply to determine what habits it produces, for what a thing means is simply what habits it involves ” (Peirce, CP 5.400). 6 For details and discussion see Chambers and Reisberg (1985; 1992); Finke et al. (1989). 7 For Peirce ’ s account of bistable images see Viola (2012). 54 Sabine Marienberg (Berlin) formed nor challenged in experience would be comparable to Shakespeare ’ s memory before Sörgel had accepted the offer to possess it, or if it had been handed over and replaced his own all in one piece. One might oppose that all this takes place in a rather fantastic literary setting. But one can also ask whether devising a model of the internal syntactic processing of mental representations is not an even more fictional enterprise. References Borges, J. L. 1999: "Shakespeare ’ s Memory", in: id.: Collected Fictions of Jorge Luis Borges, translated by Andrew Hurley, New York: Penguin, 508 - 515 Chambers, D., & Reisberg, D. 1985: "Can Mental Images be Ambiguous? ", in: Journal of Experimental Psychology: Human Perception and Performance 11 (1985): 317 - 328 Chambers, D., & Reisberg, D. 1992: "What an Image Depicts Depends on What an Image Means", in: Cognitive Psychology 24 (1992): 145 - 174 Cooper, L. A., & Shepard, R. N. 1973: "Chronometric Studies of the Rotation of Mental Images", in: Visual Information Processing, ed. by W. G. Chase, New York: Academic Press, 75 - 176 Finke, R. A., Pinker, S., & Farah, M. J. 1989: "Reinterpreting Visual Patterns in Mental Imagery", in: Cognitive Sciene 13 (1989): 51 - 78 Kosslyn, S. M. 1973: "Scanning Visual Images: Some Structural Implications", in: Perception and Psychophysics 14 (1973): 90 - 94 Kosslyn, S. M. 1980: Image and Mind, Cambridge, MA: Harvard University Press Kosslyn, S. M. 1994: Image and Brain. The Resolution of the Imagery Debate, Cambridge, MA: MIT Press Kosslyn, S. M. & Pomerantz, J. R. 1977: "Imagery, Propositions, and the Form of Internal Representations", in: Cognitive Psychology 9 (1977): 52 - 76 Kossyln, S. M., Ball, T. M. & Reiser, B. J. 1978: "Visual Images Preserve Metric Spatial Information: Evidence from Studies of Image Scanning", in: Journal of Experimental Psychology: Human Perception and Performance 4.1 (1978): 47 - 60 Noë, Alva 2006: Action in Perception, Cambridge, MA: MIT Press Peirce, C. S. 1894: "What is a Sign? ", in: Peirce Editions Project (ed.): The Essential Peirce: Selected Philosophical Writings, vol. 2 (1893 - 1913), Bloomington 1998: Indiana University Press, 4 - 10 Peirce, C. S. 1931 - 35/ 1958: Collected Papers [CP], ed. by C. Hartshorne & P. Weiss (vol. 1 - 6) and A.W. Burke (vol. 7 - 8), Cambridge, MA: Harvard University Press Pinker, S. 1994: The Language Instinct. How the Mind Creates Language, New York: Harper Pylyshyn, Z.W. 1981: "The Imagery Debate: Analogue Media versus Tacit Knowledge", in: Psychological Review 88.1 (1981): 16 - 45 Pylyshyn, Z.W. 1973: "What the Mind ’ s Eye tells the Mind ’ s Brain: A Critique of Mental Imagery", in: Psychological Bulletin 80.1 (1973): 1 - 24 Pylyshyn, Z. 2003: "Return of the Mental Image: Are There Really Pictures in the Brain? ", in: TRENDS in Cognitive Science 7.3 (2003): 113 - 118 Shepard, R. N., & Metzler, J. 1971: "Mental Rotation of Three-Dimensional Objects", in: Science 171 (1971): ,701 - 703 Shepard, R. N., & L. A. Cooper 1982: Mental Images and Their Transformations, Cambridge, MA: MIT Press Slezak, P. 1992: "The 'Philosophical' Case against Visual Images. A 'Crucial' Experiment", in: AAAI Technical Report SS92 - 02: 12 - 17 Imagery as Experience 55 Slezak, P. 1995: "The 'Philosophical' Case against Imagery", in: P. Slezak, T. Caelli & R. Clark (eds.), Perspectives on Cognitive Science. Theories, Experiments, and Foundations, Norwood NJ: Praeger, 237 - 271 Thompson, E. 2007: "Look Again: Phenomenology and Mental Imagery", in: Phenom Cogn Sci 6 (2007): 137 - 170 Viola, T. 2012: "Pragmatism, Bistable Images, and the Serpentine Line", in: F. Engel, M. Queisner & T. Viola (eds.), Das bildnerische Denken: Charles S. Peirce, Berlin: Akademie Verlag (Actus et Imago 5), 115 - 138 Wittgenstein, L. 1980: Wittgenstein ’ s Lectures, Cambridge 1930 - 32. From the Notes of John King and Desmond Lee, Oxford: Basil Blackwell 56 Sabine Marienberg (Berlin)