Polysemiotic Communication vs. Multimodality

a conceptual and terminological distinction applied in street art

Abstract

Multimodality is in fact a polysemous word, which is tightly related to the notions of modality, and (semiotic) mode and is used in conceptually different ways across different disciplines (for a review see Adami, 2016; Devylder, 2019; Green, 2014). As cognitive semiotics (Zlatev et al., 2016) aims to integrate concepts and methods from semiotics, cognitive science and cognitive linguistics, we endeavor to offer a coherent terminology, in line with the proposals of Green (2014), Stampoulidis et al. (2019) and Zlatev (2019), which distinguishes the notions of perceptual (sensory) modalities (sight, hearing, smell, touch and taste) and semiotic systems (language, depiction and gesture).

For example, using this polysemiotic/multimodal distinction would allow us to describe the work of street art displayed in Figure 1 consisting of verbal text (language) and pictorial elements (depiction) as clearly a form of polysemiotic communication, instantiated in the particular socio-cultural medium of street art, whereas the street artwork displayed in Figure 2 might be considered as an example of unisemiotic communication (only the semiotic system of depiction is present). Nevertheless, both artworks may be considered monomodal since at least one perceptual modality is involved: sight. On the other hand, artworks such as these displayed in Figure 3 and Figure 4 can be both polysemiotic and unisemiotic, respectively, and (potentially) multimodal (if) they trigger multiple senses in the viewer, such as sight and touch, for example. It is important to note that the terminological distinction and conceptual dichotomy between the semiotic systems of language and depiction are not always clear-cut, especially in the case of street art (and graffiti), as has been argued in a certain literature (Bal, 1991; Neef, 2007). Therefore, we would like to stress that street art is typically a form of polysemiotic communication, and thus, we restrict the term unisemiotic either to the case of primarily depiction-dominant or primarily language- dominant graphic representations.

In our study on street art, marking this polysemiotic/ multimodal distinction – terminological and conceptual – would help us toward a synthetic analysis of the interaction between language and depiction, and that of language, depiction, vision, and (potentially) smelling, touching or even hearing, into a whole communicative situation.