Salvio Martin Menendez
A strategic approach to multimodal
discourse analysis
Res Rhetorica nr 1, 43-53 2015
SALVIO MARTÍN MENÉNDEZ
UNIVERSIDAD NACIONAL DE MAR DEL PLATA, UNIVERSIDAD DE BUENOS AIRES, CONSEJO NACIONAL DE INVESTIGACIONES CIENTÍFICAS Y TÉCNICAS
SALVIOMENENDEZ@GMAIL.COM
A strategic approach to multimodal discourse analysis
AbstractIn the paper the author argues that multimodal discourse analysis examines the traditional issues of rhetoric from a pragmatic-discursive perspective and therefore may be regarded as a part of rhetoric. Starting from systemic-functional linguistics, the author proposes a method for studying the multimodality of discourse which highlights the notion of strategy, defi ned as a plan for achieving a particular purpose.
Key words
Rhetoric, multimodality, strategy, pragmatic-discursive perspective, systemic-functional linguistics
License
This work is licensed under a Creative Commons Attribution 3.0 Poland. The content of the license is available at http://creativecommons.org/licenses/by/3.0/pl/
DOI
SALVIO MARTÍN MENÉNDEZ
UNIVERSIDAD NACIONAL DE MAR DEL PLATA, UNIVERSIDAD DE BUENOS AIRES, CONSEJO NACIONAL DE INVESTIGACIONES CIENTÍFICAS Y TÉCNICAS
SALVIOMENENDEZ@GMAIL.COM
A strategic approach to multimodal discourse analysis
1. Introduction
M.A.K. Halliday argues in “Ideas about language” (1997) that traditionally lan-guage has been described in two different ways: as a resource and as a system of rules. Viewed as a system of rules, language is a tool for cognitive organization and relates to thinking. Viewed as a resource, however, language is a tool for com-munication and relates to action and interaction. Action and interaction are what link language to its defi ning feature: instrumentality. Language use is shaped by how people interact with one another in a given socio-cultural context. This di-scussion of action and interaction is not to downplay the importance of thinking, but simply to place it in the context to human interaction.
Two traditions of language description, then, are clearly traceable: one rooted in logic and the other in rhetoric. Those who look at language as a system of rules think in terms of ‘logical’ grammar. On the other hand, those who treat language as a set of resources, tend to think in terms of rhetoric. Therefore, from a logical point of view, linguistics can be seen as part of logic, while from an anthropologi-cal point of view, it can be seen as part of rhetoric.
The rhetorical tradition has been less successful in academic circles than the ‘logical’ grammar tradition exemplifi ed in functional theories of language. Indeed, logical approaches, such as structuralism and generative grammar, have almost completely overshadowed functional approaches. Functional approaches are rhe-torical because they orientate language in relation with how speakers interact in specifi c situations within a particular culture. Speakers choose what they consider to be the best resources available in order to achieve their goals, and combine them in a way that they consider successful.
This paper examines the relationship between two aspects of interaction that functional linguistics has always treated as mutually exclusive - the relationship between what has traditionally been called the verbal and the non-verbal. Until re-cently, an almost exclusive focus on grammar and discourse (with differing degrees of emphasis on each) had meant that other components of discourse and situation
were relegated to a secondary or complementary position in relation to the cen-trality of verbal language. If sociolinguistics is taken as a general framework for language description (Mesthrie 1994), pragmatics (Leech 1983, Levinson 1983), rhetorical analysis (Albaladejo Mayordomo 2005, Anscombre and Ducrot 1983, Carel and Ducrot, 2005, Eemeren, F.E. van 2009, Pujante 2003) and discourse analysis (Brown and Yule 1983, Fairclough 1992, 1995, Lavandera 1985, 2014) provide clear examples of this unequal relationship.
Systemic functional linguistics (Halliday 1994) can be placed within the rheto-rical tradition because it understands language as a system of meaning resources within social semiotics (Halliday 1979, Hodge and Kress 1998). A verbal language is just one system within the general framework of systems that make up social semiotics. This concept means that any given language is always part of a wider system of choices that can represent meaning. Language is characterized by two complementary axes of linguistic patterning, the paradigmatic and the syntagma-tic. The syntagmatic refers to how words in a sentence are combined (e.g., ‘John likes dogs’ rather than ‘Dogs like John’) while the paradigmatic refers to how words are selected (e.g. ‘John likes dogs’ rather than ‘John likes cats’). The two axes are interwoven but they can be focused on separately for the purposes of analysis.
In fact, any actual discourse arguably has three dimensions (Menéndez 2005): grammar (Halliday and Hasan 1976, Halliday and Matthiessen 2004), strategy (Gumperz, 1982, Menéndez 2005, 2010a, 2010b, van Dijk 1982, van Dijk and Kinstch 1982) and genre.(Bakhtin 1944, Martin and Rose 2008). These three analytical dimensions should be seen from a multimodal perspective (Kress 2010), because discourse is the place where different modes appear. So, a discourse is a multimodal unit. A multimodal perspective allows us to analyze how relationships between the resources of each mode are established and why these relationships occur in the way they do.
Accordingly, a method of multimodal discourse analysis will be presented. The method consists of analyzing discourse strategies (Menéndez 2012) to show how the different resources of each mode appear in relation to one other.
2. The multimodal perspective
Multimodality describes communication practices in terms of the textual, aural, linguistic, spatial, and visual resources - or ‘modes’ - used to compose messages. Consequently, in discourse analysis, multimodality is considered to be a perspec-tive (Jewitt 2009) rather than a theory. Based on systemic functional linguistics, which understands verbal language as part of a social semiotic system (Halliday
1978, Hodge and Kress 1998.), it can be seen as a reformulation of the pragmatic--discursive perspective (Verschueren 1999, Menéndez 2005).
The central concept of systemic linguistics is that verbal language can be de-scribed in terms of a set of choices of meaning, a set of options, such as singular/ plural, past, present, future tense, positive/negative polarity, etc., and the actual choices made by speakers. Systemic linguistics can thus be seen as a forerunner of the multimodal approach. This aspect is true because verbal language is a part of social semiotics. Verbal language always appears in actual discourse in relation to other modes. So, multimodality claims that the meaning of the discourse can only be understood if the relation of the different resources can be demonstrated (Menendez 2012). It is not only a matter of describing the previous meaning of the options in the different semiotic systems. Meaning is the result of how these different resources interact in discourse.
A discourse always presupposes an active subject who makes choices according to his or her interactional needs in a specifi c socio-cultural context - whether or not these choices are deliberate or their effects consciously intended. The subject produces this discourse in a particular register (Halliday 1978) following the co-nventions of use of a particular discursive genre (Bakhtin 1944). The multimodal perspective adds to this approach the perception that choices are made not only at the level of verbal language, but also simultaneously at other levels.
Within a social semiotics, different semiotic systems have different . grammar Grammar can be described as a meaning potential and together the different se-miotic systems make up a ‘paradigm’. A paradigm is always a reconstruction. It represents the various options of each system and allows the discourse of a culture to be formulated as a set of all the potential options. A particular discourse is the realization of the paradigmatic options from the different systems as chosen by the subject in a particular situation.
The different semiotic systems are paradigmatic disjunctions (X or Y). Like the linguistic sign, discourse can be located on two axes: the axis of simultaneity and that of succession. Indeed, from a multimodal perspective, discourse can be characterized precisely as simultaneity in succession. Therefore, a mode never ap-pears in isolation but always in relation to other modes. It is through the simulta-neous interaction of the modes that a strategic combination appears. It is important to note that there is that no way modes can be ranked in order of importance. The importance of one mode over others depends on register and genre.
As mentioned earlier, one of the aims of multimodal discourse analysis is to describe and explain the different resources combined through discourse strate-gies (Menéndez 2000). A strategy is a plan with a particular goal. This paper aims to show that that multimodality needs to focus on strategies precisely because
different resources are combined in each of the modes that constitute a particular discourse. Discourse analysis is basically strategic since it sees discourse as a combining of semiotic systems (understood as available options), register (under-stood as options realized as discourse, i.e. resources) and genre (under(under-stood as a convention of use). It should be noted, however, that discourse analysts normally focus on the resources they consider most important for the task in hand, even if other resources may also be present.
Register has been characterized as a variety of language in use determined by situation (Halliday 1978, Ghadessy 1993). Traditionally, it has been characterized by three variables: fi eld, tenor and mode. However, there are two basic problems with this characterization. The fi rst is its static nature; the second is that it depends directly on the functions of language used by systemic-functional linguistics to describe grammar.
Discourse genres have been traditionally defi ned as conventions of language use (Bakhtin 1944, Hasan 1994, 1995). These conventions shape discourses in so far as they act as pre-conditions of possible interpretations. Genre is thus the cultural context that enables a discourse to be interpreted in a certain way. In this sense, genre analysis is based on and conditioned by grammatical description and strategy analysis.
The question is how to explain the relationship between register and discourse strategies. Register is realized by discourse strategies within the framework of a genre. Any language variation shown within a given register in relation to diffe-rent resources is interpreted within this framework, which acts as a set of guideli-nes for interpretation (Menéndez 2010, 2012).
Just as grammar can describe any semiotic system, discourse can explain the functioning of any strategy, and genre can frame an interpretation for any conven-tion of use, so the choices a speaker makes can be interpreted according to the genre that frames them. Discourse, then, is doubly conditioned. On the one hand, it depends on the set of options speakers actually have (ideally all the possible options they might have as members of a particular discourse community); on the other hand, it depends on the genre conventions that shape the culture to which the speakers belong.
A discourse genre can be considered as the intersection of a set of recurring discursive strategies and a set of dominant cultural features. These features, of course, are not explicit because they are part of the cultural and communicative competence of the members of a society (Hymes 1972). Therefore, certain sets of strategies are expected in certain genres; but this does not mean they have to be displayed. These strategies establish the register and the different styles that can be found within it. From this point of view, the strategic dimension of genre is
clear and can explain how discourse is interpreted in real life.
Cultural features are present in all discourses, but their degree of activation and prioritization determine possible interpretations. For example, all discourses have a political dimension from the point of view of genre, but this dimension is not always useful for interpreting the discourse: it may be important, but it does not have to be.
So, a discourse genre should be considered as a general framework of cultu-ral and historical interpretation. It comprises both cultucultu-ral features and a set of recurrent discursive strategies. The relationship between these is what makes a particular interpretation possible and any interpretation is always framed by genre. Above all, it depends on combining semiotic resources in the form of plans - in other words, discursive strategies.
3. An example
To show how a multimodal perspective can help to clarify discourse, let us consider two situations through which a teacher of Spanish as a second language might wish to present a grammar item. The teacher decides to use two interactio-nal situations from different fi lms, but we will use them in order to ainteractio-nalyze the interaction that takes place in them from a strategic point of view.. It is very im-portant to point out that it does not matter which fi lms are used; for our purposes what matters is the situation from which the meaning of a dialogue can be inferred and explained in relation to the different modes that can be analyzed.
SOCIAL SEMIOTICS PARADIGMATICS System of available options MULTIMODAL DISCOURSE REGISTER Context of situation GENRE Context of culture Semiotic System A Option 1 Option 2 Option 3 ... Option n Semiotic System B Option 1 Option 2 Option 3 ... Option n Semiotic System C Option 1 Option 2 Option 3 ... Option n MODE A Recourse 1 MODE B Recourse 1 MODE C Recourse 1 MULTIMODALITY SINTAGMATICS Set of realized resources
Each of the two situations shown in Figure 2 (below) simultaneously combines a set of resources from different modes by means of a discourse strategy. The photographs show clearly the multimodal nature of any discourse, because they allow us to infer the register, i.e., the variation of language use according to the situation.
Figure 2: Two fi lm dialogues
The language teacher’s fi rst task would be to analyze the situations and then select suitable content for the Spanish as a second language class. However, the background knowledge required for this analysis does not need to be taught. It represents the minimum knowledge necessary to develop the teaching materials if the materials are to be successful. It is here that the multimodal approach and the strategic method are important.
The general theme of the Spanish lesson is “attitudes of the speaker speaker,” a topic which traditional grammars call “modality” and which Systemic Functional Linguistics explains by taking mode as the realization of interpersonal function (Halliday and Mathiessen 2004). The language function or strategy to be taught is “Asking for something something.” In informal situations this function is typi-cally realized by the imperative, but this resource could also be interpreted as an order or as an appeal depending on situational variables.
In order to describe the strategy involved in choosing a suitable exponent to perform the language function, two elements need to be specifi ed:
1) Cultural Assumptions (degree of knowledge) 2) Intervening Modes (strategic analysis)
A cultural assumption is an idea or knowledge that is common throughout a community. Of course, not all the members of a community share exactly the
same cultural assumptions and the degree of conformity to in-group expectations varies among individuals. But individuals must share some assumptions in order to be a part of their community. Cultural assumptions are part of our cognitive environments (Sperber and Wilson 1996).
Table 1 shows the different degrees to which cultural assumptions may be reco-gnized in a particular context - in this case, the context of a fi lm. The continuum ranges from complete recognition to complete lack of recognition depending on the student’s cultural background and familiarity with the fi lm. Obviously, not all students of Spanish as a second language will share the Anglo-Saxon assumptions of the A Streetcar Named Desired or An Affair to Remember.
Cultural Assumptions Situation 1 Situation 2 + known [degree 1] A Streetcar Named Desire An Affair to Remember +/- known [degree 2] A fi lm starring Marlon Brando
and/or Vivien Leigh
A fi lm starring Cary Grant and/or Deborah Kerr - known [degree 3] A movie A movie -/- known [degree 4] Not identifi ed Not identifi ed
Table 1: Degree of recognition of Cultural Assumptions
Table 2 shows the important intervening modes in relation to the goal of the analysis (teaching Spanish as a second language).
Discourse Strategy: Asking for something
Mode Resources (realized options)
Situation 1 Situation 2
1.Plane General American
2.Space House (private) Deck of a ship (semipublic)
3.Time 1950s 1950s
4.Gestuality
4.1.Gaze Woman at object Man at woman
Man’s and woman’s eyes meet
4.2.Arms Man towards object Woman towards object
Man stretching out his arm with a note. Female hand rests on his chest. 5.Verbal (possible dialogue ) Deme /Dame esto ya
(Give that [now/right now] )
Por favor, tomá/tome esto (Please, take this) 6.Dress Man : informal
Woman: formal
Formal
7.Type of interaction Impolite (violent) Polite (tense)
The focus of interest here is the “attitudes of the speaker speaker.” These can be identifi ed in both fi lms by comparing the way the characters combining resources from different modes in order to further their different goals. In fact, the same modes are salient in both interactions but they can be interpreted quite differently.
SOCIAL SEMIOTICS PARADIGMATICS
System of available options
GENRE: Ordinary interaction Semiotic System Gestuality Semiotic System Verbal Semiotic System Interactional relationship MULTIMODAL DISCOURSE Set of realized resources REGISTER Gaze/Arms Imperative Impolite (violent) Deme/Dame eso ya (Give me that now)
Figure 3: Multimodal analysis of a scene from A Streetcar Named Desire
SOCIAL SEMIOTICS PARADIGMATICS
System of available options
GENRE: Ordinary interaction Semiotic System Gestuality Semiotic System Verbal Semiotic System Interactional relationship MULTIMODAL DISCOURSE Set of realized resources REGISTER Gaze/Arms Imperative Polite (tense) Por favor, Tome/tomá esto (Please, take this)
For example, as mentioned earlier, the imperative could be used not only to give orders. Its function will depend on how it is combined with other modes. In the two examples shown above, this verbal resource is combined with gesture (direction of the gaze and arms) and the degree of politeness in each situation. When the situation is violent, the imperative is the expected resource for giving orders. When there is tension (involving a certain degree of violence mitigated by other factors, such as social class and education) the imperative may be used to make a request. In the two examples above, the inferred dialogue is shown by additional discourse markers: “now” (“ya”) in Situation 1; and “please” (“por favor”) in Situation 2.
4. Conclusion
This paper has shown how multimodal discourse analysis can be applied in practice. The analysis focuses on strategies because these can be used to recon-struct discursive planning in particular situations. So, the complexity of discourse can be accounted for in terms of different combinations of resources from various modes.
According to Verscheuren (1999), the understanding of meaning requires a “pragmatic perspective” based on the megastructure of language (Verschueren 1999). Following my own formulation of this concept (Menéndez 1997) shown in Figure 5, the present paper has focused mainly on the second feature (negotia-bility) in order to demonstrate that a rhetorical perspective has to be understood mainly as a strategic perspective.
VARIABILITY option in each paradigms
of a semiotic system NEGOTIABILITY combination of resources as discursive strategies ADAPTABILITY Genre adecaution of discursive strategies MULTIMODAL PRAGMATIC PERSPECTIVE
References
Albaladejo Mayordomo, T. (2005), “Retórica, comunicación, interdiscursividad.” Revista de
Investigación Lingüística 8: 7-33.
Anscombre, J. A., Ducrot, O. (1983), La argumentación en la lengua. Madrid: Gredos.
Bakhtin, M. (1944), “El problema de los géneros discursivos.” In Estética de la creación verbal.
México: Siglo XXI.
Brown, G., Yule, G. (1983), Discourse Analysis. Cambridge: CUP.
Carel, M., Ducrot, M. (2005), La semántica argumentativa. Una introducción a la teoría de los
bloques semánticos. Buenos Aires: Colihue.
Fairclough, N. (1992), Discourse and Social Change. Cambridge: Polity Press. Fairclough, N. (1995), Critical Discourse Analysis. London: Longman.
Ghadessy, M. (1993), Register Analysis: Theory and Practice. London: Routledge and Kegan Paul. Gumperz, J. (1982), Discourse Strategies. London: Cambridge University Press.
Halliday, MAK. (1978), Language as Social Semiotics. London: Arnold.
Halliday, MAK. (1994), “Systemic Theory.” In The Encyclopaedia of Language and Linguistics,
Volume 8, edited by R.E. Asher. Oxford: Pergamon Press.
Halliday, MAK. (1997), “Ideas on Language.” In On Language and Linguistics, Collected Papers
Volume I, edited by Jonathan Webster. London: Continuum.
Halliday, MAK., Hasan, R. (1976), Cohesion in English. London: Longman.
Halliday, MAK., Mathiessen, Ch. (2004), An Introduction to Functional Grammar. London: Arnold. Hasan, R. (1994), “Situation and the Defi nition of Genres.” In What´s going on here?Complementary
Studies of Professional Talk, edited by A.D. Grimshaw. Norwood: Ablex.
Hasan, R. (1995), “The Conception of Context in Text.” In Discourse and Society: Systemic Functional
Perspectives, edited by P. Fries, M. Gregorey. Ablex: Norwood.
Hodge, R. and Kress, G. (1998), Social Semiotics. New York: Cornel University Press.
Hymes, D. (1972), On Communicative Competence. Philadelphia: University of Pennsylvania Press. Jewitt, C. (ed.) (2009), The Routledge Handbook of Multimodal Analysis. London: Routledge. Kress, G. (2010), Multimodality. A Social Semiotic Approach to Contemporary Communication.
London: Routledge.
Lavandera, B. (1985), Curso de lingüística para el análisis del discurso. Buenos Aires: CEAL. Lavandera, B. (2014), Variación y signifi cado. Y discurso. Buenos Aires: Paidós.
Leech, G. (1983), Principles of Pragmatics. London: Longman. Levinson, S. (1983), Pragmatics. Cambridge: CUP.
Martin, J. and Rose, D. (2003), Working with Discourse. London: Continuum.
Menéndez, S M. (1997), Hacia una teoría del contexto discursivo. Ph.D. Dissertation. Facultad de
Filosofía y Letras, Universidad de Buenos Aires. Unpublished.
Menéndez, S M. (2000), “Estrategias discursivas: principio metodológico para el análisis pragmático
del discurso.” In: Lengua, discurso, texto (I Simposio Internacional de Análisis del discurso),edi-ted by J.J. de Bustos and P. Chareadeau,1203-1219. Madrid: Visor.
Menéndez, S M. (2005a), Gramática, análisis del discurso e interpretación crítica: las relaciones no
tan evidentes. In Proceedings of the International Conference on Critical Discourse Analysis. Universitat de Valencia, Valencia, CD.
Menéndez, S M. (2005b) ¿Qué es una estrategia discursiva?. In Actas del Congreso Internacional:
Debates Actuales. Las teorías críticas de la literatura y la lingüística, edited by S. Santos and J.
Panesi, Facultad de Filosofía y Letras, UBA, Buenos Aires. CD.
sistémico-funcional.” Tópicos del seminario, 23: 221-239.
Menéndez, S M. (2010b), “Evaluación y estrategias discursivas. El caso de las proclamas militares en
la Argentina (1930-1976)”. Páginas de guarda 10: 28-41.
Menéndez, S M. (2012), “Multimodalidad y estrategias discursivas: un abordaje metodológico.”
Revista Aled 12(1): 57-74.
Pujante, D. (2003), Manual de retórica. Madrid: Castalia.
Sperber D., Wilson, D. (1986), Relevance: Communication and Cognition. Harvard: Harvard
University Press.
Verschueren, J. 1999. Understanding Pragmatics. London: Arnold.
van Dijk, T. (1984), Prejudice and Discourse. An Analysis of Ethnic Prejudice in Cognition and
Conversation. Amsterdam: John Benjamins.
van Dijk, T., Kinstch, W. (1982), Strategies for Discourse Comprehension. New York: Academic
Press.
van Eemeren, F.H. (ed) (2009), Examining Argumentation in Context. Fifteen Studies on strategic
