Scholarly article on topic 'Motor cognition–motor semantics: Action perception theory of cognition and communication'

Motor cognition–motor semantics: Action perception theory of cognition and communication Academic research paper on "Psychology"

CC BY-NC-ND
0
0
Share paper
Academic journal
Neuropsychologia
OECD Field of science
Keywords
{Action / Perception / Language / Meaning / Pragmatics / "Social interaction" / "Mirror neuron" / "Action perception circuit" / "Cell assembly"}

Abstract of research paper on Psychology, author of scientific article — Friedemann Pulvermüller, Rachel L. Moseley, Natalia Egorova, Zubaida Shebani, Véronique Boulenger

Abstract A new perspective on cognition views cortical cell assemblies linking together knowledge about actions and perceptions not only as the vehicles of integrated action and perception processing but, furthermore, as a brain basis for a wide range of higher cortical functions, including attention, meaning and concepts, sequences, goals and intentions, and even communicative social interaction. This article explains mechanisms relevant to mechanistic action perception theory, points to concrete neuronal circuits in brains along with artificial neuronal network simulations, and summarizes recent brain imaging and other experimental data documenting the role of action perception circuits in cognition, language and communication.

Academic research paper on topic "Motor cognition–motor semantics: Action perception theory of cognition and communication"

Neuropsychologia I (I

ELSEVIER

Contents lists available at ScienceDirect

Neuropsychologia

journal homepage: www.elsevier.com/locate/neuropsychologia

Motor cognition-motor semantics: Action perception theory of cognition and communication

Friedemann Pulvermüller a,n, Rachel Moseleyb, Natalia Egorovab, Zubaida Shebanib, Véronique Boulengerc

a Brain Language Laboratory, Department of Philosophy and Humanities, Freie Universtät Berlin, Habelschwerdterallee 45, 14195 Berlin, Germany b Medical Research Council, Cognition and Brain Sciences Unit, Cambridge CB2 7EF, UK c Laboratoire Dynamique Du Langage, CNRS/Université Lyon 2, UMR 5596, 69007 Lyon, France

ARTICLE INFO

Keywords:

Action

Perception

Language

Meaning

Pragmatics

Social interaction

Mirror neuron

Action perception circuit

Cell assembly

ABSTRACT

A new perspective on cognition views cortical cell assemblies linking together knowledge about actions and perceptions not only as the vehicles of integrated action and perception processing but, furthermore, as a brain basis for a wide range of higher cortical functions, including attention, meaning and concepts, sequences, goals and intentions, and even communicative social interaction. This article explains mechanisms relevant to mechanistic action perception theory, points to concrete neuronal circuits in brains along with artificial neuronal network simulations, and summarizes recent brain imaging and other experimental data documenting the role of action perception circuits in cognition, language and communication.

© 2013 Published by Elsevier Ltd.

1. Introduction

A respectable position views motor brain mechanisms as a slave system under the dictate of cognition. Accordingly, modality-specific sensory modules channel information to the central systems for attention, memory, language, concepts, decisions and perhaps even "the soul", which, in turn, drive the motor output (for illustration, see Fig. 1, and see Fodor, 1983; Hubel, 1995). In this perspective, perception and action mechanisms work as functionally isolated ("encapsulated") modules and their respective main purposes are to filter and preprocess sensory information for cognitive operations and to dress the cognitive operations into acts of motor movement.

A radically different position has been developed by Marc Jeannerod and his friends and colleagues (Boulenger et al., 2008; Braitenberg & Schüz, 1998; Fuster, 2003; Jacob & Jeannerod, 2005; Jeannerod, 1994, 2001, 2006; Jeannerod, Arbib, Rizzolatti, & Sakata, 1995; Pulvermüller & Fadiga, 2010; Rizzolatti & Sinigaglia, 2010). In this view, cognition is built from action and perception, upon sensory and motor brain mechanisms. The action schema of GRASPING an object would thus not be a motor process only, but would connect with sensory information about small or large size (of the to-be-grasped object) and the integrated representation of

* Corresponding author. Tel.: + 49 30 838 54443; fax: + 49 30 838 52273. E-mail address: friedemann.pulvermuller@fu-berlin.de (F. Pulvermüller).

0028-3932/$-see front matter © 2013 Published by Elsevier Ltd. http://dx.doi.org/10.1016/j.neuropsychologia.2013.12.002

motor and sensory properties of the action would become the vehicle of active memory, attention-to-action, and action-related meaning or motor semantics (Fig. 2, see also Fuster, 2003, 2009). Action-supporting parts of the brain would thus change their role in corticofunctional interplay, from slave to master.

Eminent brain-theoretical arguments support the latter position, both structural and functional in nature. Anatomically, motor as well as sensory systems of the brain are not isolated modules. They are strongly connected reciprocally with both adjacent modality-preferential areas as well as more distant multimodal convergence and integration "hubs" on which other sensory and motor systems likewise converge (Braitenberg & Schuz, 1998; Sporns, 2013; Sporns, Tononi, & Kotter, 2005; Young, Scannell, & Burns, 1995). Physiologically, a major driving force of learning and plasticity in the brain is the correlation of the firing of nerve cells. Connected neurons that frequently fire together in synchrony are bound together more closely at a functional level, whereas neurons firing independently of each other, or in an antiphasic manner, weaken their mutual links; even fine-grained temporal relationships in firing patterns can be mapped by spike-timing dependent plasticity (Bi & Wang, 2002; Tsumoto, 1992). When learning to GRASP an object, the correlation between perceived object size and the motor feature of grasp aperture can be mapped by the correlated activity of nerve cells encoding these parameters so that a circuit forms that spans frontocentral motor, premotor, parietal and occipital-visual cortex. The circuit links neurons in motor and sensory areas by way of nerve cells in

F. Pulvermüller et al. / Neuropsychologia I (l

Fig. 1. Illustration of the organization of perception, action and central systems of the cortex as viewed in the modular tradition. Systems for perception, higher cognition and motor output are viewed as functionally segregated and linked in sequence. Adopted from Hubel (1995).

Fig. 2. Top panel: action perception circuits (APCs) that link motor and sensory information about actions. When learning to pronounce a syllable or word form, co-occurring activity in neurons in articulatory motor cortex (M1) and in primary auditory cortex (A1) will trigger correlated neuronal activity in areas connecting these sites, including premotor (PM), prefrontal (PF), auditory belt (AB) and parabelt (PB). Neuronal correlation and the available neuroanatomical connections lead to the formation of distributed neuronal assemblies for spoken word forms (Garagnani et al., 2007, 2008). The inset shows the approximate cortical distribution of such an articulatory-auditory APC, the putative correlate of a word form such as "grasp". Bottom panel: Phylogentic development of major long distance reciprocal cortico-cortical connections between inferiorfrontal and temporal cortex in humans. Long distance bundles are shown from left to right for the macaque, chimpanzee and human brains (adopted from Rilling et al., 2008). Note the strong dorsal connection from inferior-frontal, precentral and central sulcus (IFS, PrCS, SC) regions to areas around the inferior-parietal and superior-temporal sulcus (IPS, STS) of the arcuate fascicle, which is only present in humans.

adjacent and convergence areas (including "hubs") in the service of providing the machinery for an action schema, that is an ordered sequence of motor movements related to sensory-perceptual features, which can be adjusted to specific features of objects and contexts. Cortico-cortical links between the relevant motor, sensory and higher convergence areas (see Damasio, 1989) along with the correlated sensory and motor firing strongly argue for the existence of such action perception circuits, or APCs, as a basis of motor function (Jeannerod, 2006). If APCs carry action representations and link together sensory and motor information about actions, it appears straightforward to postulate that they should also be activated when actions, which the individual regularly performs, are being perceived (Jeannerod et al., 1995).

There is overwhelming evidence for the existence of APCs. The proof of mirror neurons active during execution of specific hand actions and likewise during visual perception of these same actions, along with their presence not only in one single area but in both premotor and inferior-parietal cortices, is strong support for the existence of action-specific links that integrate motor and sensory information about specific action schemas (Kohler et al., 2002; Rizzolatti & Sinigaglia, 2010). Action perception links exist not only for manual and mouth actions and their related visual

and acoustic perceptions, but equally in the case of human language, for linguistic-articulatory action schemas and their auditory counterparts (Fadiga, Craighero, Buccino, & Rizzolatti, 2002; Pulvermüller & Fadiga, 2010; Pulvermüller et al., 2006; Watkins, Strafella, & Paus, 2003; Yeung & Werker, in press). Further converging evidence comes from saccade-related sensor-imotor neurons (Bruce & Goldberg, 1985) and from multimodal memory cells in prefrontal, premotor, anterior-temporal and parietal cortex, which provide additional support for the claim that the cortical representations of stimuli affording actions are organized by multimodal circuits spanning cortical areas (Fuster, 2003). In theory, it might be possible that single sensorimotor or mirror neurons bind action and perception information and cause these sensorimotor interaction effects, but, because the influence of single cortical cells on other cortical neurons is known to be weak and certainly could not change the metabolism of an entire area, it appears more realistic that these sensorimotor interaction effects are caused by neuronal assemblies of hundreds to thousands of neurons acting together as functional units (Abeles, 1991; Braitenberg & Schüz, 1998; Fuster, 2003).

Action perception circuits conceived as distributed neuronal assemblies establish a functional link between specific sets of

F. Pulvermüller et al. / Neuropsychologia I (llll) lll-lll 3

sensory and motor information. As such they provide a mechanism for what Braitenberg and Schuz consider the primary function of the neocortex, namely "information mixing", that is the joining together of specific information across modalities (Braitenberg & Schuz, 1998). If a motor, sensory or intermediary neuron (which connects sensory and motor neurons) each exhibit unimodal or amodal response properties before learning, the associative learning mechanism strengthening links between these motor, sensory and intermediary connecting neurons will make them members of an APC and thus a multimodal information mixing device. As a consequence of APC formation, each neuron included in an APC will be a multimodal sensorimotor neuron. Crucially, specific sets of motor and sensory information are being joined together in distributed neuronal assemblies, each of which provides a mechanism for one specific type of action, or action schema. The need for such a mechanism is particularly apparent in the language domain, where tens and hundreds of thousand different word forms, pairs of articulatory-motor and acoustic-perceptual schemas, are being learned within a few years to set up a rich language-specific repertoire. Note that these language specific lexicons cannot be explained by an inborn or epigenetic mechan-ism(Braitenberg & Pulvermuller, 1992; Braitenberg & Schuz, 1992; Pulvermuller, 1996). The explanation requires cortico-cortical long-distance neuroanatomical links between relevant areas (for language in inferior-frontal and superior-temporal cortex) and correlated activation of both motor and sensory neuron sets. The necessary high correlation of motor and sensory activation patterns is guaranteed and explained by the mapping of actions performed by the individual and the perceptual aspects of these same actions (see next section below). As information about an action and its corresponding perceptions are joined together, action perception circuits also provide a mechanism for mirror neurons. Repetitive body action along with the Hebbian correlation learning principle and cortico-cortical connections between relevant sensorimotor and multimodal areas of cortex, therefore provide an explanation why action perception circuits emerge and, thus, why mirror neurons and other sensorimotor neurons exist (Pulvermuller & Fadiga, 2010). In neurocomputational studies, mirror neuron activity could indeed be shown to emerge as a consequence of Hebbian-associative learning of actions and their corresponding perceptions (Garagnani, Wennekers, & Pulvermuller, 2007; Hanuschkin, Ganguli, & Hahnloser, 2013). Furthermore, recent experiments in humans showed that mirror activity in the precentral cortex is indeed tied to aspects of action perception learning, an observation which further strengthens this position (e.g., Pulvermuller, Kiff, & Shtyrov, 2012). It might therefore seem appropriate to consider mirror mechanisms not as fundamental, but, instead, as a consequence of (a) action performance and self-perception, (b) the fundamental neuroscience principle of correlation learning and (c) pre-established cortico-cortical connectivity. We note, however, that for mirror activity already observable very early in life for a limited set of actions (e. g., tongue protrusion in newborn babies, Lepage & Theoret, 2007), it is important to clarify whether correlation learning in a pre-structured network is relevant - or whether perhaps genetic and epigenetic mechanisms might be sufficient.

2. Human-specific action perception circuits

Mirror-like action perception integration mechanisms have long been envisaged to play a role for specifically human capacities, especially language (Braitenberg, 1980; Fry, 1966; Galantucci, Fowler, & Turvey, 2006; Liberman, Cooper, Shankweiler, & Studdert-Kennedy, 1967). The earliest specifically human developmental stage in early ontogeny is the babbling phase (months 6-12), which is characterized

by repetitive syllable articulations, so-called "babbling", and similarly repetitive motor behavior involving all extremities (Locke, 1993; MacNeilage & Davis, 2000; Petitto & Marentette, 1991). Over time, babbling becomes more variable and, along with that, closer to the phonological structure of the language spoken by care-takers (de Boysson-Bardies & Vihman, 1991; Vihman, Macken, Miller, Simmons, & Miller, 1985). Finally, towards the end of the babbling phase, the infant repeats the first words spoken by adults (Locke, 1993). Babbling and repetitive body movements (including "manual" and "whole body babbling") during the second half of the first year have the obvious brain-internal consequence that motor activities are being mapped onto their sensory consequences. Therefore, speech articulatory gestures are mapped onto the sounds produced by these gestures, hand motor movements onto visual perceptions thereof, and all body actions can be linked with their somatosensory effects (Braitenberg & Pulvermuller, 1992; Braitenberg & Schuz, 1992). When a syllable such as "ba" is articulated, articulatory-motor and speech planning neurons in inferior-frontal primary motor (M1), premotor (PM) and adjacent inferior prefrontal (PF) cortex are active along with neurons in the superior-temporal cortex stimulated by the specific self-produced input, which are localized in auditory cortex (A1) and the adjacent areas of the auditory belt (AB) and parabelt (PB).1 As neuroanatomical connections linking inferior-frontal and superior-temporal cortex are available and correlation learning is effective, specific APCs for phonemes and syllables may be built in the babbling phase. The phonological APCs may contribute to the infant's ability to repeat words spoken by others which, in turn, is a crucial step in building the large vocabulary of spoken word forms so characteristic of humans (Braitenberg & Pulvermuller, 1992; Pulvermuller, 1999). As explained in more detail below, APCs for word forms and linguistic actions provide a mechanistic basis for social-communicative interaction. Similar action perception mapping processes brought about by human-specific "manual babbling" may lead to the formation of APCs including mirror neurons for hand actions in humans, which may, likewise, be critical for the learning of the extended repertoire of hand actions and facial expressions typical for humans (Locke, 1993), which plays a key role in human action and social interaction (Del Giudice, Manera, & Keysers, 2009; Keysers & Perrett, 2004).

Note that the outlined action perception perspective offers makes new predictions. For example, mirror neurons for speech sounds and word forms should be present in both frontoparietal as well as superior-temporal areas of the human left-perisylvian language cortex. APT also offers explanations outside the reach of some recent "association accounts" of mirror neuron activity. It has been stated that mirror neurons might be a consequence of associative learning between arbitrary motor and sensory patterns (see, for example, Heyes, 2010). This approach cannot explain why monkeys have a hard time learning the mapping of sounds on corresponding motor schemas, whereas humans easily learn a rich vocabulary of speech sounds and words. It is as if the relevant auditory-motor link was much reduced in the former from the very start, which suggests a neuroanatomical explanation (see discussion below). A tabula-rasa associationist take has further difficulty explaining that mirror neurons are commonly observed in macaques even in experimental animals without special training (e.g., for recognizing GRASPING actions). Crucially, information mixing and mirror neuron activity

1 It has been suggested "that the modalities of action and perception are integrated at the level of the sensorimotor system itself and not via higher association areas" ( Gallese & Lakoff, 2005, p. 459). Although there is indeed strong evidence for the integrative multimodal role of areas formerly considered modality-specific, the conclusion that multimodal (or "supramodal") association areas do not play a role seems too strong. As important functional links between sensory and motor areas are by way of multimodal cortex (for discussion of recent neuroanatomical data, see Garagnani & Pulvermuller, 2013), these higher multimodal hubs can contribute to the binding of sensorimotor information in action and semantic processing.

F. Pulvermüller et al. / Neuropsychologia I (I

cannot be explained by associative learning alone, but require specific cortico-cortical connectivity between relevant sensory and motor areas along with action perception mapping of information, which provide the substrate and basis for such learning. In contrast to naive associationism, the action perception model outlined here and elsewhere (Braitenberg & Pulvermüller, 1992; Del Giudice et al., 2009; Pulvermüller, 1999; Pulvermüller & Fadiga, 2010) puts that the "glue" between the circuits' action and perception parts is provided by both pre-established neuroanatomical connectivity and neuro-plasticity caused by sensorimotor correlations during self-performed actions.

This action perception account offers a coherent set of answers to three so far still open questions about language: why humans, and only they, can build huge vocabularies of tens and hundreds of thousands of words, why (only) human infants babble, and why human language functions are typically lateralized to the left hemisphere. All of these questions are answered by action perception theory in light of neuroanatomical facts. The build-up of a huge vocabulary requires the possibility for excessive articulatory-auditory mappings and thus rich connections between inferiorfrontal and superior-temporal areas. The arcuate fascicle provides these rich neuroanatomical fronto-temporal links. As it is strongly developed only in humans, but not in other primates, where it has been reported to be absent or rudimentary (Rilling et al., 2008), it may provide the specific substrate for building a large "vocabulary" of articulatory-auditory APCs, which may, in turn, be a prerequisite for a large vocabulary of spoken words. Note this statement does not imply that the strong left-fronto-temporal connectivity in itself "explains" the huge human mental lexicon, but, instead, that such connections are a necessary condition for rich human vocabularies. The arcuate fascicle may also contribute to human-specific articulatory babbling behavior, as, with acoustic input, it channels auditory input to the articulatory system, and may, therefore, lead to generally enhanced levels of activity in motor systems. Although there is, as to our knowledge, no direct evidence at the physiological level for such enhanced motor excitability, the articulatory and manual (actually whole-body) babbling present in the second half year of human life strongly suggests a general state of enhanced motor activity as its origin, a human-specific hyperactivity (high arousal) stage during months 6 to 12 so to speak. And finally, because the arcuate fascicle is more strongly developed in the left hemisphere than in the right in most individuals (Catani, Jones, & Ffytche, 2005; Nucifora, Verma, Melhem, Gur, & Gur, 2005), and action perception learning of articulatory information may preferentially engage the left hemisphere, the well-known left-laterality of language in most humans receives a putative neuroscience explanation.

A crucial experimental test of the role of the left-lateralized arcuate fascicle in vocabulary build-up can be undertaken by having healthy human subjects learn new word forms with the predictions in mind that cross-modal articulatory-auditory correlation of information results in formation of APCs distributed over superior-temporal and inferior-frontal cortices. Cross-modality learning of articulations and their resultant auditory speech sounds would be mapped by APCs relying on the left-lateralized arcuate fascicle. Crucially, both inferior-frontal activation in speech perception and left-laterality of speech-elicited brain activity would only be expected for types of language learning that provide correlated auditory-articulatory information - for example when new word forms are articulated overtly (articulatory learning), but not if subjects learn new words just by listening to them (perceptual learning). Intriguingly, articulatory learning of novel spoken word forms by repeating heard items led to an increase of brain responses to these stimuli, with additional cortical sources appearing specifically in left-inferior-frontal and premotor cortices. In the control condition of perceptual learning of novel

Fig. 3. Enhanced cortical activity (sources of event-related potentials) elicited by novel spoken word forms, which had been learned either in an articulatory condition, by hearing and repeating the novel items (top panels), or in a perceptual learning paradigm, by listening to repeated presentations of the same novel spoken word forms (bottom panels). Enhanced bilaterally symmetric superior-temporal activity to learned novel items (orange ovals, relative to before learning) is seen independently of the learning method. However, additional enhanced fast automatic left-lateralized premotor, inferior-frontal and inferior-parietal activity to spoken novel words (yellow ovals) is seen after articulatory, but not after perceptual learning (Pulvermüller et al., 2012). These results suggest that language laterality is driven by auditory-motor information linking, possibly by way of the left-lateralized arcuate fascicle. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

spoken word forms, the increase of cortical activity was due to bilaterally symmetric superior-temporal sources, without inferiorfrontal contribution. As articulatory learning specifically led to both inferior-frontal and premotor activity enhancement to learned speech items and this activity was also left-lateralized as predicted, these results provide support for action perception theory of novel word learning and hence for a role of fronto-temporal connections and especially the arcuate fascicle in such learning (Fig. 3; Pulvermüller et al., 2012). So, over and above the insight that "cerebral lateralization is not fixed from birth (...), can change with age" and may index atypical language acquisition (Bishop, 2013), the laterality of language can be linked to specific features of language learning, including the Hebbian mapping of correlations between self-produced actions and resultant perceptions, which may be essential for such learning.

3. Mirroring is not enough: action perception theory of cognition

Action perception circuits (APCs or learned mirror circuits) offer a mechanism for repetition. They provide the information necessary for the repetition process as they map motor knowledge on seen, heard or otherwise perceived actions - manual motor ones in the case of typical mirror neurons and articulatory actions in the speech domain. In the absence of motor output neuron activation and thus without performance of the respective movements, the activation of an APC can be understood as a simulation of an action (Jeannerod, 2006), or action thought. If a corresponding APC ignites due to the perception of a different individual performing a corresponding action, one can speak of motor resonance (Zwaan & Taylor, 2006) or alignment (Pickering & Garrod, 2004) between the interacting individuals. Such resonance is especially relevant in social-affective contexts where empathy and sharing of feelings is required; mirror neuron circuits for emotional behaviors seem to play a critical role here (Gallese, Keysers, & Rizzolatti, 2004; Jezzini, Caruana, Stoianov, Gallese, &

F. Pulvermiiller et al. / Neuropsychologia I (l

Rizzolatti, 2012). To the degree that bodies and action repertoires are similar between different individuals, the set of APCs in their different brains are similar, because they connect similar motor and perceptual schemas. Congruencies between bodies and action repertoires guarantee that a degree of similarity in brain activations is present when individuals interact, and when they perceive and produce complex actions, for example in language and music (Bangert et al., 2006; Pulvermuller, 2005; Zatorre, Chen, & Penhune, 2007). In conclusion, action perception mapping and mirror mechanisms have contributed substantially to a better understanding of the neurobiological mechanisms of a relevant set of cognitive functions.

However, the significant advance of demonstrating a brain basis for mirroring and simulation has been overshadowed by the belief purported by some that mirror mechanisms are sufficient for cognition. This is not the case, and it was Marc Jeannerod who, together with Pierre Jacobs, first pointed to essential limitations (Jacob & Jeannerod, 2005):

"Simulation is an automatic process which can be controlled or inhibited in the social context. The persistence of uncontrolled contagion, mimicry, mirroring and other forms of resonance beyond early developmental stages, or their reappearance in later life, would be deleterious in social relationships and would generate maladaptive responses" (Jeannerod, 2006, p. 150).

A recent faction in the literature has seized upon the points above and similar comments and tried to turn them into criticisms against the general perspective of mirror mechanisms and action perception circuits (for example, Borg, 2012; Heyes, 2010; Hickok, 2009; Hickok & Hauser, 2010; Lotto, Hickok, & Holt, 2009). In this context, it is of crucial importance to demonstrate perspectives and achievements of action perception theory and related empirical investigation, and to thus extend the brief paragraph above into a work program targeting neurobiological mechanisms of cognition and semantics so far unexplained by simple mirror or simulation theories. In particular, it seems to be of the essence to address questions such as the following: How, within a system of strongly interconnected APCs, can activity be kept within limits to guarantee network functionality? How are attention and competitive mechanisms implemented so that APC ignitions are under control? What, at the level of circuits, makes a symbol meaningful, and how can action perception theory address such "meaning making"? How can memory and decisions be modeled in APCs? And crucially: How do APCs connect with each other to provide a basis of social actions and interactions that go beyond mirroring and imitation? Answers to these hotly debated questions are necessary for providing a convincing action perception theory of cognition.

Therefore, the rest of this article will address these questions one by one, focusing on cognitive and neurofunctional issues previously under-represented in action perception theory and experiments, with the aim of opening fruitful research domains for the future. First, regulation and control mechanisms will be addressed asking how mechanisms for control, regulation and attention can be integrated within an action perception theory. Second, the brain basis of an overt movement will be contrasted with mechanisms that make such a movement or action meaningful. Third, the question of how an action is memorized, embedded into context and related to goals will be in the focus and some remarks about abstract combinatorial schemas will be added. Finally, an emphasis is put on the mechanisms underlying the processing of the same motor act performed in different social-interactive contexts and with different communicative function. All this will be done in an attempt to provide a neuromechanistic model of action, perception and language processing and to integrate attention, memory, goals and

plans, abstract combinatorial knowledge and social-communicative interaction with established neuroscientific wisdom.

4. Attention and regulation

There is a slight discrepancy between the statement that APCs provide an automatic mechanism for the mapping of sensory on motor information (see MarcJeannerod's statement cited above) and the need (pointed out there too) to control such activation flow in specific social and interactive contexts. How can a mechanism be both automatic and controlled? The obvious solution is that in default cases there is automatic activation but special circumstances may push the system towards a more constrained processing mode. A well-known example illustrates this change. At the end of their first year of life, towards the end of the babbling phase, infants become more and more likely to repeat back previously heard words. Such repetition performance can be considered an automatic -although not necessarily compulsory - process (Locke, 1993). During that time, infants also learn to supress the overt motor behavior so that listening without repetition becomes the default in subsequent years. The mechanisms for shutting down the motor output are not fully understood. In a neuromechanistic model of cortical function, the standard neurocybernetic solution is to introduce regulation and control mechanisms and adjust the gain of such regulation (Braitenberg, 1978; Garagnani, Wennekers, & Pulvermuller, 2008; Wennekers, Garagnani, & Pulvermuller, 2006), so that, with specific gain parameter values, the otherwise automatic motor mechanism is suppressed. If control mechanisms can shut down the motor output of APCs, some subliminal motor processes in the brain should still be observable, even in adults during speech perception. Notably, healthy adults not performing a motor (or other) task activate their premotor and inferior-frontal cortex when listening to, or when just passively hearing, speech and other action sounds (Fadiga et al., 2002; Hauk, Shtyrov, & Pulvermuller, 2006; Pulvermuller et al., 2006; Pulvermuller, Shtyrov, & Ilmoniemi, 2003; Wilson, Saygin, Sereno, & Iacoboni, 2004).2 Thus, the functional link between auditory and motor processes remains effective in the absence of overt motor output. This link seems to be automatically activated by the ignition of the APC, although feedback regulation can modulate the activation process.

That any brain requires regulation and control is obvious (Wiener, 1961), especially because a mechanism for keeping activity within certain bounds is necessary to prevent both extinguishing and overshooting "epileptic" activity (Braitenberg, 1978; Palm, 1990). Regulation mechanisms can be realised by feedback inhibition acting on local clusters of neurons and on specific areas of cortex (Elbert & Rockstroh, 1987; Knoblauch & Palm, 2001). A model of cortical areas that incorporates area-specific activation control can be used to learn sensorimotor patterns, as, for example, when the infants utters its first syllables and words and the cortex maps correlations between articulatory and auditory information processed by neuronal activation in auditory and motor areas. As motor-articulatory cortex (M1) and sensory-auditory cortex (A1) are not directly linked but, instead, by way of modality-preferential (premotor, PM, and auditory belt, AB, areas) and fully multimodal areas (especially inferior prefron-tal, PF, and anterior-, lateral-, and posterior superior-temporal parabelt, PB, cortex), the waves of activation brought about by

2 A controversy has recently been initiated regarding the status of such motor activation in speech processing. While established and recent data indicate a crucial role of motor systems in speech perception (Baker, Blumstein, & Goodglass, 1981; D'Ausilio et al., 2009; Moineau, Dronkers, & Bates, 2005), some authors have argued against a functional perceptual contribution (Rogalsky, Love, Driscoll, Anderson, & Hickok, 2011).

F. Pulvermüller et al. / Neuropsychologia I (l

Fig. 4. Regulation of cortical activation and sensorimotor learning in a model of the language cortex. Top panel: core areas of the left-perisylvian language cortex (inset) and model network of the language cortex including six connected "areas". Correlated activity in motor (M1) and auditory (A1) areas, as it is present during the pronunciation of a word form, leads to spreading waves of activation throughout sensorimotor and higher connecting areas (premotor, PM, prefrontal, PF, auditory belt, AB, auditory parabelt, PB), which give rise to the formation of trans-area neuronal assemblies or APCs. "Jumping" connections between Ml and PF, PM and BP etc. are also present but have been omitted for clarity (see Garagnani & Pulvermuller, 2013). One APC is illustrated by highlighted circles and lines. Bottom panel: a regulation mechanism controls activation levels in each area specifically and provides a neuronal basis for attention and task adjustment. The activation in a given brain area is compared with a target value; the discrepancy between target and actual activity determines the feedback (Braitenberg, 1978; Garagnani et al., 2008).

correlated activation of neuron populations in M1 and A1 lead to correlated activation patterns across a range of primary and higher areas and thus to the formation of distributed neuronal ensembles. These distributed ensembles are action perception circuits because they interlink specific motor and auditory patterns (Fig. 4; Garagnani et al., 2007, 2008; Wennekers et al., 2006). In such a network, the step from automatic repetition to withheld repetition is modeled by area-specific regulation, especially by increasing the gain of the regulation and therefore the degree of feedback inhibition in the motor cortex (M1).

Regulation and control mechanisms are not only essential for brain functionality, they are also highly relevant to cognition and represent prime candidates for mechanistically realising attention. The biased competition model of attention (Duncan, 2006) views attention as a combination of mechanisms for pre-activating (biasing) representations and establishing competition between them; the degree of such competition can be modulated. Modulation of feedback regulation in a model of local cortical areas and APCs alters the degree of competition between these circuits. Minimal feedback inhibition with activity enhancement means availability of ample attentional resources, whereas strong feedback inhibition means strong inter-APC competition and limited attention. With high attention, several

Fig. 5. Attention effects on language processing: theory and data. Top panels: brain response (MEG, on the left) and brain-model response (on the right) to words (in red) and pseudowords (in blue) when attention is directed towards these stimuli (solid lines/inset on the upper right) or away from them (broken lines/inset on the lower right). Note the great attention-related variability of responses to pseudo-words and the much reduced attention effect to words (adopted from Garagnani et al., 2009; Garagnani et al., 2008). Bottom panels: the significant interaction of lexicality (words vs. pseudowords) and attention is illustrated in the bar plot on the left. The strongest cortical sources underlying the attention effects for both words and pseudowords are present in left inferior-frontal cortex. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article). Adopted from Shtyrov et al. (2010).

representations may be allowed to be active together to a degree, so that, in case of perception of a meaningless "pseudoword" such as "crodocile", several competing word candidates may become temporarily active to a degree (for example those for "crop", "crocus" and "crocodile"). In contrast, with strong competition for attentional resources, such simultaneous activation of different circuits will be forcefully suppressed. As APCs are neuronal ensembles with strong internal connections, activation spreads efficiently within a single stimulated circuit, as it is the case if a specific meaningful familiar word is being perceived ("crocodile"). The single strongly activated and strongly connected APC is more difficult to supress by inhibition so that its activation dynamics predominate and are influenced less by variation of attention (inhibition). If sufficiently stimulated by the presence of the corresponding word in the input, the APC underpinning the word "crocodile" may therefore activate to similar degrees with and without attention-related competition/inhibition in the background.

The outlined attention-dependent processing differences between stored items, words (that ignite their corresponding APC), and novel ones, pseudowords (that only activate partly matching APCs), were confirmed by neuromechanistic modeling and neuroimaging experiments. Brain-model and real-brain responses did indeed vary greatly as a function of attention (competition) level for meaningless novel pseudowords, but much less so for meaningful familiar words (Fig. 5;

F. Pulvermüller et al. / Neuropsychologia I (I

Fig. 6. The semantic links between word and actions or objects the word is used to speak about may be realised by the merging of articulatory-auditory word form APCs -for example for the word form "grasp" - and the action- and object-related conceptual-semantic knowledge - for example the motor schema of GRASPING. Note that the hand action of grasping requires relating action knowledge to visual information, which involves both dorsal and ventral visual streams (Jeannerod et al., 1995). The inset illustrates the approximate cortical distribution of a semantic APC with both linguistic and conceptual parts. Referential semantic links are in red. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

Garagnani, Shtyrov, & Pulvermüller, 2009; Garagnani et al., 2008; Shtyrov, Kujala, & Pulvermüller, 2010).

These simulations and associated neuroimaging studies show that an associative learning model spelt out in terms of APCs can generate representations that behave in interesting ways. APCs can of course explain the automatic association of motor and sensory patterns, and therefore mirroring and simulation to stimulation in the absence of inhibitory control. Over and above mirroring, networks including APCs along with regulation and control mechanisms can explain attention and task effects, such as the great attention effect on pseudoword processing compared with the relatively small one on word processing. Task-dependent attention effects on brain activation to the same stimuli predominate in frontal cortices (Shtyrov et al., 2010). A neurobiological account of this topographical specificity is possible based on a perceptual ceiling effect: in the perception process, a given APC is always strongly activated at its sensory end, whereas prefrontal and motor parts receive only indirect and thus reduced input, so that activation dynamics due to regulation are most effective there (Garagnani & Pulvermüller, 2011).

5. Motor movements and meaning

Fig. 7. Manifestation of the sign-action link in motor system activation to action words. Maximal activation foci reported by a range of fMRl studies targeting the processing of verbs and nouns typically used to speak about face-, arm- or leg-actions ("talk", "grasp", "walk") and that of food and tool nouns whose referent objects afford mouth- or hand-actions. Activation foci for face-, arm- and leg-items are shown in green, red and blue, respectively. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

From Carota, Moseley and Pulvermuller (2012).

An action such as waving the hand or uttering a syllable can be viewed as an instantiation of a motor schema. Symbolic communication rests upon meaning carried by such movements. For a syllable or other movement to become meaningful, a semantic link needs to be established between the sign and something else. In the case of the monosyllabic word "grasp", a connection is made with a family of motor schemas or movement patterns typically performed with the hand. The word is used to speak about such movements. In brain mechanistic terms, an APC for an utterance, "grasp", may be linked to a different APC processing and storing the information about the motor movement the utterance is used to speak about. The brain basis of the meaningful symbol would thus be a higher-order APC resulting from merging two or more elementary APCs (Fig. 6).

Some words are used to speak about actions ("grasp"), objects ("bottle") and their sensory features ("red"); this implies that semantic links exist between these signs or utterances and motor or sensory information. Neuroimaging along with lesion data strongly support the idea that words semantically related to motor and sensory information activate the respective modality-specialized areas of cortex. The processing of semantic hand-, leg- and mouth-related action information is manifest in the brain response to words such as "grasp", "walk" and "talk", as is the processing of semantically-related sensory auditory, gustatory, olfactory, and visual information in the processing of words like "bell", "salt", "cinnamon", "triangle" and "grass" (Fig. 7; Barrós-Loscertales et al., 2012; González et al., 2006; Kiefer et al., 2012;

Pulvermüller, Kherif, Hauk, Mohr, & Nimmo-Smith, 2009; Simmons et al., 2007). These activations happen as rapidly as the earliest cortical signs of meaning processing (within 100-250 ms, Boulenger, Shtyrov, & Pulvermüller, 2012; Moseley, Pulvermüller, & Shtyrov, 2013; Pulvermüller, Shtyrov, & Hauk, 2009) and functional impairment of the respective areas impairs the processing of relevant words and utterances in a category-specific manner (Boulenger et al., 2008; Kemmerer, Rudrauf, Manzel, & Tranel, 2012; Kiefer & Pulvermüller, 2012; Miceli et al., 2001; Pulvermüller et al., 2010; Trumpp, Kliese, Hoenig, Haarmeier, & Kiefer, 2013). Also autistic subjects, most of whom are characterized by motor problems in early infancy, have recently been found to exhibit a specific deficit in the semantic processing of action-related words, which is reflected in the reduction of their language-related motor activity (Moseley et al., 2013). Even in healthy subjects, the movement of specific body parts in a drumming exercise has a deteriorating effect on memory for concordant action words (hand drumming on hand-related words and leg drumming on leg words, respectively, Shebani & Pulvermüller, 2013). Semantic links to action and perception information seem to have a causal influence on the processing of signs. The brain manifestations of word-meaning connections at the level of the brain are automatic in the sense that they are manifest even when subjects do not attend to incoming words or their meaning (Pulvermüller, Shtyrov, & Ilmoniemi, 2005), but, at the same time, they are under the control of task and context dependent modulation (Pulvermüller, 2013; Willems & Casasanto, 2011).

F. Pulvermüller et al. / Neuropsychologia I (I

Fig. 8. Brain activation (event-related fMRI) elicited by abstract emotion words (in red), which is compared with that to face-related (in green) and arm-related action words (in blue). Note that the inferior motor and premotor cortex also sparked by face and arm words is also activated by abstract emotion words. In addition, there is activation of the anterior insula and a range of other limbic structures (R. Moseley et al., 2012). (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

These results provide strong support for the position that at least some semantic links are realised brain-mechanistically as interlinked APCs reaching into sensory and motor cortices of the human brain. However, it has been argued that certain types of abstract meaning might be unexplainable by such an account. Consider abstract words that are used to speak about feelings and emotions. In some psychological frameworks, such highly abstract items can only become meaningful because the subject relates them to private inner states (Mahon & Caramazza, 2011). However, semantic theorists have argued that an internal definition of a term must fail, as there are no private criteria for accurate word usage so that arbitrary meanings would result (Baker & Hacker, 2009). The solution is to link the meaning of abstract emotion terms to the expression of these "internal states" in overt action. The implication is that the link between abstract emotional meaning and utterances and symbols is by way of the manifestation of emotion in action, that is by way of information in the motor system. The critical prediction from such semantic theorizing is that the motor system will be engaged by abstract emotion words, a prediction which was recently confirmed by data reported by Rachel Moseley (Fig. 8; Moseley, Carota, Hauk, Mohr, & Pulvermüller, 2012). A further critical prediction now is that diseases impairing the expression of emotion in action - as seen for example in autism spectrum condition - should be accompanied by a deficit in the processing of action words and in that of abstract emotion words too (Moseley, Pulvermüller, et al., 2013).

Specific types of abstract meaning may be anchored in the motor system, as illustrated by the case of abstract emotion words. However, other abstract words may not have a clear semantic link to motor or sensory information. Consider the utterance "to free", which may be used to speak about a judge discharging a culprit by a verbal action, a guard opening the door of a prison cell, or a secret agent opening the handcuffs of an ally; or consider the word "beautiful", which may be used to designate a flower, landscape, face or body. In order to learn the meaning of these abstract items, it is essential to know how to relate the terms to real-life events. A symbolic explanation of the meaning of these terms seems insufficient, because, for knowing these words, it is necessary to be able to apply them to concrete actions and objects (Harnad, 1990; Searle, 1980). However, because the variability of motor and sensory schemas to which the terms can relate is broad, not a single type of motor schema or object type links up with such abstract symbols, but rather a disjunct variety of different schemas. (Please note that this situation is very different from concrete words such as "crocodile" or "grasp", where there is some

variability, but still one (or a very small number of) prototypical schema(s) predominate.) Assuming such mechanisms of disjunction are employed in association area "hubs" receiving input from a range of motor and sensory areas, one may suggest that these multimodal areas play a main role in abstract meaning processing. Correlation learning implies that semantic circuits for abstract words with variable usage weaken their motor and sensory links (Pulvermüller, 2013). In this sense, abstract terms with highly variable usage might be semantically based on circuit parts in higher convergence zones, such as prefrontal and anterior-temporal cortex, with multiple weak links to sensory and motor circuits.3

Indeed, abstract words and sentences have been found to activate higher association hubs more strongly than concrete ones; relevant hubs were in prefrontal, anterior-temporal and temporoparietal cortex (Binder, Westbury, McKiernan, Possing, & Medler, 2005; Boulenger, Hauk, & Pulvermüller, 2009; Pulvermüller & Hauk, 2006). Such activation would falsify a sensorimotor account attributing all semantic activity to modality-specific (or -preferential) areas, but it is consistent with correlation-driven decoupling of variable semantics from specific motor and sensory patterns. Admitting the contribution of both hub and sensorimotor cortex to semantics, it becomes possible to study the interplay of the two types of semantics in abstract sentence processing. Interestingly, when processing abstract sentences with a trace of action meaning coloring the abstract content - as for example in the case of idiomatic sentences such as "she grasped the idea" -the putative prefrontal and anterior-temporal areas underpinning abstract semantic processes became active together with action networks in motor systems (Boulenger et al., 2009); these activations even appeared near simultaneously (Boulenger et al., 2012). In the analysis of an abstract sentence, semantic processes driven by the construction may therefore be effective simultaneously with word-driven compositional semantic processes grounding meaning in action (Fig. 9).

In sum, action perception theory offers a neuromechanistic perspective on meaning processing which covers concrete referential and abstract semantics at both the symbol and the sentence levels. It is difficult to see how classic approaches relating meaning to processes in a symbolic system can account for the activation and critical role of sensorimotor as well as different association cortices for semantic processing.

6. Goals and context

The male monkey mirroring the female's motor behavior in courtship would not be a particularly successful representative of his species. As Marc Jeannerod and Pierre Jacob first pointed out (Jacob & Jeannerod, 2005; Jeannerod, 2006), mirror mechanisms are not sufficient in social interaction and language, but a mechanism for learning and knowing complementary actions is required. In the most general case, this means that the individual has to know that an action of type A is typically followed by an action of type B, where these actions can be carried out by the same or different individuals. Linking an action to a far-reaching goal can be understood in terms of action sequences, as it is frequently the second action in the sequence, or the end-result of that other action, that forms the goal of an antecedent action, as for example when ordering a coffee in a coffee shop (see also Glenberg & Gallese, 2012).

3 Still, even the most abstract and semantically variable concept may require concrete information. The PROOF concept can hardly be understood without knowing at least example instantiations of the corresponding highly complex and variable action schema (see also Barsalou & Wiemer-Hastings, 2005).

10 11 12

20 21 22

60 61 62

F. Pulvermüller et al. / Neuropsychologia ■ (■■■■) ■■■-■■■

activation to idiomatic and literal sentences

example sen fences;

arm literal: Anna caught the fish._

leg literal: John climbs on his high chair ...

arm Idiomatic: Anita caught the sun ...

leg idiomatic: John climbs on his high horse...

idiomatic sentences literal sentences

£ "Î * ^ l л

fTj У¿jAЖ. ^Kme - - С БЕ

îsc-Î

Г «~ ШГI I / _

differential motor system activation by idiomatic and literal arm and leg sentences

arm sentences leg sentences

1,1 ,rr' j^fS&fiti

гаот» ч

... '4

Fig. 9. Brain activation induced by concrete literal and abstract idiomatic sentences including words related to arm and leg actions. Example stimuli are presented at the top left, fMRl results on the left side and MEG activation dynamics on the right. Top panels: abstract sentences activated temporal and prefrontal cortex more strongly than concrete ones. Green areas and yellow ovals indicate significant activation enhancement for idiomatic strings. Bottom panels: abstract and concrete sentences activated hand and leg areas relatively more strongly if they include a concordant action word. Note that this activation was seen at the point in time when sentence meaning could be understood and substantially after the action words had been perceived. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article). Adopted from Boulenger et al. (2009, 2012).

In spite of breakthroughs in understanding the brain basis of action-goal relationships (Cattaneo, Caruana, Jezzini, & Rizzolatti, 2009; Fogassi et al., 2005; Rizzolatti & Sinigaglia, 2010), sequence storage has not been a prime target of the neurophysiological literature attending specifically to the interaction of cortical area functions. Some critics therefore tried to construct an argument that the lack of coverage of action sequences and action goals might be a lethal feature of action perception theories of cognition and language (Hickok, 2009; Lotto et al., 2009). However, as shown below, this claim is not warranted.

In addition to a mechanism for mapping action perceptions on their concordant motor schemas, a mechanism for linking actions into sequences of action schemas is necessary. The neural rule of

correlation learning provides a mechanism not only for the binding of co-occurring actions and perceptions, but equally for the binding of sequential events into action chains. This is particular obvious if neurobiological learning rules are applied that are sensitive to firing sequence and precise timing. Crucially, it is apparent from neuro-physiological research that neuronal plasticity depends on the precise timing of spike sequences, and recent neurocomputational modeling incorporates such spike-timing dependent plasticity (Bi & Wang, 2002; Caporale & Dan, 2008; Deco, Buehlmann, Masquelier, & Hugues, 2011). However, there is a problem if two actions or events are far separated from each other in time (e.g., by seconds), and their neural correlates may therefore not be active at the same time or in close temporal vicinity. This situation would make it impossible for

F. Pulvermüller et al. / Neuropsychologia I (I

Fig. 10. Embedding of APCs into sequences is possible by way of direct links between APCs (red lines) and by indirect links to intermediate combinatorial representations (not shown). Such combinatorial connections can realize syntactic-constructional links of words into sentences and, critically, pragmatic social-communicative knowledge linking speech acts into schemas for interactive communication. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

the correlation learning mechanism to become effective, because Hebb-type learning generally requires simultaneous or near-simultaneous activation of neuronal sets. In this context, it is important to see that APCs are strongly connected neuronal ensembles and that one of the fundamental functional implications of their strong internal connectivity is that they retain or "hold" activity for some time (Braitenberg, 1978; Wennekers et al., 2006). Therefore, APC and the action memories they carry may also be active after stimulation or ignite spontaneously so that the phenomenon of working memory can be accounted for in neuromechanistic terms (Verduzco-Flores, Bodner, Ermentrout, Fuster, & Zhou, 2009; Zipser, Kehoe, Littlewort, & Fuster, 1993) and the spontaneous emergence of a decision to speak or act can be modeled (Garagnani & Pulvermuller, 2013). In the present context, it is important to note that even temporally dispersed events can be associated by correlation learning if at least one of them is being processed by an APC with strong internal connections exhibiting sustained working memory activity. The sustained "memory" activity bridges the time between APC ignitions dispersed in time and results in simultaneous activation of APCs.

Sequence learning allows one to store the knowledge that, in courtship, a given action has to follow a specific antecedent, and that, within a sentence, given types of words have to follow other types of words. So, obviously, action perception theory offers mechanisms for storing sequences and combinatorial information based on the connections between APCs (Fig. 10). There is ample evidence that simple chaining mechanisms are effective at the cortical level (Newman-Norlund, van Schie, van Zuijlen, & Bekkering, 2007) and that especially inferior-frontal cortex is relevant for such sequence knowledge (Fazio et al., 2009). Syntactic knowledge links together abstract categories of words which can be characterized by combinatorial features; both inferior-frontal and superior-temporal cortex seem of importance here (Friederici, 2011; Pulvermüller & Fadiga, 2010). A key question is how an APC account in terms of distributed neuronal circuits can account for the riddles immanent to human capacity to generate novel sentences from a limited vocabulary and previously encountered set of word strings. Neurocomputational studies show that generalization of syntactic regularities may result from the mapping of word strings and the frequent substitutions of similar words in similar contexts, which leads to the formation of a special type of APC processing abstract combinatorial schemas, such as "noun followed by verb" (Knoblauch & Pulvermüller, 2005; Pulvermüller, 2010; Pulvermüller & Knoblauch, 2009). In simulation studies with neuronal networks incorporating auto-and hetero-associative properties of cortical connectivity, we recently mapped information about noun-verb sequences obtained from large text corpora and found that such combinatorial mapping grouped and linked words according to both lexical-syntactic and semantic criteria. Critically, the combinatorial information stored by the networks was generalized over items with similar combinatorial properties so that new syntactically correct and semantically plausible word sequences were generated (see Pulvermüller & Knoblauch, 2009).

These considerations show that, at the linguistic levels of syntax and semantics, sequential combination of words into phrases and sentences is compatible with action perception theory. Over and above such compatibility demonstration, the neuromechanistic approach to combination may provide a key for understanding the formation of syntactic-semantic rules and construction schemas underlying the meaningful sentences (for further discussion, see Pulvermüller, 2002; Pulvermüller, Cappelle, & Shtyrov, 2013). However, there is still a gap between such syntactic-semantic knowledge and the use of symbols in social interaction.

7. Social-communicative actions and their sequence structure

Brain research on language has so far mostly dealt with linguistic structures, that is words and sentences, without considering language use and communication as a social form of interaction. The neuromechanistic basis of such linguistic pragmatic knowledge is still poorly understood. What are the brain circuits of the communicative actions or speech acts for which linguistic forms are used as tools (see Austin, 1962; Searle, 1969; Wittgenstein, 1953)? A word such as "apple" can be used as a tool to achieve different goals: for example, the goal to NAME an object in the context of a language exercise, or, alternatively, the goal to REQUEST and obtain an apple from a shopkeeper or host.4 It is obvious that in these contexts, the same word and utterance is a tool to reach different goals and that the word is therefore associated with context-specific sets of assumptions and intentions (see, for example, Alston, 1964; Ehlich, 2007; Fritz & Hundsnurscher, 1994; Stalnaker, 2002): In the NAMING context with the expectation that the label might be approved as a tool to refer to the object, but in the REQUEST context with a range of additional social-interactive expectations, that the other party will hand over the object, that such an object is available to the other party and that he or she is in principle willing to pass it, that the object can be identified unambiguously and so on. In close relationship to these different expectations, assumptions, intentions and goals, the different speech acts, in the present example NAMING and REQUEST, are linked into different typical action sequences or dialog schemas (see Fig. 11). Note that the different contexts can be captured by two different types of action sequence schemas. Description of such action structure is a main task in pragmatic-linguistic research (Alston, 1964; Ehlich, 2007; Fritz & Hundsnurscher, 1994; Stalnaker, 2002).

The proposal here is that over and above the level of APCs for linguistic forms (words, utterances) and their referential semantic meaning (the objects, actions, features etc. they can be used to

4 Note that it is not necessary that the different pragmatic roles the utterance takes on in the different contexts is also manifest in different articulatory- or acoustic-phonetic features (cf., Borg, 2012). Exactly the same utterance can serve the different speech act functions.

F. Pulvermüller et al. / Neuropsychologia I (l

10 11 12

20 21 22

60 61 62

Fig. 11. Illustration of the action sequence structures for the speech acts of NAMING and REQUESTING and brain areas most relevant for in-depth processing of the semantic link between word and object (in the case of NAMING, in cyan) and for that of the pragmatic links between the utterance and its typical follower actions (in the case of REQUEST, in magenta). Note that, upon NAMING by a communication partner A, the other partner, B, is expected to correct or clarify, whereas REQUESTING opens the additional options of following the request and rejecting or denying it (see Egorova et al., 2013).

speak about; Fig. 6), there are sequentially-linked action schemas for interaction types (interlinked APCs as in Fig. 10). The nature of the sequences is not a simple line-up but rather a tree structure typical for interaction schemas in which different communication partners can participate (as sketched in Fig. 11). The sequence schemas are abstract in the sense that a range of different word forms and utterances can be used to perform each of the individual actions specified at each node of the tree structure. If two partners, A and B, interact according to the same interaction schema, they both chose actions from the same complex action tree schema and understand their partner's action against the background of this schema. Therefore one may say, in a sense, that communication partners "predict" each other's actions (see Pickering & Garrod, 2013). However, in a strict sense, it is incorrect to speak of "prediction" here, as typically a broad range of different utterances and several action schemas are possible and common at each stage of a conversation. For example, following upon the waiter's offer "what can I get you?", a range of utterances can be used to make specific requests and the offer can also be rejected. It seems more appropriate to speak of pragmatic priming of a range of action schemas and utterance forms, which are "opened up" by each move of the communication game.5 We submit that, when interacting according to one dialog schema (for example REQUESTING), the communication partners have both the same sequence schema representation active in their brains.

The brain mechanisms for the postulated action sequence schemas can be assumed to lie in sensorimotor systems and, when the words (or even larger constructions) appear in the different contexts, their APCs will ignite while the respective sequence schema is active as well. In the NAMING context, the

5 Pickering and Garrod put much emphasis on communication in which the listener can exactly predict the wording of the speaker (Pickering & Garrod, 2013). Such "high entropy" situations appear rare and should not obscure the view on more typical low entropy unpredictable dialogs, where a range of action schemas are expectable but individual utterance forms are "predictable" only with low probability.

APC of the word "apple" is activated, including its word form part and its object-related referential semantic part. The referential word-object link is known to draw upon inferior-temporal ventralvisual stream circuits (Hickok & Poeppel, 2007; Pulvermüller, 1999). In contrast, when using the same word to REQUEST an object, the emphasis is on the link of the utterance with expected actions by the partner. Therefore, motor and action sequence circuits in frontocentral cortex need to be sparked in addition to the word-related circuit (Fazio et al., 2009; Pulvermüller, 2005). By assumption, this results in a relative emphasis of motor systems activation in the REQUEST context.

The proposed linguistic pragmatic extension of action perception theory makes novel predictions on the neurobiological mechanisms of language. One and the same utterance should, depending on the social-interactive context in which it is embedded, dynamically link with different APCs for action sequence schemas or object circuits, primarily drawing on fronto-central or temporal cortex respectively. To use the above example for illustration once again: the word form "apple", which is semantically linked with object knowledge, would strongly activate this referential knowledge in the NAMING context. The naming context leads to an emphasis of object knowledge and attention enhancement of object features by way of reduced negative feedback regulation in inferior-temporal cortex (attention-to-object). If the same word is used in the REQUEST context, the emphasis of action knowledge and up-regulation of motor systems activation is predicted for processing the pragmatic representation of the sequence structure characterizing the social interaction context of REQUESTS. In addition, the word-related APC is, in this case, linked into a sequence schema of social-communicative actions, as the word uttered for REQUESTING gives rise to the expectation of the requested object being handed over, or the addressee replying with an excuse etc. So, in essence, the same meaningful utterance is predicted to respectively activate inferior-temporal or precentral/prefronal areas relatively more strongly when used as a tool for different social-communicative actions.

Natalia Egorova recently carried out experiments to elucidate the time course of cortical area activation in speech act processing. The same words were used for different speech acts and experiment participants had to passively watch and understand these social-communicative actions performed by interacting individuals. Already ~ 100 ms after the word critical for understanding the speech acts could be recognized, brain activation distinguished between NAMING and REQUEST actions. There was stronger activation to REQUESTS and a relevant part of the additional cortical sources were in frontocentral motor systems (Fig. 12; Egorova, Pulvermüller, & Shtyrov, in press; Egorova, Shtyrov, & Pulvermüller, 2013).

These results on the brain basis of speech act processing suggest a local cortical difference which, in part, confirms the above predictions, although results need to be confirmed with neurome-tabolic imaging methods and no strong support is so far available for a relative enhancement of inferior-temporal activation in the NAMING context. Nevertheless, these results represent a first step towards understanding the brain basis of "How to do things with words" in social communication contexts (Austin, 1962), which may be extended in future, for example in the investigation of populations with deficits in social-communicative interaction with putative relationship to mirror neuron circuits (e.g., autism, Rizzolatti, Fabbri-Destro, & Cattaneo, 2009). This topic, so far largely neglected in the neurobiology of language, is at the heart of language as a social phenomenon, as an interactive game activity characterized by action sequences, goals and intentions, commitments about the "theory-of-mind" assumptions of communication partners, and the use of general and specific knowledge related to linguistic forms. Future research into this area may propel the neuroscience of

F. Pulvermüller et al. / Neuropsychologia I (I

Fig. 12. Brain correlates of social-interactive communication. Results of an EEG experiment on understanding of NAMING and REQUEST actions performed by use of the same words. Compared with NAMING (in blue), stronger brain activation was seen in the REQUEST context (in red) starting already at 100-150 ms post critical (written) word onset. Plotted is global field power, GFP, calculated from multiple electrodes. Sources of this enhanced activity to REQUESTs (L1 minimum norm current estimates, see inset) were primarily localized in left frontocentral and right parietotemporal cortex (adopted from Egorova et al., 2013). These additional activations may reflect the processing of knowledge about interaction sequences, intentions and action goals relevant to understanding REQUESTs. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article).

language into a social brain science for which the term neuroprag-matics had once been proposed (Bara, Tirassa, & Zettin, 1997). Action perception theory is critical in this new endeavor of spelling out the neuronal circuits of social-communicative interaction.

8. Conclusions

The brain basis of cognition can be understood in terms of interlinked action perception representations. Mirror neurons and other sensorimotor neurons document the linking-together of motor and sensory information into coherent distributed sensor-imotor cortical circuits, which can provide mechanisms for repetition and simulation. Over and above such mirroring, a model of action perception circuits incorporates attention mechanisms as regulating cortical activity and resultant degree of area-specific competition between APCs. Action perception circuits are themselves mechanisms for maintaining neuronal activity as the strongly connected neuronal assemblies allow activity to reverberate for some time, thus providing a mechanism for working memory. Critically, the link between APCs and merging of circuits into higher-order ones is available as a mechanism of attaching meaning to motor acts and to object representations. A novel perspective is offered by the linking of APCs for linguistic forms with action sequence representations. This linkage is required to bind words into constructions and, critically, for embedding linguistic signs into their specific social-interactive contexts where they bear their role as tools for communication. The brain basis of this latter flexible binding between linguistic form and social function is addressed by recent developments in the field of neuropragmatics.

In sum, the outlined perspectives of mirror neurons, their explanation in terms of action perception circuits, APCs, and the broader strategy to build cognition and communicative action sequence structures from networks of APCs offer a fruitful perspective on the neuronal mechanisms of attention, language, meaning and social communicative interaction and can address the issues addressed in Marc Jeannerod's important comment cited above. The previous strategy to postulate separate modules for perception, motor movement, attention, memory, concepts and theory of mind falls short of providing concrete mechanisms for how the brain relates a word to a referent object or how it links the same word into a sequence of socially relevant actions characterized by specific goals and action sequences. Action perception theory fills this gap by providing not only strong experimental data on action and perception, cognition and communication (Pulvermüller & Fadiga, 2010), and informed guesses on the neuronal circuits underlying not

hitherto investigated cognitive domains (Kiefer & Pulvermüller, 2012), but, in addition, concrete brain-embodied simulation studies spelling out cognitive processing in computational models compatible with the structure and function of human cortex (Wennekers & Palm, 2007; Wermter et al., 2009). These novel neurotheoretical, -experimental and -computational achievements demonstrate the importance of mechanisms of motor cognition and motor meaning (Jeannerod, 2006).

Acknowledgments

We thank Sabina Mollenhauer, Anne Reboul and two anonymous Q4 referees for their help at different stages of this work. Supported by the Freie Universität Berlin, the Deutsche Forschungsgemeinschaft (Excellence Cluster Languages of Emotion), the Medical Research Council (UK) (MC_US_A060_0034, U1055.04.003.00001.01) and the Engineering and Physical Sciences Research Council (UK) (BABEL grant).

References

Abeles, M. (1991). Corticonics - neural circuits of the cerebral cortex. Cambridge: Cambridge University Press.

Alston, W. P. (1964). Philosophy of language. Englewood Cliffs, NJ: Prentice-Hall.

Austin, J. L (1962). How to do things with words. Oxford: Claredon Press.

Baker, E., Blumstein, S. E., & Goodglass, H. (1981). Interaction between phonological and semantic factors in auditory comprehension. Neuropsychologia, 19, 1-15.

Baker, G. P., & Hacker, P. M. S. (2009). Wittgenstein: Understanding and meaning, part 1 - Essays, Vol. 1. Oxford, Chichester: Wiley-Blackwell.

Bangert, M., Peschel, T., Schlaug, G., Rotte, M., Drescher, D., Hinrichs, H., Heinze, H. J., & Altenmüller, E. (2006). Shared networks for auditory and motor processing in professional pianists: Evidence from fMRI conjunction. Neuroimage, 30, 917-926.

Bara, B. G., Tirassa, M., & Zettin, M. (1997). Neuropragmatics: Neuropsychological constraints on formal theories of dialog. Brain Language, 59, 7-49.

Barrós-Loscertales, A., González, J., Pulvermüller, F., Ventura-Campos1, N., Busta-mante, J. C., Costumero, V., Parcet, M. A., & Ávila, C. (2012). Reading salt activates gustatory brain regions: fMRI evidence for semantic grounding in a novel sensory modality. Cerebral Cortex, 22, 2554-2563.

Barsalou, L W., & Wiemer-Hastings, K. (2005). Situating abstract concepts. In: D. Pecher, & R. Zwaan (Eds.), Grounding cognition: The role of perception and action in memory, language, and thought (pp. 129-163). New York: Cambridge University Press.

Bi, G. Q., & Wang, H. X. (2002). Temporal asymmetry in spike timing-dependent synaptic plasticity. Physiology & Behavior, 77, 551-555.

Binder, J. R., Westbury, C. F., McKiernan, K. A., Possing, E. T., & Medler, D. A. (2005). Distinct brain systems for processing concrete and abstract concepts. Journal of Cognitive Neuroscience, 17, 905-917.

Bishop, D. V. (2013). Cerebral asymmetry and language development: Cause, Q5 correlate, or consequence? Science, 340, 1230531.

Borg, E. (2012). More questions for mirror neurons. Consciousness and Cognition

F. Pulvermüller et al. I Neuropsychologia l (l

6 l В 9

i0 ii i2

20 2i 22

26 2l 2B

36 3l 3B

46 4l 4B

56 Sl SB

60 61 б2

6S бб

Boulenger, V., Hauk, O., & Pulvermüller, F. (2009). Grasping ideas with the motor system: Semantic somatotopy in idiom comprehension. Cerebral Cortex, 19, 1905-1914.

Boulenger, V., Mechtouff, L., Thobois, S., Broussolle, E., Jeannerod, M., & Nazir, T. A. (2008). Word processing in Parkinson's disease is impaired for action verbs but not for concrete nouns. Neuropsychologia, 46, 743-756.

Boulenger, V., Shtyrov, Y., & Pulvermüller, F. (2012). When do you grasp the idea? MEG evidence for instantaneous idiom understanding. Neuroimage, 59, 3502-3513.

Boulenger, V., Silber, B. Y., Roy, A. C., Paulignan, Y., Jeannerod, M., & Nazir, T. A.

(2008). Subliminal display of action words interferes with motor planning: A combined EEG and kinematic study. Journal of Physiology (Paris), 102, 130-136.

Braitenberg, V. (1978). Cell assemblies in the cerebral cortex. In: R. Heim, & G. Palm (Eds.), Theoretical approaches to complex systems (pp. 171-188). Berlin: Springer ((Lecture notes in biomathematics, vol. 21)).

Braitenberg, V. (1980). Alcune considerazione sui meccanismi cerebrali del lin-guaggio. In: G. Braga, V. Braitenberg, C. Cipolli, E. Coseriu, S. Crespi-Reghizzi, J. Mehler, & R. Titone (Eds.), L'accostamento interdisciplinare allo studio del linguaggio (pp. 96-108). Milano: Franco Angeli Editore.

Braitenberg, V., & Pulvermüller, F. (1992). Entwurf einer neurologischen Theorie der Sprache. Naturwissenschaften, 79, 103-117.

Braitenberg, V., & Schüz, A. (1992). Basic features of cortical connectivity and some considerations on language. In: J. Wind, B. Chiarelli, B. H. Bichakjian, A. Nocentini, & A. Jonker (Eds.), Language origin: A multidisciplinary approach (pp. 89-102). Dordrecht: Kluwer.

Braitenberg, V., & Schüz, A. (1998). Cortex: Statistics and geometry of neuronal connectivity ((2 ed.). Berlin: Springer.

Bruce, C. J., & Goldberg, M. E. (1985). Primate frontal eye fields. I. Single neurons discharging before saccades. Journal of Neurophysiology, 53, 603-635.

Caporale, N., & Dan, Y. (2008). Spike timing-dependent plasticity: A Hebbian learning rule. Annual Review of Neuroscience

Carota, F., Moseley, R., & Pulvermüller, F. (2012). Body-part-specific representations of semantic noun categories. Journal of Cognitive Neuroscience, 24, 1492-1509.

Catani, M., Jones, D. K., & Ffytche, D. H. (2005). Perisylvian language networks of the human brain. Annals of Neurology, 57, 8-16.

Cattaneo, L., Caruana, F., Jezzini, A., & Rizzolatti, G. (2009). Representation of goal and movements without overt motor behavior in the human motor cortex: a transcranial magnetic stimulation study. Journal of Neuroscience, 29, 11134-11138.

D'Ausilio, A., Pulvermüller, F., Salmas, P., Bufalari, I., Begliomini, C., & Fadiga, L.

(2009). The motor somatotopy of speech perception. Current Biology, 19, 381-385.

Damasio, A. R. (1989). Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition, 33, 25-62.

de Boysson-Bardies, B., & Vihman, M. M. (1991). Adaptation to language: Evidence from babbling and first words in four languages. Language, 67, 297-319.

Deco, G., Buehlmann, A., Masquelier, T., & Hugues, E. (2011). The role of rhythmic neural synchronization in rest and task conditions. Frontiers in Human Neuroscience, 5, 4.

Del Giudice, M., Manera, V., & Keysers, C. (2009). Programmed to learn? The ontogeny of mirror neurons. Developmental Science, 12, 350-363.

Duncan, J. (2006). EPS Mid-Career Award 2004: Brain mechanisms of attention. Quarterly Journal of Experimental Psychology (Colchester), 59, 2-27.

Egorova, N., Pulvermüller, F., & Shtyrov, Y. (2013). Neural dynamics of speech act comprehension: an MEG study of Naming and Requesting. Brain Topography (in press)

Egorova, N., Shtyrov, Y., & Pulvermüller, F. (2013). Early and parallel processing of pragmatic and semantic information in speech acts: neurophysiological evidence. Frontiers in Human Neuroscience, 7, 1-13.

Ehlich, K. (2007). Sprache und sprachliches Handeln. Berlin: Walter de Gruyter.

Elbert, T., & Rockstroh, B. (1987). Threshold regulation - A key to the understanding of the combined dynamics of EEG and event related potentials. Journal of Psychophysiology, 4, 317-333.

Fadiga, L., Craighero, L., Buccino, G., & Rizzolatti, G. (2002). Speech listening specifically modulates the excitability of tongue muscles: a TMS study. European Journal of Neuroscience, 15, 399-402.

Fazio, P., Cantagallo, A., Craighero, L., D'Ausilio, A., Roy, A., Pozzo, T., Calzolari, F., Granieri, E., & Fadiga, L. (2009). Encoding of human action in Broca's area. Brain, 132, 1980-1988.

Fodor, J. A. (1983). The modulatity of mind. Cambridge, MA: MIT Press.

Fogassi, L., Ferrari, P. F., Gesierich, B., Rozzi, S., Chersi, F., & Rizzolatti, G. (2005). Parietal lobe: from action organization to intention understanding. Science, 308, 662-667.

Friederici, A. D. (2011). The brain basis of language processing: From structure to function. Physiological Reviews, 91, 1357-1392.

Fritz, G., & Hundsnurscher, F. (1994). Handbuch der Dialoganalyse. Tübingen: Niemeyer Verlag.

Fry, D. B. (1966). The development of the phonological system in the normal and deaf child. In: F. Smith, & G. A. Miller (Eds.), The genesis of language (pp. 187206). Cambridge, MA: MIT Press.

Fuster, J. M. (2003). Cortex and mind: Unifying cognition. Oxford: Oxford University Press.

Fuster, J. M. (2009). Cortex and memory: Emergence of a new paradigm. Journal of Cognitive Neuroscience, 21, 2047-2072.

Galantucci, B., Fowler, C. A., & Turvey, M. T. (2006). The motor theory of speech perception reviewed. Psychonomic Bulletin and Review, 13, 361-377.

Gallese, V., Keysers, C., & Rizzolatti, G. (2004). A unifying view of the basis of social cognition. Trends in Cognitive Sciences, 8, 396-403.

Gallese, V., & Lakoff, G. (2005). The brain's concepts: The role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology, 22, 455-479.

Garagnani, M., & Pulvermuller, F. (2013). Neuronal correlates of decisions to speak and act: Spontaneous emergence and dynamic topographies in a computational model of frontal and temporal areas. Brain Language, 127, 75-85.

Garagnani, M., & Pulvermüller, F. (2011). From sounds to words: A neurocomputa-tional model of adaptation, inhibition and memory processes in auditory change detection. Neuroimage, 54, 170-181.

Garagnani, M., Shtyrov, Y., & Pulvermüller, F. (2009). Effects of attention on what is Q6 known and what is not: MEG evidence for functionally discrete memory circuits. Frontiers in Human Neuroscience, 3, http://dx.doi.org/10.3389/ neuro.3309.3010.2009.

Garagnani, M., Wennekers, T., & Pulvermüller, F. (2007). A neuronal model of the language cortex. Neurocomputing, 70, 1914-1919.

Garagnani, M., Wennekers, T., & Pulvermüller, F. (2008). A neuroanatomically-grounded Hebbian learning model of attention-language interactions in the human brain. European Journal of Neuroscience, 27, 492-513.

Glenberg, A. M., & Gallese, V. (2012). Action-based language: A theory of language acquisition, comprehension, and production. Cortex, 48, 905-922.

González, J., Barros-Loscertales, A., Pulvermüller, F., Meseguer, V., Sanjuán, A., Belloch, V., & Ávila, C. (2006). Reading cinnamon activates olfactory brain regions. Neuroimage, 32, 906-912.

Hanuschkin, A., Ganguli, S., & Hahnloser, R. H. (2013). A Hebbian learning rule gives rise to mirror neurons and links them to control theoretic inverse models. Front Neural Circuits, 7, 106.

Harnad, S. (1990). The symbol grounding problem. Physica D, 42, 335-346.

Hauk, O., Shtyrov, Y., & Pulvermüller, F. (2006). The sound of actions as reflected by mismatch negativity: Rapid activation of cortical sensory-motor networks by sounds associated with finger and tongue movements. European Journal of Neuroscience, 23, 811-821.

Heyes, C. (2010). Where do mirror neurons come from? Neuroscience & Biobeha-vioral Reviews, 34, 575-583.

Hickok, G. (2009). Eight problems for the mirror neuron theory of action understanding in monkeys and humans. Journal of Cognitive Neuroscience, 21, 1229-1243.

Hickok, G., & Hauser, M. (2010). (Mis)understanding mirror neurons. Current Biology, 20, R593-594.

Hickok, G., & Poeppel, D. (2007). The cortical organization of speech processing. Nature Reviews Neuroscience, 8, 393-402.

Hubel, D. (1995). Eye, brain, and vision (2 ed.). New York: Scientific American Library.

Jacob, P., & Jeannerod, M. (2005). The motor theory of social cognition: A critique. Trends in Cognitive Sciences, 9, 21-25.

Jeannerod, M. (1994). The representing brain: Neuronal correlates of motor intention and imagery. Behavioral and Brain Science, 17, 187-202.

Jeannerod, M. (2001). Neural simulation of action: A unifying mechanism for motor cognition. Neuroimage, 14, S103-109.

Jeannerod, M. (2006). Motor cognition: What actions tell to the self. Oxford: Oxford University Press.

Jeannerod, M., Arbib, M. A., Rizzolatti, G., & Sakata, H. (1995). Grasping objects: The cortical mechanisms of visuomotor transformation. Trends in Neuroscience, 18, 314-320.

Jezzini, A., Caruana, F., Stoianov, I., Gallese, V., & Rizzolatti, G. (2012). Functional organization of the insula and inner perisylvian regions. Proceedings of the National Academy of Sciences of USA, 109, 10077-10082.

Kemmerer, D., Rudrauf, D., Manzel, K., & Tranel, D. (2012). Behavioural patterns and lesion sites associated with impaired processing of lexical and conceptual knowledge of action. Cortex, 48, 826-848.

Keysers, C., & Perrett, D. I. (2004). Demystifying social cognition: A Hebbian perspective. Trends in Cognitive Sciences, 8, 501-507.

Kiefer, M., & Pulvermüller, F. (2012). Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions. Cortex, 48, 805-825.

Kiefer, M., Trumpp, N., Herrnberger, B., Sim, E. J., Hoenig, K., & Pulvermüller, F. (2012). Dissociating the representation of action- and sound-related concepts in middle temporal cortex. Brain Language, 122, 120-125.

Knoblauch, A., & Palm, G. (2001). Pattern separation and synchronization in spiking associative memories and visual areas. Neural Networks, 14, 763-780.

Knoblauch, A., & Pulvermüller, F. (2005). Sequence detector networks and associative learning of grammatical categories. In: S. Wermter, G. Palm, & M. Elshaw (Eds.), Biomimetic neural learning for intelligent robots (pp. 31 -53). Berlin: Springer.

Kohler, E., Keysers, C., Umilta, M. A., Fogassi, L., Gallese, V., & Rizzolatti, G. (2002). Hearing sounds, understanding actions: action representation in mirror neurons. Science, 297, 846-848.

Lepage, J. F., & Theoret, H. (2007). The mirror neuron system: grasping others' actions from birth? Developmental Science, 10, 513-523.

Liberman, A. M., Cooper, F. S., Shankweiler, D. P., & Studdert-Kennedy, M. (1967). Perception of the speech code. Psychology Review, 74, 431-461.

Locke, J. L. (1993). The child's path to spoken language. Cambridge, MA: Harvard University Press.

Lotto, A. J., Hickok, G. S., & Holt, L. L. (2009). Reflections on mirror neurons and speech perception. Trends in Cognitive Sciences, 13, 110-114.

MacNeilage, P. F., & Davis, B. L. (2000). On the origin of internal structure of word forms. Science, 288, 527-531.

F. Pulvermüller et al. / Neuropsychologia I (I

Mahon, B. Z., & Caramazza, A. (2011). What drives the organization of object knowledge in the brain? Trends in Cognitive Sciences, 15, 97-103.

Miceli, G., Fouch, E., Capasso, R., Shelton, J. R., Tomaiuolo, F., & Caramazza, A. (2001). The dissociation of color from form and function knowledge. Nature Neuroscience, 4, 662-667.

Moineau, S., Dronkers, N. F., & Bates, E. (2005). Exploring the processing continuum of single-word comprehension in aphasia. Journal of Speech, Language, and Hearing Research, 48, 884-896.

Moseley, R., Carota, F., Hauk, O., Mohr, B., & Pulvermüller, F. (2012). A role for the motor system in binding abstract emotional meaning. Cerebral Cortex, 22, 1634-1647.

Moseley, R. L., Mohr, B., Lombardo, M. V., Baron-Cohen, S., Hauk, O., & Pulvermüller, F. (2013). Brain and behavioural correlates of action semantic deficits in autism. Frontiers in Human Neuroscience, 7.

Moseley, R. L., Pulvermüller, F., & Shtyrov, Y. (2013). Sensorimotor semantics on the spot: Brain activity dissociates between conceptual categories within 150 ms. Scientific Reports, 3, 1928.

Newman-Norlund, R. D., van Schie, H. T., van Zuijlen, A. M., & Bekkering, H. (2007). The mirror neuron system is more active during complementary compared with imitative action. Nature Neuroscience, 10, 817-818.

Nucifora, P. G., Verma, R., Melhem, E. R., Gur, R. E., & Gur, R. C. (2005). Leftward asymmetry in relative fiber density of the arcuate fasciculus. Neuroreport, 16, 791-794.

Palm, G. (1990). Cell assemblies as a guideline for brain research. Concepts in Neuroscience, 1, 133-147.

Petitto, L. A., & Marentette, P. F. (1991). Babbling in manual mode: evidence for the ontogeny of language. Science, 251, 1493-1496.

Pickering, M. J., & Garrod, S. (2004). Toward a mechanistic psychology of dialog. Behavioral and Brain Science, 27, 169-190 (discussion 190-226).

Pickering, M. J., & Garrod, S. (2013). An integrated theory of language production and comprehension. Behavioral and Brain Science, 36, 329-347.

Pulvermüller, F. (1996). Hebb's concept of cell assemblies and the psychophysiology of word processing. Psychophysiology, 33, 317-333.

Pulvermüller, F. (1999). Words in the brain's language. Behavioral and Brain Sciences, 22, 253-336.

Pulvermüller, F. (2002). The neuroscience of language. Cambridge: Cambridge University Press.

Pulvermüller, F. (2005). Brain mechanisms linking language and action. Nature Reviews Neuroscience, 6, 576-582.

Pulvermüller, F. (2010). Brain embodiment of syntax and grammar: Discrete combinatorial mechanisms spelt out in neuronal circuits. Brain Lang, 112, 167-179.

Pulvermüller, F. (2013). How neurons make meaning: Brain mechanisms for embodied and abstract-symbolic semantics. Trends in Cognitive Sciences, 17, 458-470.

Pulvermüller, F., Cappelle, B., & Shtyrov, Y. (2013). Brain basis of meaning, words, constructions, and grammar. In: T. Hoffmann, & G. Trousdale (Eds.), Oxford handbook of construction grammar (pp. 397-416). Oxford: Oxford University Press.

Pulvermüller, F., Cooper-Pye, E., Dine, C., Hauk, O., Nestor, P. J., & Patterson, K. (2010). The word processing deficit in Semantic Dementia: All categories are equal but some categories are more equal than others. Journal of Cognitive Neuroscience, 22, 2027-2041.

Pulvermüller, F., & Fadiga, L. (2010). Active perception: Sensorimotor circuits as a cortical basis for language. Nature Reviews Neuroscience, 11, 351-360.

Pulvermüller, F., & Hauk, O. (2006). Category-specific processing of color and form words in left fronto-temporal cortex. Cerebral Cortex, 16, 1193-1201.

Pulvermüller, F., Huss, M., Kherif, F., Moscoso del Prado Martin, F., Hauk, O., & Shtyrov, Y. (2006). Motor cortex maps articulatory features of speech sounds. Proceedings of the National Academy of Science of USA, 103, 7865-7870.

Pulvermüller, F., Kherif, F., Hauk, O., Mohr, B., & Nimmo-Smith, I. (2009). Cortical cell assemblies for general lexical and category-specific semantic processing as revealed by fMRI cluster analysis. Human Brain Mapping, 30, 3837-3850.

Pulvermüller, F., Kiff, J., & Shtyrov, Y. (2012). Can language-action links explain language laterality? An ERP study of perceptual and articulatory learning of novel pseudowords. Cortex, 48, 471-481.

Pulvermüller, F., & Knoblauch, A. (2009). Discrete combinatorial circuits emerging in neural networks: A mechanism for rules of grammar in the human brain? Neural Networks, 22, 161-172.

Pulvermüller, F., Shtyrov, Y., & Hauk, O. (2009). Understanding in an instant: neurophysiological evidence for mechanistic language circuits in the brain. Brain Language, 110, 81-94.

Pulvermüller, F., Shtyrov, Y., & Ilmoniemi, R. J. (2003). Spatio-temporal patterns of neural language processing: An MEG study using Minimum-Norm Current Estimates. Neuroimage, 20, 1020-1025.

Pulvermüller, F., Shtyrov, Y., & Ilmoniemi, R. J. (2005). Brain signatures of meaning access in action word recognition. Journal of Cognitive Neuroscience, 17, 884-892.

Rilling, J. K., Glasser, M. F., Preuss, T. M., Ma, X., Zhao, T., Hu, X., & Behrens, T. E. (2008). The evolution of the arcuate fasciculus revealed with comparative DTI. Nature Neuroscience, 11, 426-428.

Rizzolatti, G., Fabbri-Destro, M., & Cattaneo, L. (2009). Mirror neurons and their clinical relevance. Nature Clinical Practice Neurology, 5, 24-34.

Rizzolatti, G., & Sinigaglia, C. (2010). The functional role of the parieto-frontal mirror circuit: Interpretations and misinterpretations. Nature Reviews Neuroscience, 11, 264-274.

Rogalsky, C., Love, T., Driscoll, D., Anderson, S. W., & Hickok, G. (2011). Are mirror neurons the basis of speech perception? Evidence from five cases with damage to the purported human mirror system. Neurocase, 17, 178-187.

Searle, J. R. (1969). Speech acts: An essay in the philosophy of language. Cambridge: Cambridge University Press.

Searle, J. R. (1980). Minds, brains, and programs. Behavioral and Brain Sciences, 3, 417-457.

Shebani, Z., & Pulvermüller, F. (2013). Moving the hands and feet specifically impairs working memory for arm- and leg-related action words. Cortex, 49, 222-231.

Shtyrov, Y., Kujala, T., & Pulvermüller, F. (2010). Interactions between language and attention systems: early automatic lexical processing? Journal of Cognitive Neuroscience, 22, 1465-1478.

Simmons, W. K., Ramjee, V., Beauchamp, M. S., McRae, K., Martin, A., & Barsalou, L. W. (2007). A common neural substrate for perceiving and knowing about color. Neuropsychologia, 45, 2802-2810.

Sporns, O. (2013). Network attributes for segregation and integration in the human brain. Current Opinion in Neurobiology

Sporns, O., Tononi, G., & Kotter, R. (2005). The human connectome: A structural description of the human brain. PLoS Computational Biology, 1, e42.

Stalnaker, R. C. (2002). Common ground. Linguistics and Philosophy, 25, 701-721.

Trumpp, N. M., Kliese, D., Hoenig, K., Haarmeier, T., & Kiefer, M. (2013). Losing the sound of concepts: Damage to auditory association cortex impairs the processing of sound-related concepts. Cortex, 49, 474-486.

Tsumoto, T. (1992). Long-term potentiation and long-term depression in the neocortex. Progress in Neurobiology, 39, 209-228.

Verduzco-Flores, S., Bodner, M., Ermentrout, B., Fuster, J. M., & Zhou, Y. (2009). Working memory cells' behavior may be explained by cross-regional networks with synaptic facilitation. PLoS One, 4, e6399.

Vihman, M. M., Macken, M. A., Miller, R., Simmons, H., & Miller, J. (1985). From babbling to speech: A re-assessment of the continuity issue. Language, 61, 397-445.

Watkins, K. E., Strafella, A. P., & Paus, T. (2003). Seeing and hearing speech excites the motor system involved in speech production. Neuropsychologia, 41, 989-994.

Wennekers, T., Garagnani, M., & Pulvermüller, F. (2006). Language models based on Hebbian cell assemblies. Journal of Physiology (Paris), 100, 16-30.

Wennekers, T., & Palm, G. (2007). Modeling generic cognitive functions with Q7 operational Hebbian cell assemblies. In: M. L. Weiss (Ed.), Neural network research horizons (pp. 225-294). Nova Science Publishers.

Wermter, S., Page, M., Knowles, M., Gallese, V., Pulvermuller, F., & Taylor, J. (2009). Multimodal communication in animals, humans and robots: An introduction to perspectives in brain-inspired informatics. Neural Networks, 22, 111-115.

Wiener, N. (1961). Cybernetics: Or, control and communication in the animal and the machine. New York: London: MIT Press, John Wiley & Sons.

Willems, R. M., & Casasanto, D. (2011). Flexibility in embodied language understanding. Frontiers in Psychology, 2, 116.

Wilson, S. M., Saygin, A. P., Sereno, M. I., & Iacoboni, M. (2004). Listening to speech activates motor areas involved in speech production. Nature Neuroscience, 7, 701-702.

Wittgenstein, L. (1953). Philosophical Investigations. Oxford: Blackwell Publishers.

Yeung, H. H., & Werker, J. F. (2013). Lip movements affect infants' audiovisual Q2 speech perception. Psychological Science (in press)

Young, M. P., Scannell, J. W., & Burns, G. (1995). The analysis of cortical connectivity. Heidelberg: Springer.

Zatorre, R. J., Chen, J. L., & Penhune, V. B. (2007). When the brain plays music: auditory-motor interactions in music perception and production. Nature Reviews Neuroscience, 8, 547-558.

Zipser, D., Kehoe, B., Littlewort, G., & Fuster, J. (1993). A spiking network model of short-term active memory. Journal of Neuroscience, 13, 3406-3420.

Zwaan, R. A., & Taylor, L. J. (2006). Seeing, acting, understanding: motor resonance in language comprehension. Journal of Experimental Psychology General, 135, 1-11.