Ernst Strüngmann Forum

 

Where’s the Action?

The Pragmatic Turn in Cognitive Science

October 26–31, 2014

Frankfurt am Main, Germany

Andreas K. Engel, Karl Friston, and Danica Kragic, Chairpersons

Program Advisory Committee

Andreas K. Engel, Dept. of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany

Karl Friston, Wellcome Trust Centre for Neuroimaging, University College London, London, UK

Bernhard Hommel, Faculty of Social Sciences, Institute of Psychology, University of Leiden, The Netherlands

Danica Kragic, School of Computer Science and Communication, Centre for Autonomous Systems, Royal Institute of Technology, Stockholm, Sweden

Julia Lupp, Ernst Strüngmann Forum, Frankfurt, Germany

Paul Verschure, Center of Autonomous Systems and Neurorobotics, Technology Department, Universitat Pompeu Fabra, Barcelona, Spain

Gottfried Vosgerau, Institute of Philosophy, University of Düsseldorf, Düsseldorf, Germany

Goals

In cognitive science, we are currently witnessing a "pragmatic turn" away from the traditional representation-centered framework toward a paradigm that focuses on understanding cognition as "enactive," as a form of practice. A key premise of this view holds that cognition should not be understood as serving to make models of the world, but rather as subserving action and being grounded in sensorimotor skills. Accordingly, cognitive states and their associated neural activity patterns should be studied primarily with respect to their functional role in action generation. Such an action-oriented paradigm seems not only conceptually viable, but is already supported by much experimental evidence. Numerous neurobiological findings demonstrate either overtly the action-relatedness of cognitive processing or can be reinterpreted in this new framework. Moreover, new vistas on the functional relevance and the presumed "representational" nature of neural processes are likely to emerge from this paradigm.

The goal of the proposed Forum would be to examine the key concepts of an emerging action-oriented view of cognition and the consequences of such a paradigm shift.

This Forum would contribute to this paradigm shift by enabling a collective discussion and elaboration of key concepts involved in this pragmatic turn, which in turn could be applied and ultimately result in novel approaches in a number of fields, including cognitive science, neuroscience, psychology, robotics, and philosophy of mind. While action-oriented views are currently beginning to emerge in these individual fields, strong links across these domains are mostly lacking. Bringing together leading proponents from these fields to a Strüngmann Forum would substantially speed up interdisciplinary interactions and, by confronting different approaches and types of data, enable novel and more integrated perspectives.

Top of page

Background

Since its formation as a discipline aimed at providing a naturalistic account of the mind and its processes, cognitive science has been dominated by a computational-representational view of cognition. The following key assumptions characterize the classical representation-centered paradigm in cognitive science:

  • Cognition is understood as computation over mental representations.
  • The subject of cognition is a detached observer with a "bird’s eye" view of the world.
  • Intentionality is explained by the representational nature of mental states.
  • The architecture of cognitive systems is conceived as being highly modular.
  • Processing in subsystems is assumed to be largely context-invariant.
  • Computations are thought to occur in a substrate-neutral manner (functionalism).
  • Models of cognition take into account only inner states of a cognitive system (individualism).

These key assumptions, which go back to the work of Fodor (1981), Newell and Simon (1972), and other protagonists of the representational theory of mind, seem to be present, albeit with different emphasis, in most theoretical accounts on cognition. While this paradigm has been highly fruitful in stimulating important research in the early decades of cognitive science, massive criticisms have been summoned, claiming that the classical view may be highly biased if not misleading in nature (Winograd and Flores 1986; Brooks 1991; Varela et al 1991; Dreyfus 1992; Clark 1995, 1998; O'Regan and Noe 2001; Noe 2004; Engel et al 2013).

In response, an action-oriented paradigm is emerging (Varela et al 1991; Clark 1998; O'Regan and Noe 2001; Noe 2004, 2009, O'Regan 2011). Initially, this paradigm shift began, and has most explicitly been developed, in robotics (Winograd and Flores 1986; Brooks 1991; Dreyfus 1992; Pfeifer and Bongard 2007). More recently it has begun to gain impact on cognitive psychology (Hommel et al 2001; O'Regan and Noe 2001; Schütz-Bosbach and Prinz 2007) and neuroscience (Jeannerod 2001; Beauchamp and Martin 2007; Friston 2010; Friston et al 2010; Pulvermüller and Fadiga 2010; Engel 2010; Engel et al 2013).

The basic concept of the action-oriented paradigm is that cognition should not be understood as a capacity of deriving world models, which then might provide a database for thinking, planning, and problem-solving. Rather, it emphasizes that cognitive processes are closely intertwined with action and that cognition can best be understood as "enactive"; that is, as a form of practice itself. The view, which is advocated by proponents of the pragmatic turn, can be summarized as follows (Varela et al 1991; Clark 1998; Noe 2009; Engel 2010; O'Regan 2011; Engel et al 2013):

  • Cognition is understood as capacity of generating structure by action.
  • The cognitive agent is immersed in its task domain.
  • System states acquire meaning through their functional role in the context of action.
  • The functioning of cognitive systems is thought to be inseparable from embodiment.
  • A holistic view on the architecture of cognitive systems prevails, emphasizing the dynamic nature and context-sensitivity of processing.
  • Models of cognition take into account the "extended" nature of cognitive systems.

It should be noted that the concept of action, as used here, is neither coextensive with that of behavior nor with that of movement (Mead 1938; Engel et al 2013). The expression of action is used in a wider sense, which also includes acts not involving any overt movements like, e.g., thinking, calculating, imagining or deciding. The description of acts or actions typically makes references to goals, whereas behavior can be described without making any reference to mental states.

An action-oriented paradigm is supported by a number of prominent and highly discussed conceptual approaches. The notion that cognition can only by understood by considering its inherent action-relatedness is a key postulate of the "enactive" approach developed by Varela, Thompson and Rosch (1991). According to their view, cognition should not be considered as producing veridical representations of the environment but, rather, as the capacity of generating structure by action (Varela et al 1991). A related, strongly action-oriented view of cognition has also been advocated by Clark (1995, 1998).

Of particular relevance in this context is the sensorimotor contingency theory put forward by O’Regan and Noë (2001). This approach builds on earlier approaches in explaining the fundamental role of action for perception and awareness like, e.g., Gibson's affordances (Gibson 1979). It also relates to older neurobiological concepts such as the "reafference principle" of von Holst and Mittelstaedt (1950), who discovered that an efference copy is needed for the unambiguous interpretation of sensory signals (Wolpert and Flanagan 2001; Wolpert et al 2011; Friston 2011). According to O’Regan and Noe (2001), the agent’s sensorimotor contingencies (SMCs) are constitutive for cognitive processes. In this framework, SMCs are defined as law-like relations between movements and associated changes in sensory inputs that are produced by the agent’s actions. Once acquired, an agent can use these SMCs to predict consequences of its own actions.

Recent work in cognitive robotics suggests that learning of such predictions could, for instance, mediate the acquisition of object concepts (Krüger et al 2007; Bergstrom et al 2011; Maye and Engel 2012), grounding knowledge of objects in repertoires of actions that can be executed upon them. This theoretical perspective is also closely related to the active inference approach to action and perception (Friston and Stephan 2007; Friston 2010; Friston et al 2010) as well as to models of predictive coding (Rao and Ballard 1999).

An action-oriented paradigm has the potential of changing our view of the brain and its function profoundly. If mapped to the neuroscientific level of description, the conceptual premises of the pragmatic stance may lead to a redefinition of some of the basic explananda. Then, neuroscience would not need to explain how brains act as world-mirroring devices (Marr 1982; Churchland 1994), but rather how they can serve as "vehicles of world-making" (Varela et al 1991) that support, based on individual learning history, the construction of the experienced world and the guidance of action. Data from developmental and cognitive neuroscience seem to support such a departure from more classical views on cognition and brain function. These findings and their potential implications would constitute the starting point for the proposed Forum.

The time is ripe for a discussion and elaboration of such action-oriented approaches in a number of fields including cognitive science, neuroscience, psychology, robotics, and philosophy of mind. The Ernst Strüngmann Forum is an ideal setting in which to explore the preconditions and possible consequences of such a paradigm shift. Due to its highly interdisciplinary nature and the openness of the discussions that are key elements of the Forum, action-oriented views emerging in these different fields could cross-fertilize and start to integrate, paving the way toward a novel theory of cognition.

Top of page

Key Controversial Issues to be Discussed at the Forum

Challenging representations: The functional role and the semantics of neural states is one of the key topics for the planned Forum. The insight that cognition may be fundamentally grounded in action seems to enforce a radical change in how we conceive of the functional significance of neural activity patterns. Rather than describing states of the outside world, brain states prescribe possible actions (Clark 1998). Thus, they might better be understood as “directives” guiding action (Engel et al 2013), rather than as “representations.”

Cognitive role of motor brain structures: A key question is to what degree the function of motor regions can be understood as directly supporting cognition, as opposed to a view which assigns merely “output” functions to these circuits. For instance, attention and decision making may rely much more on motor regions than previously assumed (Engel et al 2013). From the viewpoint of an integrated sensorimotor approach, it may be questioned whether it makes sense to use the classical categories of “motor” and “sensory” cortex. Rather, these cortical regions might by viewed as proprioceptive and exteroceptive sensorimotor areas encoding SMCs. For example, visual cortex might be considered as the recipient of top-down predictions about the consequences of oculomotor acts.

Role of sensorimotor contingencies: While the role of SMCs in basic sensorimotor integration seems undisputed, a critical question is to what extent more complex cognitive functions, such as object recognition or action planning, can be achieved by learning SMCs (Maye and Engel 2012). The challenging question is whether SMC knowledge is sufficient for implementing complex cognitive functions or whether higher cognitive processes based on other principles are required.

Predictive coding and active inference: A key mechanism for cognitive processing seems to be the optimization of predictions and the minimization of prediction errors. It has been suggested that new views unifying perception, cognition and motor control may emerge from this basic principle (Friston 2010; Friston et al 2010). The implications of this principle for a novel understanding of the relation between action and cognition would be another key topic for the Forum.

Action-relatedness of phenomenal states: In more classical accounts, conscious awareness is largely detached from action and the activation of motor circuits. An important question is whether implications of the pragmatic turn would also concern current models of consciousness. The sensorimotor account (O’Regan and Noe 2001)rt the notion of the pragmatic turn. For developmental robotics this issue is equally relevant to answer questions like how robots can achieve mastery of high-dimensional action spaces.

Joint action and social cognition: Social aspects of cognitive processing have, in many accounts, been conceptualized as mainly involving “theory-of-mind” type representations in the individual brain. A key question is whether this largely disembodied approach fully captures the nature of social interactions. Alternatively, a perspective aiming at grounding social cognition in joint action, including e.g. synchronized movements, might have large potential. Ambitious questions are whether enactive approaches to social cognition would also extend to interactions between humans and robots and what mechanisms might establish social cognition in artificial agents.

Agenda for pragmatic neuroscience: A key question is whether the conceptual shifts implied by the pragmatic turn may lead to the development of novel experimental paradigms and strategies. A radical action-orientedness would violate many practical constraints and theoretical premises of neuroscience as they have been functioning in the past decades.

Top of page

Working Structure and Key Questions for the Discussion Groups
We have defined four themes for the group discussions of the proposed Forum. The first group will address action-oriented views in relation to the development of cognition. The second group will focus on novel perspectives emerging for the understanding of consciousness. The third group will discuss action-oriented models for other cognitive processes. Finally, the fourth discussion group will address, at a more general level, the potential contributions of action-oriented views to a paradigm shift in cognitive science. A number of overarching questions for the discussion in these four groups are suggested below.

Issues to be addressed by all four discussion groups:

  • Which methodological and theoretical principles does the pragmatic turn suggest?
  • How can they be tested?
  • What is novel and/or radical about this paradigm shift?
  • How is the new paradigm different from earlier precedents (e.g., behaviorism)?

(For explication)

  • Which methodological and theoretical principles can be derived from the data, experiments and concepts discussed in the group?
  • What are testable hypotheses that derive from those principles and which critical experiments could serve to validate these?
  • What are the relations and possible points of convergence/divergence between different action-oriented accounts in cognitive science?
  • What is the impact on current theories of “representation”? Is the concept of representation still useful? What are “units of knowledge” in the new paradigm? To what extent are these principles representational in nature?
  • What are the limitations of action-oriented explanatory strategies?
  • What are the defining features of the new paradigm?
  • What are methodological implications for empirical studies?
  • Which empirical data support action-oriented models of cognitive processing?

Group 1: Development, acquisition, and adaptation of action-oriented processing

  • What mechanisms are involved in acquiring action-derived cognitive processing?
  • What synergies exist between perception and action in development (e.g. leading to learning of SMCs)?
  • To what extent do these synergies provide a scaffold for adaptive behavior and cognition in the mature agent?
  • Which cognitive operations can be understood as internalized overt action?
  • How can episodic memory and declarative knowledge be grounded in procedural/habitual learning and its neural mechanisms?
  • Can SMCs be exploited to acquire abstract concepts (e.g. the notion of an electromagnetic field or of a prime number)?
  • Is knowledge of SMCs based exclusively on own sensorimotor activity, or can it also be acquired by generalization, abstraction, or inter-individual communication?
  • How real is the sensorimotor division in the brain; is it a cause or consequence of sensorimotor activity?

Group 2: Action-oriented models of cognitive and functional processing

  • Which cognitive functions (e.g. categorization, attention, decision making, rule following, tool use, language) can be grounded in sensorimotor processes?
  • How can this be modeled and formalized exploiting, e.g., predictive coding, active inference or information theory?
  • What is the relation between habits (model-free behavior) and goal-oriented (model-based) action?
  • What is the role of action in understanding social interactions?
  • Which neuroscientific evidence/constraints specifically speak to an action-oriented account of cognition?
  • Does the action-oriented approach furnish novel hypotheses on neural mechanisms of cognitive processes?
  • What are the limitations of an action-oriented account of cognitive functions?

Group 3: Action-oriented understanding of consciousness and the structure of experience

  • Which empirical data support an action-oriented account of conscious awareness?
  • Can SMCs account for conscious awareness?
  • How do SMCs-derived predictions constrain or define conscious states?
  • Does the emergence or quality of conscious states depend on the complexity (e.g., hierarchical depth) of action repertoires?
  • Do action-oriented approaches help to account for the function of consciousness?
  • Would cognitive capacities and consciousness deteriorate in chronically movement-disabled patients with, e.g., amyotrophic lateral sclerosis or locked-in syndrome?
  • Is sensory awareness a necessary consequence of mastering SMCs?
  • Would altered states of consciousness imply changes in action-perception coupling?
  • Can theory of mind and self-awareness be grounded in action observation?
  • Do action-oriented approaches speak to integration accounts of consciousness (e.g., global workspace, integrated information)?

Group 4: Implications of action-oriented paradigm shifts in cognitive science

  • What are epistemological implications of the pragmatic turn, e.g., for our view of reality and our ways of acquiring knowledge?
  • What are societal implications of action-oriented approaches, e.g., for educational programs, structuring of social processes?
  • What are the implications of a pragmatic turn for research programs and experimental strategies in cognitive science?
  • What are implications for the modeling of cognitive processes and the implementation of artificial sentient systems?
  • What are the potential implications for a better understanding of cognitive dysfunctions and the pathogenesis of neuropsychatric disorders?
Top of page

References

Beauchamp MS, Martin A (2007) Grounding object concepts in perception and action: evidence from fMRI studies of tools. Cortex 43: 461-468

Bergstrom N, Bjorkman M, Kragic D (2011) Generating object hypotheses in natural scenes through human-robot interaction. In: Intelligent Robots and Systems (IROS), 2011 IEEE/RSJ International Conference on, pp 827-833

Brooks RA (1991) Intelligence without representation. Artif Intell 47: 139-159

Churchland PS et al (1994) A critique of pure vision. In Large-scale Neuronal Theories of the Brain (Koch C, Davis J, eds), pp 23-60, The MIT Press

Clark A (1995) Moving minds: situating content in the service of real-time success. Phil Perspect 9: 89-104

Clark A (1998) Being There: Putting Brain, Body, and World Together Again. The MIT Press

Dreyfus HL (1992) What Computers Still Can't Do: A Critique of Artificial Reason. The MIT Press

Engel AK (2010) Directive minds: how dynamics shapes cognition. In Enaction: Toward a New Paradigm for Cognitive Science (Stewart J et al, eds), pp 219-243, The MIT Press

Engel AK, Maye A, Kurthen M, König P (2013) Where‘s the action? The pragmatic turn in cognitive science. Trends Cogn Sci 17: 8-15

Fodor JA (1981) Representations: Essays on the Foundations of Cognitive Science. The MIT Press

Friston KJ, Stephan KE (2007) Free-energy and the brain. Synthese 159: 417-458

Friston KJ (2010) The free-energy principle: a unified brain theory? Nat Rev Neurosci 11: 127-138

Friston KJ, Daunizeau J, Kilner J, Kiebel SJ (2010) Action and behaviour: a free-energy formulation. Biol Cybern 102: 227-260

Gibson JJ (1979) The Ecological Approach to Visual Perception. Houghton Mifflin, Boston

Hommel B et al (2001) The Theory of Event Coding (TEC): a framework for perception and action planning. Behav Brain Sci 24: 849-878

Jeannerod M (2001) Neural simulation of action: a unifying mechanism for motor cognition. Neuroimage. 14: S103-S109

Krüger V, Kragic D, Ude A, Geib C (2007) The meaning of action: a review on action recognition and mapping. Advanced Robotics 21: 1473-1501

Marr D (1982) Vision: A Computational Approach. Freeman & Co

Maye A, Engel AK (2012) Time scales of sensorimotor contingencies. In: Advances in Brain Inspired Cognitive Systems (BICS) 2012 (Zhang H, eds), pp. 240-249. Springer-Verlag

Mead GH (1938) The Philosophy of the Act. University of Chicago Press

Newell A, Simon HA (1972) Human Problem Solving. Prentice-Hall

Noe A (2004) Action in Perception. The MIT Press

Noe A (2009) Out of Our Heads. Hill & Wang

O'Regan JK, Noe A (2001) A sensorimotor account of vision and visual consciousness. Behav Brain Sci 24: 939-973

O'Regan JK (2011) Why Red Doesn't Sound Like a Bell: Understanding the Feel of Consciousness. Oxford University Press

Pfeifer R, Bongard J (2007) How the Body Shapes the Way We Think: A New View of Intelligence. Bradford Books

Pulvermüller F, Fadiga L (2010) Active perception: sensorimotor circuits as a cortical basis for language. Nat Rev Neurosci 11: 351-360

Rao RP, Ballard DH (1999) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat Neurosci 2: 79-87

Schütz-Bosbach S, Prinz W (2007) Perceptual resonance: action-induced modulation of perception. Trends Cogn Sci 11: 349-355

Varela FJ et al (1991) The Embodied Mind: Cognitive Science and Human Experience. The MIT Press

von Holst E, Mittelstaedt H (1950) Das Reafferenzprinzip. Die Naturwissenschaften 37: 464-476

Winograd T, Flores F (1986) Understanding Computers and Cognition: A New Foundation for Design. Ablex Publishing Corporation

Wolpert DM, Flanagan JR (2001) Motor prediction. Curr Biol 11: 729-732

Wolpert DM, Diedrichsen J, Flanagan JR (2011) Principles of sensorimotor learning. Nat Rev Neurosci 12: 739-751

Top of page