Hands and Objects in Language, Culture, and Technology: Manual Actions at Workplaces between Robotics, Gesture, and Product Design

MANUACT studies gestures as part of human communication, investigating both their cultural relevance and practical applications. It focuses on object-related gestures (such as the gesture for “making a telephone call”), asking how such gestures come about, how they are related to current and past technologies, and how they are used for communication.

A further focus lies on gesture control as an intuitive and effective means of human-machine interaction. The guiding assumption is that artificial gestures will be easy to learn if they are connected with our natural gesture repertoire, and avoid conflict with it. Machines cannot decide weather they or a human are addressed, so we should avoid ambiguity. At the same time, it is important to conventionalize gestures. For example, a “Stop!” gesture should be easy to remember and intuitive to use when a self-driving car comes towards you.

These and related research questions are pursued in a multidisciplinary setting that combines three different perspectives: a linguistic and semiotic approach, an ergonomic approach, and a perspective that unites museum studies with artistic research. The anthropological dimension of gestures, their cultural history, their role in face-to-face communication, and their rapidly increasing technological applications are investigated on the basis of extensive empirical studies.

For this purpose, a video corpus of gestures in human dialogues, and object handling in different settings, has been collected. A further approach consists in the recontextualisation of industrial objects in a museum setting, and artistic reflections of everyday practices at work and at home. In MANUACT, the professor of German Linguistics, Semiotics and Multimodal Communication and the professor of Ergonomics and Innovation Research at the Chemnitz University of Technology cooperate with the Saxon Museum of Industry Chemnitz and the Ars Electronica Futurelab (Linz). We thank the artist Anette Rose for her support during the application phase of the MANUACT research project from 2012 to 2015 (www.anetterose.de).

Selected results and outcomes of the project will be presented to the public in an exhibition at the Saxon Museum of Industry in Chemnitz.

MANUACT – Hands and Objects in Language, Culture, and Technology: Manual Actions at Workplaces between Robotics, Gesture, and Product Design

Logo des Projekts MANUACT

Project leader: Prof. Dr. Ellen Fricke (Linguistics and Semiotics, Chemnitz University of Technology)

Prof. Dr. Angelika Bullinger-Hoffmann (Ergonomics and Innovation Management, Chemnitz University of Technology)

In cooperation with the Saxon Museum of Industry, Chemnitz and the Ars Electronica Futurelab, Linz

Duration: April 2015 to March 2018

Funded by the Federal Ministry of Education and Research (BMBF)

 

Logo der TU Chemnitz
Logo der Ars Electronica Futurelab
Logo des Industriemuseums Chemnitz
Logo des BMBF
Logo des deutschen Luft- und Raumfahrtzentrums

Project participants

Von links nach rechts: Ulrike Lynn, Angelika C. Bullinger-Hoffmann, Thomas Seeling, Ellen Fricke, Jana Bressem und Daniel Schöller.

 

Prof. Dr. Ellen Fricke (project leader)
Professor of German Linguistics, Semiotics and Multimodal Communication

ellen.fricke [at] phil.tu-chemnitz.de
Tel.: 0371 531-32895

Visitor address:
Thüringer Weg 11, Raum 210
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Philosophische Fakultät
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Prof. Dr. Angelika C. Bullinger-Hoffmann
Professor of Ergonomics and Innovation Research

angelika.bullinger-hoffmann [at] mb.tu-chemnitz.de
Tel.: 0371 531-23210

Visitor address:
Erfenschlager Straße 73, Gebäude F/102
09125 Chemnitz

Mailing address:
Technische Universität Chemnitz
Fakultät für Maschinenbau
Professur Arbeitswissenschaft und Innovationsmanagement
09107 Chemnitz

 


 

Dr. Jana Bressem (project coordinator)

jana.bressem [at] phil.tu-chemnitz.de
Tel.: 0371 531-32992

Visitor address:
Thüringer Weg 11, Raum 210A
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Dr. Ulrike Lynn (research associate)

ulrike.lynn [at] phil.tu-chemnitz.de
Tel.: 0371 531-32997

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Johannes Müller-Viezens, M.A. (research associate)

johannes.mueller-viezens [at] phil.tu-chemnitz.de
Tel.: 0371 531-31504

Visitor address:
Thüringer Weg 11, Raum 302
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Daniel Schöller, M.A. (research associate)

daniel.schoeller [at] phil.tu-chemnitz.de
Tel.: 0371 531-31654

Visitor address:
Thüringer Weg 11, Raum 302
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Dipl.-Soz. Thomas Seeling (research associate)

thomas.seeling [at] mb.tu-chemnitz.de
Tel.: 0371 531-39885

Visitor address:
Erfenschlager Straße 73, Gebäude F/007
09125 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Arbeitswissenschaft und Innovationsmanagement
09107 Chemnitz

 


 

Dr. des. Matthias Meiler (associated member)

matthias.meiler [at] phil.tu-chemnitz.de
Tel.: 0371 531-31503

Visitor address:
Thüringer Weg 11, Raum 301
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 


 

Dr. Martin Siefkes (associated member)

Visitor address:
Thüringer Weg 11, Raum 213
09126 Chemnitz

Mailing address:
Technische Universität Chemnitz
Professur Germanistische Sprachwissenschaft, Semiotik und Multimodale Kommunikation
09107 Chemnitz

 

Events

A number of events will be organized in connection with the MANUACT project.

 

 

Publications

  • Bernhagen, Max und Angelika C. Bullinger (in press). Die Auswirkungen von Lernkurven auf die Effizienz gestenbasierter Eingabegeräte – eine Longitudinalstudie.
  • Bernhagen, Max, André Dettmann und Angelika C. Bullinger (2015). Einflussfaktoren der Beanspruchung durch gestenbasierte Eingabegeräte. In: Martin Pielot et al. (ed), Mensch und Computer 2015 – Tagungsband, pp. 291–294. Berlin: De Gruyter.
  • Dettmann, André, Max Bernhagen und Angelika C. Bullinger (in press). Eignung gestenbasierter Eingabegeräte an Bildschirmarbeitsplätzen – eine vergleichende Untersuchung. pdf
  • Fricke, Ellen und Irene Mittelberg (in press). Gesten. In: Frank Liedtke (eds.). Handbuch Pragmatik. Stuttgart: Metzler.
  • Fricke, Ellen (in press). Mental Spaces und deiktische Raumerzeugung in der Interaktion. In: Robert Mroczynski und Alexander Ziem (eds.), Mental Spaces in der interaktionalen Semantik. Special Issue, Zeitschrift für Gesprächsforschung.
  • Fricke, Ellen (in press). Deixis, gesture, and space from a semiotic point of view. In: Federica da Milano (eds.), Space and Language – On Deixis. Amsterdam, Philadelphia: John Benjamins.
  • Fricke, Ellen (2015). Grammar, deixis, and multimodality between code-manifestation and code-integration or why Kendon's Continuum should be transformed into a gestural circle . In: Gaëlle Ferré & Mark Tutton (eds.), Gesture and Speech in Interaction – 4th edition (GESPIN 4), pp. 14–16. <https://hal.archives-ouvertes.fr/hal-01195646>
  • Fricke, Ellen (2015). Gesture. In: Federica da Milano und Konstanze Jungbluth (eds.), Manual of Deixis in Romance Languages. Berlin, Boston: Mouton de Gruyter, pp. 48–76
  • Fricke, Ellen (2015). Grammatik und Multimodalität. In: Christa Dürscheid und Jan Georg Schneider (eds.), Handbuch Satz, Äußerung, Schema. Berlin und Boston: De Gruyter.
  • Lynn, Ulrike (2015). Aspects of digital gesture lexicography pdf. In: Gaëlle Ferré & Mark Tutton (ed.), Gesture and Speech in Interaction – 4th edition (GESPIN 4), pp. 173–177. <https://hal.archives-ouvertes.fr/hal-01195646>
  • Seeling, Thomas, Ellen Fricke, Daniel Schöller und Angelika C. Bullinger (2016): Natürliche User Interfaces durch den Einbezug von Nutzern gestalten: Implikationen für ein Entwickler-Gestenmanual. Mensch-Technik-Interaktion im Industrie 4.0 Zeitalter, Tagungsband 8. VDI/VDE Fachtagung USEWARE. 06.10.2016 bis 07.10.2016, Dresden. pp. 65–81.
  • Seeling, Thomas, Frank Dittrich, Ellen Fricke, Angelika C. Bullinger (2016). Gestenbasierte Google-Earth-Bedienung. Implikationen für ein natürliches Gesten-Set am Beispiel einer 3D-Topographieanwendung. In: Gesellschaft für Arbeitswissenschaft e.V. (eds.), Arbeit in komplexen Systemen – Digital, vernetzt, human?! 62. Kongress der Gesellschaft für Arbeitswissenschaft. (S. 88–92). Dortmund: GfA-Press.
  • Seeling, Thomas et al. (2016). Entwicklung eines Gestenmanuals zur Gestaltung von Natural User Interfaces. In: Proceedings Innteract Conference 2016. Awi&I-Wissenschaft & Praxis. Chemnitz.
  • Seeling, Thomas, Ulrike Lynn, Daniel Schöller, Ellen Fricke und Angelika C. Bullinger (in press). Intuitiv, komfortabel, leicht zu erinnern? Evaluation grundlegender Gesten zur Selektion, Navigation und Manipulation von Objekten in interaktiven Systemen. In: Butz, A.; Koch, M.; Schlichter, J. (eds.), Sozial Digital – Gemeinsam auf neuen Wegen, Mensch & Computer 2016. Berlin: De Gruyter Oldenbourg.
  • Siefkes, Martin (zusammen mit Doris Schöps) (in press). Body postures as signs: the embodiment of socialist roles in East German feature films. In: Irene Mittelberg und Alexander Gerner (eds.), Body Diagrams. John Benjamins.
  • Siefkes, Martin (in press). Perceptual qualities of literary style. In: Scientific Study of Literature 7, 1.
  • Siefkes, Martin (2015). How semiotic modes work together in multimodal texts: Defining and representing intermodal relations. 10plus1 – Living Linguistics, 1, 2015: pp. 113–131.
  • Siefkes, Martin & Emanuele Arielli (in press), The Aesthetics and Multimodality of Style: Experimental Research on the Edge of Theory. Bern: Peter Lang.
  • Siefkes, Martin (in press). “Intermodal Contrast in Film: Looking for the Aesthetics of Intermodal Relations”, in: Frida Forsgren & Elise Seip Tønnessen (eds.), Multimodality and Aesthetics. London: Routledge.

 

Talks and lectures

  • Ellen Fricke, 20 to 22 September 2017, Third Bremen Conference on Multimodality, University of Bremen. Keynote: Language, gestures, and pictures: Multimodality between code manifestation and code integration (invited talk).
  • Daniel Schöller, Jana Bressem, Ellen Fricke, 19 to 21 September 2017, Electronic Lexicography in the 21st Century: Lexicography from Scratch, Leiden, Niederlande. Lemmatization and Meaning Representation of Gestures in an Electronic Dictionary.
  • Jana Bressem, 12 to 16 September 2017, 15th International Conference “Borders. Contact – Communication – Contrast” of the German Semiotics Society, Passau. Panel “Objektgebrauch als Zeichenprozess in ‚realen‘ und virtuellen Welten: Wie aus Handlungen Gesten werden” (organised by Jana Bressem, Ellen Fricke, Irene Mittelberg, Roland Posner). Wischen, werfen, halten – Handlungen als Quelle für konventionalisierte Gesten des Alltagsgebrauchs.
  • Ellen Fricke and Daniel Schöller, 12 to 16 September 2017, 15th International Conference “Borders. Contact – Communication – Contrast” of the German Semiotics Society, Passau. Panel “Objektgebrauch als Zeichenprozess in ‘realen’ und virtuellen Welten: Wie aus Handlungen Gesten werden” (organised by Jana Bressem, Ellen Fricke, Irene Mittelberg and Roland Posner). Objektgebrauch und seine multimodale Versprachlichung: Wie Handlungsschemata, Verbsemantik und redebegleitende Gesten zusammenwirken.
  • Johannes Müller-Viezens, 12 to 16 September 2017, 15th International Conference “Borders. Contact – Communication – Contrast” of the German Semiotics Society, Passau. Panel “Objektgebrauch als Zeichenprozess in ‘realen’ und virtuellen Welten: Wie aus Handlungen Gesten werden” (organised by Jana Bressem, Ellen Fricke, Irene Mittelberg and Roland Posner). Virtuelle Kamera und Objektgebrauch im Computerspiel.
  • Ellen Fricke, 26 to 28 July 2017, Conference “Pictures and Texts – Pictures as Texts. Iconicity and Indexicality in Graphic Communication”, Exzellenzcluster TOPOI, Berlin (invited talk).
  • Ellen Fricke and Jana Bressem, 16 to 21 July 2017, 15th International Pragmatics Conference, Belfast. Panel “Empractical speaking and knowledge construction” (organised by Stefan Hauser and Wolfgang Kesselheim). Gestures and technological innovation: Metaphorical concepts and empractical speaking as basis for constructing human-machine interfaces.
  • Johannes Müller-Viezens, 09 to 11 June 2017, “International Conference on Multimodal Communication (ICMC 2017): Developing New Theories and Methods”, University of Osnabrück (organised by Alexander Bergs and Mark Turner). Performing Speech Under Copresence. Multimodality and its Conceptualizing in Face-to-Face Role-Playing Games.
  • Thomas Seeling and Angelika C. Bullinger, 16 February 2017, 63th Conference of the Society of Ergonomics “Soziotechnische Gestaltung des digitalen Wandels – kreativ, innovativ, sinnhaft”, FHNW Brugg-Windisch and ETH Zürich. Natürliche User Interfaces mit Anwendern gestalten – Eine Handreichung für die Gestaltung intuitiver Gesten-Sets.
  • Ulrike Lynn, 2016, Culture – Media – Communication Conference, Univerzity Palackého v Olomouci. Gesture Dictionaries as a Tool for Comparative Research into Gesture Development (keynote lecture).
  • Martin Siefkes, 06 December 2016, Institute for Applied Linguistics, University of Technology Dresden. Das virtuelle Museum. Ein multimodaler Blick auf Online-Ausstellungen (invited talk).
  • Martin Siefkes, 15 October 2016, International Conference “Transmediations! Communication across Media Borders”, Linnæus University, Växjö, Schweden. Your Exhibition Will Soon Be Virtual. Transmediating Museums into the Digital Sphere.
  • Thomas Seeling, Ellen Fricke, Daniel Schöller, Angelika C. Bullinger, 06 October 2016, 8th VDI/VDE-Fachtagung USEWARE, Dresden. Natürliche User Interfaces durch das Einbeziehen von Nutzern gestalten. Implikationen für ein Entwickler-Gestenmanual.
  • Ellen Fricke, 21 July 2016, ISGS 2016 “Gesture – Creativity – Multimodality”, Université Sorbonne Nouvelle, Paris 3. Gesture, virtual globes and technological innovation: Metaphorical concepts as basis for gesturally steering Google Earth with Leap Motion.
  • Daniel Schöller, 08 July 2016, “1st Chemnitz/Dresden Corpus Linguistics & Digital Humanities Conference” (CorpDH 2016), University of Technology Chemnitz. Korpuslinguistische Methodik und Multimodale Grammatik.
  • Martin Siefkes, 08 July 2016, “1st Chemnitz/Dresden Corpus Linguistics & Digital Humanities Conference” (CorpDH 2016) , University of Technology Chemnitz. Die Verbindung von multimodaler Semiotik und Digital Humanities in der Analyse von Online-Ausstellungen.
  • Ellen Fricke, Ulrike Lynn, Daniel Schöller, Thomas Seeling, Angelika C. Bullinger, 24 June 2016, Innteract Conference 2016 “3D Sensation”, University of Technology Chemnitz. Hände und Objekte: Perspektiven linguistischer Gestenforschung für die Gestaltung von Mensch-Maschine-Schnittstellen.
  • Ellen Fricke, 31 May 2016, Linguistic colloquium, University of Würzburg. Perspektiven linguistisch-semiotischer Multimodalitätsforschung am Beispiel der Negation im Deutschen.
  • Thomas Seeling, Ellen Fricke, Frank Dittrich und Angelika C. Bullinger, 2 to 3 March 2016, 62th Conference of the Society of Ergonomics, RWTH Aachen. Gestenbasierte Google-Earth-Bedienung. Implikationen für ein natürliches Gesten-Set am Beispiel einer 3D-Topographieanwendung.
  • Ellen Fricke, 28 January 2016, Ringvorlesung “Grenzen”, University of Passau. Sprache und Raum: Multimodale Grenzziehungen in der Interaktion. (Talk on invitation by Jan-Oliver Decker)
  • Ellen Fricke, 21 January 2016, “Gebärdensprache und Gestik. Grundlagen und Anwendungen”, Eröffnungstagung SignGes 2016 an der RWTH Aachen, Perspektiven sprachwissenschaftlicher Multimodalitätsforschung. (Talk on invitation by Irene Mittelberg.)
  • Ellen Fricke, 26 October 2015, BMBF-Statustagung “Die Sprache der Objekte”, University Club Bonn. Project presentation: Hands and Objects in Language, Culture, and Technology and discussion participant in the panel “Digitale Methoden und Neue Medien in der geistes- und sozialwissenschaftlichen Objektforschung”.
  • Ellen Fricke, 16 October 2015, Opening event, MANUACT, Sächsisches Industriemuseum und Technische University of Chemnitz. Opening Lecture: Hands and Objects in Language, Culture, and Technology, and presentation of the MANUACT project.
  • Ellen Fricke, 2 October 2015, Conference “Linguistische Komplexität – ein Phantom?”, Justus-Liebig-University Gießen, Schloss Rauischholzhausen. Deixis, Grammatik und sprachliche Komplexität: Prozesse multimodaler Integration in Nominalgruppen des Deutschen. (invitation by Mathilde Hennig) Vortragsabstract 
  • Martin Siefkes, 22 September 2015, International Conference “Second Bremen Conference on Multimodality” (BreMM15), University of Bremen. Multimodal Text Interpretation: Modelling the Whole Process.
  • Ulrike Lynn, 2 to 4 September 2015, International Conference “Gesture and Speech in Interaction” (GESPIN 4), Université de Nantes. Poster: Keep in touch – A dictionary of contemporary physical contact gestures.
  • Ellen Fricke, 2 September 2015, International Conference “Gesture and Speech in Interaction” (GESPIN 4), Université de Nantes. Keynote: Grammar, deixis, and multimodality between code-manifestation and code-integration or why Kendon's Continuum should be transformed into a gestural circle. Abstract.

Acknowledgments

We gratefully acknowledge the support of the following businesses and museums who have helped us with the data collection process:

 

Scroll to top