Welcome

Workshop: SCENE > Room 'TD2', 2nd floor of Building T, Villejean Université

Clarisse Bardiot, Jacob Hart, David Rouquet, Université Rennes 2 (France)

Workshop: Distant Viewing Toolkit

Lauren Tilton, Taylor Arnold, University of Richmond (USA)

Opening

Keynote: Software for Dancers

Scott deLahunta, Coventry University (UK)

Chair: Clarisse Bardiot

Cocktail

Welcome

Audiovisual Documents Analytics

Chair: Jacob Hart

• Advene, a Look Back on 20 Years of Video Annotation Instrumentation

  Olivier Aubert, Nantes Université (France)

  Advene is a research project and a video annotation tool that started in 2002. It aims at enabling users to practice active reading on their audiovisual documents, originally focusing on movies. At that time, video on the web was basically not usable, except through proprietary plugins that prevented external control. We chose to consider DVDs, which were the main stable and legal source of video documents. Presently, Advene remains actively maintained and utilized, and has adapted to the technological and conceptual changes of the ecosystem, to be able for instance to annotate online videos using ontologies. It is still used in its original application of media studies, but has also been used in other fields such as sociology, ethology, arts…Positioned within the realm of Digital Humanities, Advene reflects the dichotomy inherent to such projects: digitization brings powerful tools and methods to humanities, but also a number of constraints, limitations and methodological changes that have to be taken into account. We would like to take a look back on this 20-year experience to provide some lessons learned about furnishing end-users with an effective video annotation tool, based on the experience of past and current projects. 

  — Olivier Aubert is an independent consultant in knowledge engineering and an associate professor in computer science at the University of Nantes (Polytech Nantes - LS2N) in the DUKe team. He engages in interdisciplinary research focusing on knowledge modeling, engineering, and visualization, with a specific emphasis on audiovisual annotation and digital humanities. As an author and contributor to open-source software, he is the architect and lead developer of the video annotation softwareAdvene, actively involved in projects likeAdA (audio-visual rhetorics of affect) andREMIND, a method to understand the micro-dynamics of experience. He also conducts research related to education through open educational content (CominOpenCourseware), and has developed dedicated video annotation tools such asCocoNotes.

• Deep Screens and Evocative Surfaces: New Research from The Media Ecology Project and the DEV Lab at Dartmouth

  Mark J. Williams, Dartmouth College (USA)

  Deep Screens is a Mellon Foundation Public Knowledge Project that will rip moving image files from hundreds of select dvd's and then use machine learning software to extract and analyze performance movements and expressions across this vast curated collection of U.S. film and television texts from 1895 to the 1970s. This movement and performance data will then be statistically analyzed, with derivatives and results made available in Dataverse through a partnership with the Dartmouth Library. To make the movement data more relatable, motions and gestures will also be applied to animated avatars that can be viewed in virtual reality, abstracted from the context of the original film or television text. The combination of quantitative analysis of the data itself and qualitative viewing of the abstracted movements will provide insight into how acting, cinematography, and technology have evolved across the span of moving image history.

  — Mark Williams received both of his graduate degrees in Critical Studies from The School of Cinema-Television at The University of Southern California. He has previously taught at USC, Loyola Marymount, UC Santa Barbara, and Northwestern. His courses at Dartmouth include surveys of U.S. and international film history, television history and theory, and new media history and theory. He is the director of The Media Ecology Project (MEP), an NEH-supported digital resource at Dartmouth which is developing a virtuous cycle of new interdisciplinary scholarship about archival media that adds value back to participating archives.

• Machine Intelligence for Motion Exegesis (MIME): Applying Pose Estimation and Related Technologies to Analyze Archival Performance Recordings

  Michael Rau and Peter Broadwell, Stanford University (USA)

  MIME is a collaborative effort between faculty in the Department of Theater and Performance Studies at Stanford University and developers at the Stanford Libraries’ Center for Interdisciplinary Digital Research. The project incorporates a suite of software tools for applying deep learning-based pose and gesture estimation to recorded theatrical performances, coupled with a database-backed web interface to facilitate close evaluation and refinement of data inputs and analytical outputs via interactive visualizations and built-in code notebooks. Previously unexplored, computationally oriented research inquiries in performance studies drive the collaboration; these include characterization of the range of choreographed poses within a performance, tracking the evolution of actors’ embodiment of a role, and comparative thematic and stylistic analysis of performances.

  — Michael Rau is a live performance director specializing in new plays, opera, and digital media projects. He has worked internationally in Germany, Brazil, the UK, Ireland, Canada, and the Czech Republic. He has created work in New York City at Lincoln Center, The Public Theater, PS122, HERE Arts Center, Ars Nova, The Bushwick Starr, The Brick, 59E59, 3LD, and Dixon Place. Regionally, his work as been seen at the Ingenuity Festival in Cleveland, OH, and the American Repertory Theatre in Cambridge, MA. He has developed new plays at the Eugene O’Neill National Playwrights Conference, the Lark and the Kennedy Center. Michael Rau is a recipient of fellowships from the Likhachev Foundation, the Kennedy Center, and the National New Play Network. He has been a resident artist at the Orchard Project, E|MERGE, and the Tribeca Performing Arts Center. He has been an associate director for Anne Bogart, Les Waters, Robert Woodruff, and Ivo Van Hove. He is a New York Theater Workshop Usual Suspect and a professor of directing and devising at Stanford University.

  — Peter Broadwell is a Digital Scholarship Research Developer at the Stanford University Libraries’ Center for Interdisciplinary Digital Research, where his work applies machine learning, web-based visualization, and other methods of digital analysis to complex cultural data. He has a Ph.D. in Musicology from the University of California, Los Angeles and an M.S. degree in Computer Science from the University of California, Berkeley. Recent studies in which he has participated have involved automatic translation and indexing of folklore collections in multiple languages, deep learning-based analysis of dance choreography on social media, and multimedia annotation of Japanese Noh theater performances.

The Temporal Dimensions of Distant Viewing

Chair: Michael Sinatra

• Crossing Borders Archives. The Circulation of Stock Shots in Audiovisual Media.

  Matteo Treleani, Université Côte d'Azur (France)

  The ANR CROBORA project aims to shed light on the repetitive and memorial dimension of stock shots in audiovisual media. In partnership with Ina, Rai Radiotelevisione Italiana and Mediaset, 35,000 stock images linked to European integration were collected from twenty years of French and Italian television news, as well as a selection of web accounts. A visualization platform and analysis tools have been implemented to analyze this sample of mediatized visual memory. This talk will focus on the issues involved in collecting, structuring, annotating and visualizing a complex and heterogeneous corpus.

  — Matteo Treleani is a semiotician and media analyst, Associate Professor in Media Studies at the Université Côte d'Azur. He holds a PhD in history and semiotics of texts and images from Paris Diderot University (2012) which was funded by the Institut national de l'audiovisuel. His researches are mainly focused on the use and digitisation of audiovisual heritage. He was associate professor at the University of Lille and visting professor at the University of Turin (2022) and at the Università Cattolica of Milan (2023).

• The Structures of Visual Exchanges

  Nicola Carboni, Université de Genève (Switzerland)

  In the history of representation, the illustrated press has functioned as a significant driving force, curating, and disseminating ideas of visuality to artists and a wider audience. However, how can we even grasp the interaction and circulation of the visual at scale? How is it possible to analyse and comprehend image globalization? To address these questions, the Visual Contagions project has developed (i) a large corpus of digitized image data and (ii) methodologies to analyse image exchanges across the globe. Thanks to this groundwork, the project has conducted several large-scale analyses of visual transmissions. However, this computational study intertwines algorithms, information and data with a historical and conceptual complexity that makes computational investigations difficult to frame. Each of these analyses, in fact, lays its foundation on an ontological decision about the nature of circulation: what does circulation entail? How can we express it? What actual insights can we gain from it? The answers to these questions are fundamental to the creation, and function, of information systems able to support the historical analysis of cultural exchanges. The article explores the technological limitations and possibilities in conceptualizing, curating, and integrating historical data faced by the Visual Contagions project, focusing specifically on the documentation and analysis of visual exchanges.

  — Nicola Carboni is a Postdoctoral research within the Visual Contagions project and Lecturer at the University of Geneva, where he teaches Digital Image, Data Curation, and Knowledge Graph. Previously Fellow at the Swiss Art Research Infrastructure - University of Zurich, Digital Humanities Fellow at Villa I Tatti and Marie Curie Fellow at CNRS MAP. He works on the intersection between knowledge graphs, big visual data and cultural interpretation.

• Using Multidimensional Vector Embeddings to Study Temporal Dimensions of Historical Newsreel Data

  Mila Oiva, Tallinn University (Estonia)

  How to study the visual discourses, emerging from large-scale visual heritage data, in a way that allows the findings to rise from the data, instead of assigning the types of visual representations in advance? And how to grasp temporal changes in these visual discourses? This presentation showcases uses of the Collection Space Navigator (CSN) (Ohm et al.) to analyze central shot frames of historical newsreels to reveal continuities and changes in visual discourses of Soviet newsreels 1944-1992. We used a pre-trained ResNet50 model to encode the newsreel frames into 2048-dimensional feature vectors. After that we used the CSN tool to visualize the encoded images with two-dimensional UMAP projection. This approach allows visual exploration of image collections, which in our case study can be used to reveal also temporal visual patterns.

  Ohm, Tillmann, Mar Canet Solá, Andres Karjus, and Maximilian Schich. “Collection Space Navigator: Interactive Visualization Interface for Multidimensional Datasets,” 2023. https://collection-space-navigator.github.io/.

  — Mila Oiva holds a PhD from Cultural History with the specialization on history of knowledge, media history, and computational research methods. Her current research interests focus on circulation of information and formation of knowledge in long temporal continuum. She runs a project dedicated to studying Soviet and Estonian newsreels in 1922-1997 computationally. Earlier she has been studying the 19th century global news flows, circulation of fake historical narratives in the Russian language internet forum discussions, and the Cold War era transnational information circulation.

Short Papers

Chair: Mila Oiva

• AVAnnotate: Creating Scholarly Editions and Exhibits with IIIF and AV Archives

  Tanya Clement, the University of Texas at Austin (USA)

  Increased concern over media degradation and obsolescence combined with the decreasing cost of digital storage has led libraries, archives, and museums (LAMs) to digitize audiovisual (AV) materials for improved access and long-term preservation. Yet, improving preservation and access must go beyond digitization. While they are sometimes the only record of an event or an aural, visual, or performance tradition, AV digital artifacts remain underused and understudied. IIIF (International Image Interoperability Framework) is one standardized solution that LAMs have adopted to give users the ability to perform basic humanities methods including annotating, comparing, discovering, illustrating, referring, representing, and sampling images. In 2020, IIIF extended the IIIF API to accommodate rendering AV in a web browser. In response to the need for a workflow that supports IIIF manifest creation, collaborative editing, flexible modes of presentation, and permissions control, we developed the AVAnnotate project. AVAnnotate facilitates sharing annotations on AV archives through a sustainable workflow that leverages IIIF and simplifies the production of online AV projects that provide commentary and context around under-used and culturally sensitive AV collections. Existing projects include curricula for recorded interviews with jailed student protestors during the Civil Rights movement, a bilingual edition of Radio Venceremos programs, a documentary of events at the Furious Flower Poetry Center, as well as oral histories from the Syilx Okanagan Peoples. The AVAnnotate project builds on IIIF to address gaps in increasing engagement with archival AV and providing a solution for standardized annotation collaboration and presentation.

  — Tanya E. Clement is an Associate Professor in the Department of English, an affiliate faculty in the School of Information, and Director of the Initiative for Digital Humanities at the University of Texas at Austin. Her areas of research are modernist, textual, sound, and infrastructure studies as these concerns impact academic research, research libraries, and the creation of research tools and resources in Digital Humanities (DH). She leads High Performance Sound Technologies for Access and Scholarship (HiPSTAS) for the development and interrogation of socio-technical infrastructures to increase access and scholarship with audiovisual cultural heritage collections. AVAnnotate, a HiPSTAS project, is currently being funded by a Mellon foundation grant. Her current book project is Dissonant Records: Close Listening to Literary Archives, out with MIT Press in August 2024.

• Bipartite Frame Networks in the Analysis of Film: a Case Study Utilizing Commercial Computer Vision APIS

  Nabeel Siddiqui, Susquehanna University (USA)

  This paper overviews a novel methodology that enhances traditional count-based analysis by leveraging, what I call, bipartite frame networks. Bipartite frame networks consist of two types of nodes. The first node type represents the frames of a film while the second refers to the measurable elements within them (e.g., characters, objects). These nodes are then linked through edges that reflect their cooccurrence within a frame. By applying techniques from network science, bipartite frame networks make it possible for scholars to identify significant frame-to-element relationships, revealing patterns of composition and narrative structure that are not easily discernible through traditional count-based methods.

  — Nabeel Siddiqui is an Assistant Professor of Digital Media at Susquehanna University’s Communications Department where he also serves as Associate Director for their Center for Teaching and Learning. He specializes in data science, cultural analytics, the digital humanities, the history of information science, communication, new media rhetoric, and science and technology studies.

• Visualizing Rhythms Through Digital Annotations : Challenges and Issues in the Performing Arts

  Théo Heugebaert, Université Rennes 2 (France)

  The rhythm of a performance, as an aesthetic and temporal sensation, although easily perceptible at first glance, becomes, upon closer examination, a complex object to apprehend : despite the striking impressions it may leave, it does not leave traces other than those in memory. While audio or video recordings allow us to access a reproduction of movements, our interest in rhythm lies in what moves within the movement. Moreover, as we seek to understand the evolution of rhythms during the creative process, our focus shifts to the mutations of what moves within the movement. With the help of audio and video recordings annotations, we can precisely attempt to overcome these challenges by detecting new traces and indicative clues revealing the nature of rhythms and their mutations during the creative process. Here, we will present the method that led us to deduce two indicators from the concept of rhythm, which are exploitable through the annotation of audio and video recordings and allow us to visualize and analyze the rhythms and their mutations during the creative process : speaking rate and a typology of rhythms performed by the actors.

  — Under the supervision of Sophie Lucet and Clarisse Bardiot, Théo Heugebaert is a PhD candidate in theater studies at the University of Rennes 2. His involvement in dance, acting, and directing has gradually led him to focus on the question of rhythm. His studies aim to understand and visualize how rhythms develop during the creative process, particularly through an approach oriented towards the digital humanities. Among other roles, he has been an assistant director for Arthur Nauzyciel's mises en scènes since 2020.

• Intuitive Access to Oral History Video (The Pellaton Experience)

  Bérénice Serra and Léna Frei, Institute Digital Communication Environments (Switzerland)

  The Pellaton Experience, a specialized web portal for dance research, was developed by the Institute Digital Communication Environments (IDCE) at HGK Basel FHNW in collaboration with the SAPA Foundation (Swiss Archive of Performing Arts). The SAPA Foundation's archive houses interviews, dance notations, video recordings, and reviews, offering a comprehensive view of the Swiss dance scene. Recent emphasis on the "oral history" research method resulted in a substantial collection of audiovisual data, posing the challenge of making this unique content accessible to diverse audiences, leading to the creation of the Pellaton Experience project. The lecture will explore how the research project strategically presents archival materials from oral history interview series, aiming for a comprehensive understanding of the discussed topics. This effort was made possible through collaboration with an interdisciplinary team, including experts in video/media, art history, and IT (SAPA), as well as professionals in graphic design, project management, game design, and architecture (HGK Basel FHNW). Key research questions focus on user interface and user experience design, aiming to optimize scenarios like "expert search" and "uncoordinated browsing" to enhance the overall choreographic experience. The project strives to provide innovative solutions to improve accessibility and deepen the understanding of the rich history within the SAPA Foundation's archives. Consequently, we will delve into specific aspects of this interface while also establishing broader connections with themes such as digital archives, dance documentation, oral history, gesture in dance, and design.

  — Bérénice Serra is a media artist based in Zurich whose works focus on the notion of publication how content goes public in the digital age. In 2022, she published the results of her research project on Tactical Publishing in collaboration with socio-anthropologist Jean-Paul Fourmentraux and the director of Espace Multimedia Gantner, Valérie Perrin. She is currently a guest professor at the Institute Digital Communication Environments (IDCE), FHNW Basel.

  — Léna Frei works as a research assistant at the Institute Digital Communication Environments (IDCE), FHNW Basel. Her research interests are centered around the interplay between humans and technology, with a special focus on game design. Her fascination with computer games – both as a gamer, researcher, and designer – has been the starting point for various projects in academia, design, or interactive narration.

• The Linked Editing Academic Framework (LEAF) in the Multimodal Annotation Ecosystem

  Diane Jakacki, Bucknell University (USA), Susan Brown, University of Guelph (Canada), Michael Ilovan, University of Alberta (Canada) and Luciano Frizzera, Concordia University (Canada)

  The Linked Editing Academic Framework (LEAF) is a virtual research environment, a platform and suite of tools designed to support researchers working collaboratively in cultural heritage spaces, enabling them to undertake sophisticated editorial work in multiple media formats. Built with the Islandora repository framework, LEAF is based on linked data principles and interlinked with the larger LOD ecosystem, not only through its use of PIDs (Persistent Identifiers) but through a sophisticated toolset that allows novice as well as expert users to annotate cultural heritage materials at both meta- and granular-levels. The platform allows scholars to focus on their subject of study rather than to learn how to code, enabling researchers without extensive technical expertise to annotate content as part of their scholarly workflows. LEAF mobilizes some tools by directly integrating them into the platform; others are supported indirectly by either providing compatible outputs or supporting/retooling their outputs; and some core components are also modular so they can serve as stand-alone tools or be integrated in other systems.

  — Diane K. Jakacki is Digital Scholarship Coordinator and Associated Faculty in Comparative & Digital Humanities at Bucknell University (US). She is lead investigator on the Mellon Foundation-funded Liberal Arts Based Digital Editions Publishing Cooperative and a co-lead of the Linked Editing Academic Framework (LEAF); she participates in LINCS in that capacity, as well as a researcher on early modern London. Her research focuses on digital humanities scholarship and pedagogy, early modern British literature and drama, critical making, digital scholarly production and publication. In 2022-23 she was a Fulbright Canada Research Chair in Digital Humanities, modeling historical events encoded in TEI for development as Linked Open Data.

  — Susan Brown is Canada Research Chair in Collaborative Digital Scholarship and Professor of English at the University of Guelph. She engages from an intersectional feminist perspective with the use of semantic technologies for cultural scholarship through the Orlando Project in women’s literary history. Her critical infrastructure work explores how online systems for creating, enhancing, and sharing cultural knowledge can advance collaborative knowledge production, diversity and inclusivity, respectful data creation and dissemination, sustainable access to cultural scholarship, and research data management and preservation. She directs the multi-institutional Canadian Writing Research Collaboratory (CWRC) and the Linked Infrastructure for Networked Cultural Scholarship (LINCS). She founded with colleagues at Guelph an interdisciplinary major in Culture and Technology Studies and The Humanities Interdisciplinary Collaboration (THINC) Lab. She is the past-President (2022-23) of the governing board of the Alliance of Digital Humanities Organizations and of the Canadian Society for Digital Humanities (2013-2019).

Keynote: A Multimodal Turn?: Navigating AI Developments in Digital Humanities.

Melvin Wevers, University of Amsterdam (Netherlands)

Chair: Jacob Hart

Welcome

Annotations for Contextualization and Narratives

Chair: Nicola Carboni

• 3D Annotations as Multimodal Storytelling

  Øyvind Eide, Kai Michael Niebes, Nadjim Noori, Vyshantha Simha and Elisabeth Reuhl, University of Cologne (Germany)

  Kompakkt is a web-based software solution that allows 3D, 2D and other AV multimedia objects to be published, explored and annotated in a collaborative way in the browser. Through personalised and group level collections of 3D models, images, sounds and videos it enables a novel solution for gathering and generating artefact information. Kompakkt is an open source tool, originally developed at the Department for Digital Humanities at the University of Cologne, with a growing user base among academic and cultural institutions in Germany as well as beyond. While Kompakkt is a useful tool for many institutions and individual, the devel- opment is connected to ongoing research into the nature of annotations beyond textual ones, as well as the semiotics and mediality of 3D modelling. We are cur- rently working on the development of annotations of lines and areas, which opens up interesting questions about the meaning of such forms for 3D models. We are also investigating the implementation of measurements, which raise similar ques- tions, in addition to the scale of measuring, which is different in born digital and in digitised models. The presentation will introduce Kompakkt as a tool, as well as a research project, opening up the discussions mentioned above, linking it to the multidisciplinary theory behind the development of Kompakkt.

  —Øyvind Eide is a deputy professor in Digital Humanities (Historisch-Kulturwissenschaftliche Informationsverarbeitung) at the University of Cologne. He holds a PhD in Digital Humanities from King's College London (2012) which was funded by a grant from the The Research Council of Norway. He was been an employee in various positions at The University of Oslo from 1995 to 2013, most recently as a Senior Analyst at The Unit for Digital Documentation. From 2013 to 2015 he was s a Lecturer and research associate with the Chair of Digital Humanities at The University of Passau.

• Coding the Encoder: Situating Subjective and Contextual Aspects in High-Level Image AnnotationsDelfina Sol Martinez Pandiana, Università di Bologna (Italy)

  The talk explores the assignment of subjective high-level annotations to visual data in computer vision pipelines. It addresses how this annotated data is utilized to train AI models, highlighting the common lack of situational context, and the biases introduced. The talk describes the SituAnnotate methodology, which extends the data annotation process to include rich contextual information, such as sources, and financial/temporal/geographical contexts. This approach not only provides a comprehensive understanding of the situational grounding of annotations but also offers insights to mitigate biases, enhance model understanding, and empower AI practitioners in curating datasets aligned with specific criteria. The talk serves as a bridge between high-level semantic annotations and the imperative to reimagine annotation for cultural data, emphasizing the crucial role of situating context and socio-cultural factors in responsible AI system development.

  — Delfina Sol Martinez Pandiana is a postdoctoral researcher at the Centrum Wiskunde & Informatica (Amsterdam), specializing in high-level multimodal sensemaking and misinformation detection. Their PhD in Computer Science focused on automatically detecting abstract social concepts in images. They hold a B.A. in Human Evolutionary Biology from Harvard and an M.A. in Digital Humanities from the University of Bologna. Delfina combines cognitive research, semantic technologies, and computer vision to challenge assumed dichotomies in areas like computational propaganda and cultural representation.

• VR & AR Prototypes for Multi-sensory and Haptic Forms of Documentation and Archiving of Digital Art (LeFo Project)

  Marie-Claude Poulin, University of Applied Arts Vienna (Austria)

  The presentation introduces the development of virtual and augmented reality prototypes designated as a "Virtual Meta-Set" for three existing digital art installation/performance works and their processual artifacts. These are designed to simulate and reconstitute the artworks and their genesis history in a virtual environment that allows travel between them and their documentary and annotative material. The goal is to investigate the constitution of a 3D archival user experience in which the metadata of each creation process are accessible and can be cross-referenced at any time.The database contains “classical” and atypical documentary components such as lines of code and user data, distributed and staged in the same multi-layered virtual landscape. A portable controller allows users to access hyperlinks, manipulate virtual assets, and reorganize the space. Through their actions, they explore the environment, and a tracking of their movements, combined with their manual and verbal commands, turns the virtual space in a walkable, haptic experience.Bringing distinct artworks to converge in this “Virtual Meta-Set” is not simply a technical feat, it is an artistic mission which, through simulation and immersive reconstitution, generates a new form of re-enactment. The challenge is also to test whether this type of epistemological-artistic environment, in which the immersion factor is an integral part, can lead to a new kind of meta-reflection on creative processes.

  — Marie-Claude Poulin is Senior Artist at the fulldomeXR Lab/University of Applied Arts Vienna and founder and co-director of the digital performance group kondition pluriel. Her research combines choreographic strategies, motion analysis/notation, digital visualization, programming rules, and modes of user interaction in XR and AI environments. She has performed for Meg Stuart and Benoit Lachambre and presented her work at venues such as ZKM, ISEA, EMPAC, SAT, CYNETart, ICA and Ars Electronica.

Performing and Visual Arts Documentation and Analysis

Chair: Clarisse Bardiot

• TV as Cultural Heritage: Exploring Multimodal Approaches

  Lauren Tilton, University of Richmond (USA)

  The talk will explore text, image, and sound analysis of TV sitcoms. As syndicated TV becomes understood as a form of cultural heritage, institutions and researchers are looking more closely at how to access and study these popular culture forms through computational methods. We will look at some initial multimodal explorations and discuss future possibilities.

  — Lauren Tilton is the E. Claiborne Robins Professor of Liberal Arts and Digital Humanities at the University of Richmond. Her latest book Distant Viewing: Computational Exploration of Digital Images with Taylor Arnold is now open access from the MIT Press.

• Multimodal Video Annotations as Metadata for Performing Arts Documentation

  Carla Fernandes, NOVA University Lisbon (Portugal)
  

  This talk will focus on the affordance of idiosyncratic video annotations to be understood as indispensable metadata when digitally documenting performing arts materials or visible intangible heritage in general.Annotating and tagging directly over video in real-­time have started to be mostly a personal practice used by those involved in the footage or analytical processes of video documentation for dance, but they have not yet been perceived as complementary metadata by cultural heritage (CH) archivists. Based on previous experience with TKB (a Transmedia Knowledge-Base for performing arts), we have continued to research on the multimodal annotation practices as part of the creative process in performing arts. The innovation at that time (2010) was to accept individual tagging as indexation cues and means of interaction amongst the artists who were self-­curating their works, therefore generating dynamic and idiosyncratic “archives of processes”. During TKB, several artists expressed the need for a handy digital tool to assist them in rehearsal periods where they could take notes (drawings with touch pen, text, marks, links and sound) on what was being filmed.Under subsequent EU projects, we developed a friendly web-­based tool to work as a video annotator in real-­time, i.e., a digital notebook to replace paper notes. “MotionNotes” is now freely available on the web and is being used by choreographers, ethnographers and educators in general. Most of the users report that their notes become inseparable of the video footage at the end of the documentation process, and suggest imaginative ways to embed those personal annotations in future presentations of their work, as well as to include them as an integral part of the metadata when the work is digitally archived.to discuss with the I I'd like to discuss with audience the future usability of idiosyncratic multimodal annotations as great potential for new approaches to documenting, archiving and sharing intangible cultural heritage video content.

  — Carla Montez Fernandes is currently Principal Investigator and Professor at Universidade Nova de Lisboa, FCSH, where she is Head of the ‘BlackBox – Arts&Cognition Lab’ (funded by the European Research Council) since 2014. She directs the Performance & Cognition Research Group at ICNOVA since 2021. She has been designing and leading interdisciplinary R&D projects for over 12 years with a focus on TKB and BlackBox, and has participated in several EU-funded projects (Inside Movement Knowledge; Labo21; MotionBank; Europeana-Spaces). More recently she has been a full partner in CultureMoves, Weave and the Horizon2020 T-Factor project.  She supervises numerous doctoral and master theses at FCSH and at other national and international faculties. At present her research focus is in the intersection of Dance Data, Cognition, and Multimodal Communication, particularly concerning the analysis of bodily behaviour in creative and collaborative settings. With a strong interest in collaborative digital archives for the analysis/documentation of contemporary dance and Intangible Heritage in general , she is author of book chapters and numerous papers in international journals and conferences in the fields of Multimodal Communication, Performing arts, Intangible Heritage preservation and digital media. +INFO: https://blackbox.fcsh.unl.pt/home.html

• Curating Born-Digital Archives at the National Library of France: the Amos Gitai collection’s Case Study

  Rime Touil, Bibliothèque nationale de France (France)

  In 2018, the Performing Arts Department of the National Library of France received an exceptional donation from director Amos Gitaï comprising his entire physical and digital archives surrounding his film Rabin, The Last Day (2015). Representing almost 19 TB of data including text files, photographs, sound files, web pages and most of all, videos, covering all stages of the film's creation from the script writing to its final editing, this collection poses unprecedented scientific and technical challenges. The massive number of documents and the wide variety of formats they represent, as well as the skills required to handle them, raised the question of the processes to use to identify these archives, to make them available to the public and, ultimately, to preserve them on the long term, bringing into light the tension that can exist between preservation requirements and the need for easily accessible archives. To accommodate this kind of complex digital collection, the BnF reflected on new tools specifically designed for born-digital archives, one of them being an application dedicated to the curating of born-digital archives, called TriNum. Conceived to enable the classifying, the annotation and the creation of preservation units, as well as to manage and trace the transformations carried out on these documents prior to their preservation, this interface keeps evolving to meet the challenges raised by the numerous born-digital collections received by the BnF these past few years. This lecture intends to present the state of the art on digital curating at the BnF while displaying the methodology, solutions and strategies implemented during the processing of the Amos Gitai collection.

  — Rime Touil is a digital curator at the Performing Arts department of the National Library of France (BnF). She is in charge of the coordination of digitalization programs and digital projects the department is currently involved in, as well as taking part in the development of a processing workflow for born-digital documents at the BnF. She is also taking part in working groups on digital archives and formats.

Designing Tools and Workflows by and for Researchers

Chair: Servanne Monjour

• Two Historians' Relationship with Sources in the Digital Age

  Luca Federico Cerra and Sean Takats, Université du Luxembourg (Luxembourg)

  This paper examines the annotation practices behind DHARPA (the Digital History Advanced Research Accelerator project) and its innovative data orchestration tool, kiara. By focusing on the importance of annotation across various stages of research, kiara aims to enhance transparency and traceability in humanities research. Kiara’s design draws on the wide range of research experiences of the project’s team members and its collaborators. Here we explore two specific research problems experienced by this paper’s authors: Takats’s research on colonial medical practitioners involves the identification of geographic patterns of expert knowledge production and circulation; Cerra seeks to highlight the interactions between craftsmen and administrators in Revolutionary France, thus, uncovering a network of exchanges involving family, neighborhood, or professional ties. In both cases, our research depends not just on the creation, analysis, and visualization of data drawn from archival sources and finding aids, but also on the careful supervision and documentation of the decisions made along the way. Kiara aims to assist this process through a combination of automated and intentional annotation. Our paper considers the complexity of developing software which promotes critical reflection rather than simply racing to a solution.

  — Luca Federico CERRA is a doctoral researcher in modern history at the C²DH. He is writing a thesis in cotutelle with the University of Namur on the causes and consequences of the abolition of guilds in Luxembourg at the end of the 18th century. Specialized in economic and social history, as well as the history of emotions and mentalities, he is also involved in the DHARPA digital humanities project, and is a member of the “aRaiRe”cluster.

  — Sean TAKATS leads the Digital History Advanced Research Projects Accelerator (DHARPA) at the Centre luxembourgeois d'histoire contemporaine et numérique (C²DH). He is currently Professor of History and FNR PEARL Chair at the University of Luxembourg. Until 2019 he was Associate Professor at George Mason University and Director of Research at the Roy Rosenzweig Center for History and New Media. Director of Zotero and Tropy, he is president of the Corporation for Digital Scholarship.

• AI Toolkits for the Social Sciences and Humanities: A Closer Look at ModOAP, BaOIA, EyCon, and PictorIA

  Julien Schuh, Université Paris Nanterre (France)

  The exploration of vast digital archives, now increasingly accessible due to the digitization policies of numerous heritage institutions, poses a unique challenge. This communication offers an insight into a series of projects deploying AI toolkits to transform the management, analysis, and valorization of cultural heritage. Bridging the gap between historians, linguists, engineers, and conservators, these initiatives offer methods for complex image segmentation, pattern recognition, and automated text analysis. These projects utilize accessible data via APIs from various institutions for model training and fine-tuning, focusing on creating specialized corpora and refining AI methodologies. The collective outcomes of these projects have led to the inception of a national consortium project focused on exploring extensive image corpora within heritage collections and research archives.

  — Julien Schuh (Senior Lecturer in French Literature and Digital Humanities, Université Paris Nanterre, Deputy Director of the Maison des Sciences de l'Homme Mondes); main coordinator of the pictorIA consortium, which aims to promote interdisciplinary research on automatic pattern recognition in the social sciences.

• The Linked Infrastructure for Networked Cultural Scholarship (LINCS): Bridging the Research/Heritage Collection Gap

  Susan Brown and Kim Martin, University of Guelph (Canada)

  Culture operates increasingly as data, as does scholarship. While GLAM institutions create highly standardized and generalized metadata for their collections, scholars embed deep knowledge of those items within collections through their research. Web annotations, via the Web Annotation Data Model (Sanderson et al., 2017), are one way to bring together the data about these items and point to the expert contextualization, scholarly debates, and situated knowledge created about them. Linked Infrastructure for Networked Cultural Scholarship (LINCS) is an infrastructure to allow scholars to transform or create linked open data (LOD) about cultural materials. LINCS is working with GLAM institutions in Canada to build relationships between people; between data; between people and data, cultural organizations, and machines. The LINCS team chose the CIDOC-CRM and the WADM as complementary ontologies (Application Profile). CIDOC is increasingly used by cultural heritage organizations, but the WADM offers a flexible W3C standard for linking data to various resources on the web, including and beyond projects related to LINCS (Canning et al., 2022). In this respect, LINCS is aligned with Europeana, which has adopted elements of both CIDOC-CRM and the WADM in their Annotations API (https://pro.europeana.eu/page/annotations). This paper will reflect on the implications, benefits, and challenges that LINCS has encountered to date from creating and publishing data that combines these two ontological structures.

  — Susan Brown is Canada Research Chair in Collaborative Digital Scholarship and Professor of English at the University of Guelph. She engages from an intersectional feminist perspective with the use of semantic technologies for cultural scholarship through the Orlando Project in women’s literary history. Her critical infrastructure work explores how online systems for creating, enhancing, and sharing cultural knowledge can advance collaborative knowledge production, diversity and inclusivity, respectful data creation and dissemination, sustainable access to cultural scholarship, and research data management and preservation. She directs the multi-institutional Canadian Writing Research Collaboratory (CWRC) and the Linked Infrastructure for Networked Cultural Scholarship (LINCS). She founded with colleagues at Guelph an interdisciplinary major in Culture and Technology Studies and The Humanities Interdisciplinary Collaboration (THINC) Lab. She is the past-President (2022-23) of the governing board of the Alliance of Digital Humanities Organizations and of the Canadian Society for Digital Humanities (2013-2019).

  — Kim Martin is an Assistant Professor in History at the University of Guelph. In this position, she works to create community around digital humanities by organizing workshops, speaker series, and hands-on events (like Programming Historian meet-ups and Wikipedia Edit-a-thons). She is also the Associate Director of The Humanities Interdisciplinary Collaboration Lab (THINC Lab), a research space in the McLaughlin Library that provides space and expertise for graduate students and faculty working on digital, interdisciplinary projects.

Perspectives (round table)

Chair: Michael Sinatra, Université de Montréal (Canada)

• Nicolas Larrousse, Huma-Num (France)

• Philippe Effantin, Ouest-Valorisation (France)

• Arthur Lezer, Le Lab, INA (France)

• Susan Brown, University of Guelph (Canada)

• Clarisse Bardiot, Université Rennes 2 (France)