Introduction: Special Issue on AudioVisual Data in DH

Towards AV DH

Many scholars have repeatedly demonstrated how expanding our areas of inquiry builds new routes for the field ^{1 2}. Yet, often those interested in working with AV data have found themselves swimming upstream. Text and word culture have enjoyed a dominant position in DH, bolstered by factors such as the prominence of text analysis and the form of academic journals ^{3 4}.⁵ Replicating the larger structures of higher education, racialized and gendered beliefs about what counts as “rigorous” scholarship that marginalize fields such as cultural studies and visual culture studies also permeate DH and further explain why text (analysis) has enjoyed a privileged position along with their related academic fields ⁶.⁷ However, exciting developments are continuing to support and amplify the work of AV data in DH.

One of those developments is shifts in technology. The ability for computers to create, “read”, and store AV data followed by advances in areas such as machine learning have augmented computational image and sound analysis. Pioneering approaches such as cinemetrics that once relied on hand coding and text-based annotations can now be automated. Within DH, this has led to new theories and methods such as cultural analytics, distant listening, and distant viewing ^{8 9 10}. These developments have led scholars such as Melvin Wevers and Thomas Smits ¹¹ to argue that DH is seeing a “visual, digital turn”. Meanwhile, Mary Caton Lingold, Darren Mueller and Whitney Trettien ¹², in the edited volume Digital Sounds Studies , demonstrate how pairing digital tools with “interpretive practices that always attend to the human” forges new paths for sound studies and DH.

Another critical development is digital access to AV materials. As DH scholars increasingly think of their sources as data, they have benefited from large-scale digitization of audiovisual collections.¹³ At the forefront over the last three decades have been governmental and GLAM (gallery, libraries, archives, museums) institutions around the world to digitize collections, whose initiatives have often been spurred by deteriorating physical collections.¹⁴ With millions of items digitized, platforms were funded by cultural, government, and scientific organisations for providing access to audiovisual heritage collections alongside the emergence of platforms by for-profit multinational corporations, all of which enabled the circulation of digitized and digital-born materials online.¹⁵ While issues such as copyright and funding still loom large, DH scholars have greatly benefited from the significant investment in digitization over the last 30 years.

Finally, we turn to institutional developments.¹⁶ Along with conferences specifically dedicated to DH and media [e.g. Transformations Conference], new journals have developed such as the International Journal of Digital Art History and Journal of Cultural Analytics featuring scholarship at the intersection of AV research and DH.¹⁷ In order to further amplify AV work in DH, colleagues worked to develop a Special Interest Group (SIG) within the Alliance of Digital Humanities Organizations (ADHO). The AudioVisual in DH (AVinDH) SIG was proposed after a successful workshop on how to integrate the Audiovisual in the Digital Humanities in Lausanne at DH2014, and formalized a year later at DH2015 in Melbourne. One result of the SIG’s work is this special issue.¹⁸

New pathways are bringing about exciting opportunities in DH as exemplified by the articles in this issue. They model how AV research can be the subject of analysis (e.g. film) or result of analysis (e.g. podcast). They highlight less visible humanities disciplines in the DH, model the importance of interdisciplinary collaboration across institutional structures, demonstrate how cutting edge scholarship comes from a plethora of positions, and offer new questions that the field is only beginning to grapple with. The contributors’ model paths for constructing entry points, building bridges, or adding intersections for engaging with audiovisual in the digital humanities. Amplifying the work of scholars with a range of disciplinary, institutional, and political commitments, the special issue constructs a more capacious configuration of DH.

Contributions

The special issue is organized into five sections. The first focuses on annotation of AV material as method and theory. The second explores analyzing (meta)data, which often includes annotation, to build and analyze AV corpora. The third focuses on creative and liberatory ways to remix AV (meta)data as a way to innovate pedagogically and methodologically while furthering discipline specific interventions. The fourth dives further into computational methods, particularly machine learning, in turn demonstrating how DH is reconfiguring these methods and ways of knowing to answer humanities questions. The special issue ends with a focus on how AV forms such as podcasts and film can also be the form of scholarly knowledge in the field, highlighting how form and argument can be mutually constituted.

Next, we turn to the contributions that comprise each section. The first explores annotation as a powerful way to add context and analyze AV data. A particularly prominent area of such work has been in film studies. Therefore, the first three articles offer a snapshot of different approaches and tools for film annotation. Cooper, Nascimento and Francis present their KinoLab and discuss the opportunities and challenges of Omeka for narrative film language analysis, including the challenges related to copyrights. They argue for a universally accepted data model for film analysis. We then turn to a new annotation platform called Mediate built by a team at the University of Rochester. Burges et al. discuss how they use the annotation tool in the classroom in three different disciplines - film and media studies, music history, and linguistics - and introduce the concept of “audiovisualities” as a theoretical frame for understanding remediation through annotation. Next, Williams and Bell discuss how the Media Ecology Project is conceived as a virtuous cycle and incubator working to increase access and discovery of moving images, with a particular focus on tools such as the Semantic Annotation Tool. Zooming out, Clement and Fisher theorize a new approach to annotation. They bring together sound and literary studies to introduce the concept of “audiation” as a framework for audiated annotations that increase access and discovery.

The next section focuses on how (meta)data can open up analytical possibilities. Using metadata to reunite AV materials, Sapienza et al. describe the process of reuniting radio and text files virtually that belong together but ended up at different institutions. After discussing why audiovisual collections in general are heavily under described, they describe how virtual reunification and integrated access was realized through the use of linked data, minimal computing, and synced transcripts. Next, Hoyt et al. discuss the analytical possibilities afforded by metadata. Focused on podcasting, they discuss three different methods for studying RSS feeds and podcast metadata, and point at the specificities of methods for born-digital media vis-à-vis digitized media. Carrivé et al. then focus on the first development phase of their ANTRACT project for the transdisciplinary content analysis of 1262 newsreels containing more than 20,000 French news reports. They discuss how they dealt with the project’s main technological challenge to process data and build tools to familiarize historians with the automated research of large audiovisual corpora in order to then use the data to pursue inquiry about Les Actualités Françaises news reports. Finally, Gienapp et al. show what can be done when data is brought together from different sources to analyze music collaboration. They demonstrate how network analysis can reveal the contours of collaboration among musicians.

In the third section, the authors creatively (re)mix AV data with a focus on audio data. Using audio data, Tyechia and Carrera demonstrate how centering Afrofuturism in DH pedagogy through mixtapes can not only realize the goals of an undergraduate composition course but realize a liberatory DH praxis. Bonnett et al. combine data art, landscape art and augmented reality in their DataScapes Project. Departing from the premise that data can be translated into visual and sonic forms, they use protein data and texts from the bible, turn them into sequences, and translate these into visualisations and compositions. Kramer then asks what if we listened to images. Building off of previous work on “image sonification”, he argues that transforming the visual into audio opens up new ways of seeing and hearing the past. Next, Have and Enevoldsen demonstrate how toggling from close to distant listening offers insights about the longue durée of Danish radio content by scrutinizing what is audible with the human ear and searching for patterns using AI. Next, we turn to work that makes field specific interventions. Martin constructs a new path for listening to gentrification in Washington DC by combining ethnography, passive acoustic recording, and computational sound studies. The work also demonstrates how centering Black DH offers new ways to understand the relationship between embodied and computational audio analysis in DH, in turn forging new liberatory possibilities for the field.

The next section continues with the application and reconfiguration of computational techniques, particularly machine learning, to conduct data analysis on large collections of AV data. Looking at a large collection of artwork showcasing musical instruments, Sabatelli et al. introduce the usage of computer vision techniques to automatically locate musical instruments in images. They investigate the algorithmic properties of their analysis and show how it leads to innovative scholarship in music iconography. A paper by Lupker and Turkel illustrates the potential of investigating novel intersections between research in the humanities by using musical theory to guide the training and usage of machine-learning algorithms applied to a large corpus of digitized music. A born-digital collection of K-pop dance videos hosted on YouTube is analyzed using state-of-the-art computer vision techniques in a paper by Broadwell and Tangherlini. Their article develops a typography for describing and analyzing poses and choreography to facilitate the data-driven analysis of time-based media. Oyallon-Koloski et al. present a different approach to the study of movement in space by showing how motion capture technology can be used within film, dance, and movement studies. As with the Lupker and Turkel article, Oyallon-Koloski et al. illustrate the novel integration of theoretical frames from the humanities – in their case, Laban Movement Analysis and Bartenieff Fundamentals and Movement Studies – and the usage of computational techniques. In another take on the detection of bodies moving in space, Fragkiadakis et al. introduce an automatic system and taxonomy for tagging and describing digital videos of sign-language usage. Together, these articles illustrate the potential for work in AV DH to infuse machine learning with analytical commitments from the humanities.

Finally, we turn to articles that use DH to push the boundaries of form for scholarly knowledge. In order to demonstrate how the podcast format expands our definitions of text, Edwards and Hershkowitz reveal how podcasts can realize intersectional feminist approaches to DH. Along with demonstrating and discussing the creation of the Books Aren’t Dead (BAD) podcast in the article, they discuss the process in a podcast for this special issue. Kim then explores how motion caption and virtual reality can be used to record and visualize movement histories as a form of cultural heritage preservation. Through these forms, one can then use visual storytelling, she argues, to demonstrate how movement, dance, and ritual cannot be separated from a person’s personal narration of the experience. Finally, Mittel’s contribution showcases twenty audiovisual deformations of the classic musical “Singin in the Rain” in still image, GIF, and video formats. The essay considers both what each new deformation reveals about the film and the way we engage with the by algorithmic practices derived object as a product of the “deformed humanities.”

The invitation of the authors in this special issue to think about the relationship between form and argument is one we also embraced. The publication needs, some of which weren’t possible, of our special issue pushed the boundaries of the form and format of DHQ as a journal that is catered for linear reading of articles as written texts in XML. As a result, the articles in this special issue include 5 sound files, 52 embedded videos, and 176 gifs and images. These AV components are key elements of the authors’ argumentation. The special issue attempts to more closely mirror how scholars of AV materials in DH actually produce and create scholarship. In this context we take as a guide other pioneering initiatives in this field such as Scalar, VIEW journal, Audiovisualcy, and [in]transition that question the relation between the affordances of a publication platform and the interactivity and multimodality of scholarship that increasingly embraces creative modes of production. We hope that these incremental steps within DHQ can forge exciting new possibilities for the field.

Finally, our decision to partner together to co-edit was driven by features of AV work in DH. First, our own areas of expertise – statistics and digital images, history and audio, american studies and photography, and media studies, television & film – reflect a range of audio and visual scholarship that animates DH. Second, the inclusion of a colleague housed in a Math & Computer Science department, Tayor Arnold, demonstrates how digital humanities scholarship often requires working with and giving proper credit to experts trained in computational fields. Third, we wanted to build collaborations across geopolitical boundaries and languages that might help us think critically and beyond the particular configurations of DH that shape our local, regional, or national context. We recognize that our positionalities as White able-bodied scholars living in the global west and north also brings limits. As a part of our efforts, we paid special attention to circulating the CFP beyond our immediate DH circles with particular attention to reaching beyond the US and Western Europe. However, there is still more work to do. Yet, we do hope that the issue in aggregate reveals how thinking across disciplinary, cultural, and spatial boundaries enables a more capacious configuration of the field than currently articulated.

Conclusion

As we look toward the future, we are enthused about the possibilities and realistic about the challenges. Along with the work featured in this special issue, areas such as 3D, AR/VR, and game studies are forging exciting paths. As disciplines (albeit slowly) adopt more capacious guidelines for what counts, forms of scholarship such as films, multimodal digital projects, podcasts, and software are receiving well overdue credit. Because of the teamwork and expertise often required to access and work with AV data, this area of DH also pushes us to work across ossified divisions such as the “Humanities” and “Sciences”, “faculty” and “staff”, and “university” and “cultural institution” in ways that can help us realize a more collaborative, equitable, and generous configuration of the field.¹⁹

At the same time, challenges remain. There are major obstacles to working with AV. For example, digitized images have significantly larger file sizes than textual data making them hard to transfer and process even in light of recent technological advances ²⁰. This makes computational analysis of large collections of digitized visual materials difficult for institutions that do not have access to extensive computational resources. Audiovisual materials are also often subject to varying degrees of copyright and access restrictions, dictated often by large multimedia producers ²¹. This makes it relatively difficult to work with certain collections, such as television news programs and feature films, and risks limiting the kinds of work with which digital humanists can work. Even when we do have access, the scale of AV data is growing rapidly, particularly given the rise of born digital AV content, and with this comes implications for how and who is positioned to analyze these materials. Existing audiovisual archives are heavily skewed towards European- and U.S.-centric collections. As we work through these challenges and opportunities, we need to continue to listen and engage with the cautions and critiques about computation and algorithms from scholars such as dana boyd and Kate Crawford ²², Ruha Benjamin ²³, Jessica Marie Johnson ²⁴, Catherine d’Ignazio and Lauren Klein ²⁵, and Safiya U. Noble ²⁶.

Finally, we want to thank the contributors, reviewers, and DHQ, specifically Managing Editor Cassandra Cloutier, for their work. Even under what were once “normal” conditions, writing an essay for publication is demanding. The challenges quickly mounted as authors revised amidst a global pandemic that disrupted everyone’s daily lives and affected communities unequally due to structural inequalities. As authors and our colleagues at DHQ tried to balance caregiving, jobs, and their own health, among other duties, they still carved out time to make this issue possible. This is quite an achievement, and for which we are grateful. Finally, we want to leave with an invitation. We encourage readers interested in continuing to further engage with AVinDH to join the SIG. We look forward to all that lies ahead.