Sentiment Analysis in Literary Studies. A Critical Survey

1. Introduction

After years of disinterest and neglect, Sentiment Analysis (SA) has recently become one of the most discussed topics in computational literary studies. Generally known as the field of study that analyzes “people’s opinions, sentiments, appraisals, attitudes, and emotions towards entities and their attributes” ¹, SA aims at accomplishing an automated extraction of the emotional content of text by converting it into machine-readable information, such as numbers and discrete labels (e.g., positive vs. negative), which can be then analyzed statistically or visualized via plots and graphs. It is precisely after June 2014, when Matthew Jockers published a series of though-provoking posts on his personal blog ^{2 3 4}, that this computational technique started to acquire a growing relevance in DH research. To date, SA has been used in multiple studies, ranging from the identification of the basic shapes of stories cf. ^{4 5}, to the large-scale investigation of children’s literature ⁶ and online reading communities ⁷, with applications on both contemporary and historical languages ⁸. This success is paralleled — even if without a causal relation — by the so-called affective turn ⁹ in literary studies ¹⁰, through which emotions regained a key role in narrative theory and reader response studies, after the neglect — and even the patent opposition — of both structuralist and post-structuralist studies.

However, as it happens with all vanguards and new trends, SA is currently experiencing its peak of theoretical and methodological instability as well. While research in a field like stylometry (the computational analysis of style) has already reached some of the highest levels in terms of scientificity and theoretical awareness,¹¹ the same cannot be said with SA, where most tools and methods still lack a validation, connections with literary theory are frail or disputable, and an organizational effort of the research community (such as that of the Special Interest Group “Digital Literary Stylistics” , or of the Federation of Stylometry Labs) is still lacking.

A first, extensive survey of SA for computational literary studies was proposed by ¹², who distinguished five main areas of application:
Classification of literary texts in terms of the emotions they convey Genre and story-type classification Modeling sentiments and emotions in texts from previous centuries Character network analysis based on emotional relations Miscellaneous and more general applications

However, at the end of their survey, Kim and Klinger notice how “the methods of sentiment analysis used by some of the DH scholars nowadays have gone or are almost extinct among computational linguists” ¹². There is a clear gap in terms of methodology that needs to be filled (or, at least, directly addressed), if research wants to move further. This necessity is also accompanied by a more general need to understand the internal logics of tools that are frequently used in a purely goal-oriented research. In other terms, a criticism of the tools and methods currently adopted in SA is as necessary as a free exploration of its potential. Without denying the usefulness of exploratory — or even “deformative” ¹³ — research, a critical awareness of shortfalls and limitations in distant reading should constitute the necessary groundwork for its fruitful application.

With this article, I will attempt a comprehensive criticism of SA tools in literary criticism by combining two approaches. First, as already done by ¹⁴ for stylometry and by ¹⁵ for topic modeling, I will conduct a theoretical inquiry on the possible correspondences and inconsistencies between the concepts of literary theory and the computational techniques of SA. This, in order to provide a solution to a basic question of modeling (cf. ^{16 17 18}): to what extent can SA model literary phenomena and thus make possible an operationalization (cf. ^{19 20}) of the fundamental questions of literary studies? Main distinction will be in where the emotions reside: if in the text itself (a perspective generally preferred by narratological approaches) or in its readers (main focus of reader-response studies). Second, by narrowing the perspective on some of the most widely used SA tools in DH, this article will perform an analysis of the technical limitations of such tools, in order to heighten awareness of the risks implicit in any research that bases itself uncritically on their outcomes.

2. Theoretical criticism

2.1 Towards an affective narratology

Examining the taxonomy by ¹², it becomes immediately evident that the strongest connection between literary theory and SA takes place in the field of narratology. The classification of genres and story types (application 2), in particular, depends primarily on the structure of narratives. ²¹ has also demonstrated how network analysis (application 4) can prove a powerful approach for the study of plot, while ²² have chosen the narratological device of the happy ending to classify their selection of literary texts (application 1). While this does not exclude that SA can also be fruitfully applied to the study of different subjects with different approaches (such as poetry in a neurocognitive perspective, as done by ²³), the field of narratology and the subject of the novel are clearly the most dominant,²⁴ with a distinctive interest towards the classification of literary texts (shared by both applications 1 and 2). It is not by chance, then, that Jockers decided to call his SA tool Syuzhet ³, with a direct reference to the distinction made by Russian formalists between the fabula and the plot (also known as syuzhet, récit, or intreccio).

However, when looking for the actual location where this connection takes place, the model reveals itself as apparently faulty. Both ⁴ and ⁵ used SA for demonstrating that the story arcs of the entire (Western) literary production are dominated by six basic shapes.²⁵ This conclusion sparkled a lively debate — and not a few criticisms (see ²⁶) — , but it was based on a very problematic assumption. In fact, these basic shapes were obtained by tracking the evolution of positive vs. negative emotions throughout thousands of stories. A precedent for such an approach was found in a lecture by Kurt Vonnegut, based on a (rejected) Master’s thesis and circulated in the form of a mock-academic pièce (now available on YouTube). Some connections in narratology might be found, like in the various proposals for a “generative grammar of narrative” ²⁷, which already stimulated applications in the field of artificial intelligence — but just for creative purposes ²⁸ — , or in the structuralist assumption that all stories can be reconducted to a universal model ²⁹ — but just when focusing on the fabula. Antecedents of the concept of story arc can even be found in Gustav Freytag’s pyramid of the dramatic action ³⁰ and in Northrop Frye’s “U” and an “inverted U” shapes ³¹, representing the archetypical structures of comedy and tragedy. However, when we look into the most established theorizations of narrative form, such as the still-fundamental proposal by ³² or the most recent reformulations by ³³, almost no trace of emotions can be found. Aspects like focalization, space, time, and characters constitute a story through their manipulation and interaction, while affect is nothing but a device used to engage the reader, with just secondary and indirect consequences on the structure of the narrative. In conclusion, the question becomes unavoidable: is the SA approach actually able to develop a formal model of the phenomenon which, traditionally, has been known by literary scholars as the plot?

A series of recent studies, produced in the wake of the already-mentioned affective turn, help mitigating a straightforwardly negative answer. Patrick Colm Hogan was the first to introduce the concept of “affective narratology” ³⁴. His goal was clear and apparently in line with the approaches by Jockers and Reagan: starting from the acknowledgement that “narratological treatments of emotion have on the whole been relatively undeveloped” ³⁴, he aimed at highlighting how “emotion systems define the standard features of all stories, as well as cross-culturally recurring clusters of features in universal genres” ³⁴. This statement could easily be adopted as an epigraph for many SA literary studies. However, when reaching the core of Hogan’s proposal, the correspondences become much more problematic. In particular, Hogan defines a nested system, where works are composed by stories (i.e., by many stories intertwined together), stories by episodes, and episodes by events and incidents, all governed by emotional principles. One of the most important principles in this system is normalcy: in fact, “an episode begins with a shift away from normalcy [and] ends when there is a return to normalcy. […] If normalcy is restored in a more enduring way, we have not just an episode, but a story” ³⁴. This poses a fundamental problem for computational modeling, because the basic unit of measurement is not uniform (the pages of a book or the words in a sentence, generally taken as reference points by SA tools), but depends on the development of the story itself (episodes can be closed in a few sentences, or develop through multiple pages). In addition, normalcy is not determined by a simple positive vs. negative emotional status, but by more complex emotion systems (e.g., attachment and sexual desire in the romantic genre), which take shape not in the more general context of the story, but with reference to the goals of single characters. The combination of these elements alone makes the emotional analysis of stories much more complex and sophisticated than a simple tracking of the emotional levels through the pages of a book, involving for example the issues of focalization (as the nature of a sentiment inevitably depends on the chosen perspective), style (if we follow its affective-aesthetic interpretation, cf. ¹⁴), symbolism, and many others.

It should be noted, however, that research in computational literary studies is currently paving the way for solving most of these problems. For example, the identification of scenes in narratives (and thus of Hogan’s episodes) is at the center of the effort by ³⁵, who already highlighted its relevance for narratological studies. ³⁶ produced their story arcs by distinguishing Plutchik’s eight basic emotions (joy, trust, fear, surprise, sadness, anticipation, anger, and disgust) and thus moved beyond the flattening distinction between positive and negative sentiments. Finally, all the studies collected by ¹² under the “character network analysis” label focus on the emotions of characters. What is currently missing, is a fruitful integration of all these approaches in a coherent framework.

In her recent contribution on the subject, ³⁷ compares the results of more than thirty-six different computational methods in the creation of emotional arcs (a term which she prefers to plot arcs, as it indicates “an underlying sentiment structure that occurs even when very little happens plot-wise” ³⁷). While strengthening the approach introduced by Jockers and providing multiple arguments to confirm its usefulness in literary studies, Elkins consciously decides not to face the more complex narratological issues described here, thus still leaving unrealized a true operationalization of Hogan’s narrative theory.

2.2 From narratology to reader response

Hogan’s is not the only proposal for an inclusion of affect into narrative science. In her sophisticated contribution, ³⁸ takes a distance from Hogan, by distinguishing affect from emotions. Following the Deleuzian interpretation, affect can be intended as an “(asubjective, asymbolic) intensity ” ³⁸, which resists any formalization or reduction to universal features. The interesting aspect of Breger’s proposal is that the narratological function of affect is consequently limited to the process of worldmaking (the mental creation of fictional worlds), which happens only through an active collaboration between author, text, and readers. While Hogan tried to ground his model uniquely in the inherent features of narratives, excluding — or at least putting aside — the readers, Breger seems to follow a growing tendency in literary studies, which gives a new relevance to readers (be they real or implied). ³⁹

Such a tendency is also evident in ⁴⁰, who devises a series of experiments with his own readers. By intermixing the chapters of an original short story with a taxonomy of emotions in literature, Oatley shows how such emotional systems sustain all narratives. A powerful support is found in the Sanskrit theory of rasas, intended as “essences of the everyday emotions […] that seem not so much individual […], but universal: aspects of all humankind” ⁴⁰.

In a similar effort, ⁴¹ refer to Monika Fludernik’s theory of a natural narratology ⁴², which “foregrounds the reader and focuses on the cognitive mechanisms underlying reader’s construction and interpretation of narrative” ⁴¹. Fludernik’s naturalism depends on the idea that narratives are built in readers’ minds through a re-shaping and variation of everyday human experience. By including the affective component, such a narratology may “expand its purview and become even more natural ” ⁴¹.

These contributions are just a sample of a currently growing trend in literary studies. The widest body of research on reader’s affects and emotions, in fact, can be found in the field of reader response theory, whose origins can be traced back to Aristotle’s concept of catharsis ⁴³, but which distinguished itself more recently for its scientific approach to the study of reading. Through the adoption of empirical methods ⁴⁴, in fact, reader’s experiences are analyzed and measured via questionnaires and interviews, but also using technologies like eye tracking and fMRI scans. In a recent series of papers, SA also has been introduced in the field.

⁴⁵ adopted SA on Shakespeare’s sonnets not to visualize their (frequently improbable) plot arcs, but to measure the “emotion potential” of their verses, with the goal of predicting possible readers’ reactions and thus devising new experiments on selected texts. ⁷, then, used SA on a social reading platform ⁴⁶, Wattpad, with a goal that connects even more strongly narratology and reader response theory. Given that on Wattpad readers can write comments for each paragraph in a novel (reaching even millions of comments), Jocker’s technique was adapted to compare two emotional arcs: that of the text and that of readers’ reactions. Results were used to isolate the passages that showed the highest levels of harmony or discrepancy (i.e., where the correlations between the two emotional arcs were the highest or lowest), thus identifying the textual features that support or hinder narrativization.

These studies prove how fruitful the integration between SA tools and literary studies can be. And if, on the one hand, they risk moving too deeply into the realm of social science thus losing contact with literature stricto sensu, on the other hand, they confirm how such a tendency is inscribed in the very practice of distant reading ⁴⁷. However, extensive work and reasonable carefulness are still required: main risk is that of an oversimplification of phenomena that necessarily escape any reductionism, while the current results of computational analyses are just scraping their surface. In any case, it seems more and more evident that, while a theory for distant reading is still lacking ⁴⁸, it is precisely through theoretical reasoning that SA (and other computational methods with it) can actually meet the needs of literary scholars. The possibility of testing proposals like those by Hogan and Oatley can prove extremely valuable for the development of literary studies, whatever the result (a confirmation or a denial) will be. And to the skeptics who — sometimes with good reasons — fear the imposition of quantitative methods over the irreducible subjectivity of literary criticism, it can be answered with ⁴⁹ that the process of modeling in computational literary studies is never the simple reduction of the phenomenon to a formula, but rather a continuous dynamics between the construction of a model and the confrontation with a reality that always escapes it — the same dynamics that, in the end, sustains any theorization about literature.

3. Tools criticism

A direct confrontation with literary theory proves fundamental when setting up a criticism of SA tools. However, it is not sufficient when trying to reach a full awareness of the potential and limitations in their use for literary studies. Once the context in which a tool can be employed has been identified, an equal attention should be dedicated to the specific method it adopts. Indeed, each method implies a model — and thus it also implies a theory. SA, in fact, can be performed by selecting or combining an ample variety of approaches, ranging from simple wordcount to the most complex deep learning techniques, and connecting with multiple psycholinguistics theories. Choosing one approach over the other means also defining the very nature of the object under examination.

3.1 A stratified taxonomy

When trying to propose a taxonomy of SA tools, at least three main distinctions should be made, based on three interconnected aspects: (1) the emotion theory adopted by the tool (T); (2) the technique used to build the emotion resources (ER); (3) the method adopted to accomplish the analysis (M).

3.1.1. Emotion theories

As for the emotion theory, an ample selection of competing frameworks is currently available, with their advantages/disadvantages and a lively dispute on which one is the best. However, they can be divided into two main families:

• T1. Dimensional representations of emotions
• T2. Discrete (or systemic) representations of emotions

Dimensional representations are generally connected to ⁵⁰, who proposed a bi-dimensional system able to chart all emotional states. By combining the two dimensions of valence (positive vs. negative, e.g., good vs. bad) and arousal (calm vs. intense, e.g., pleasurable vs. exciting) any human emotion could be logically represented. Many SA tools adopt this theory by simplifying it further, i.e., by reducing it to valence alone, on a continuous scale that ranges between two extremes (e.g., -1 and +1). This solution, chosen in studies such as ², ⁵, and ³⁷, offers an efficient simplification for the analysis, but it also implies the loss of relevant information, when for example aesthetic appreciation (e.g., beautifulness or ugliness) needs to be distinguished from embodied response (e.g., pleasure or pain). It should be noted, incidentally, that this interpretation is also at the basis of the very idea of SA. Especially in its commercial applications, SA aims at mining opinions (by stressing this specific meaning of the word sentiment), thus a distinction in terms of positive/negative valence becomes sufficient for accomplishing the task.

On the other hand, discrete representations multiply the number of dimensions, while at the same time distinguishing them more strictly into a series of discrete categories (or basic emotions). Two main theories dominate this context: ⁵¹, who proposed eight basic emotions (joy, trust, fear, surprise, sadness, anticipation, anger, and disgust) based on differences in human behavior; and ⁵², who reduced the categories to seven (anger, contempt, disgust, fear, joy, surprise, and sadness), based mainly on differences in facial expressions. However, theories are much more numerous ⁵³, and many SA approaches even combine them or reduce them to a simple dichotomy (all positive vs. all negative emotions). A unique framework is still far from being defined, while the results of any SA analysis depend heavily on the system that is chosen as a reference.

The biggest issue in the applicability of SA for literary studies, however, concerns the very possibility of a unique framework. ⁵⁴ demonstrated how the simple distinction between positive and negative sentiment in historical texts is an almost impossible task for human annotators. By evaluating inter-annotator agreement scores on a series of excerpts from historical political speeches, ⁵⁴ noted that, if the performance of humans is below the threshold of acceptability, delegating this task to a computer might make no sense at all. This warrants an extreme carefulness when applying SA to literary texts. However, ⁵⁵ has also demonstrated how, while inter-annotator agreement remains low, SA is still able to catch significant correlations, especially when comparing the emotional valence of a text with its connected readers’ responses.

3.1.2. Emotion resources

Once the theoretical framework has been decided upon, a second, fundamental decision concerns the choice of the emotion resources. In fact, the measurement of the overall emotion or sentiment expressed by a text depends primarily on the emotional values assigned to the smaller units that compose it, be they words, clauses, or sentences. Based on categorizations such as ⁵⁶ and ⁵⁷, three main approaches can be distinguished:

• ER1. Word lists
• ER2. Vector space models
• ER3. Labeled texts

The first two approaches pertain to the more general category of emotion dictionaries, where lists of words are associated to a series of (basic) emotions. Overall, emotion dictionaries are still the most used SA resource in DH, even if the interest towards labeled texts increased in recent years.

Word lists are the simplest approach, but they also require extensive preparatory work and frequently prove too rigid for an adaptation to different contexts. For example, the NRC Emotion Lexicon — also known as EmoLex ⁵⁸ — was developed through crowdsourcing: by using the Amazon Turk online service, its developers asked users to annotate a series of words both in terms of sentiment and (Plutchik’s) basic emotions. Final values were then assigned via majority consent. Issues were many, however, starting from the limited trustworthiness of Amazon Turk annotators (even if the developers devised methods to avoid errors or cheating) and culminating in the system of values unavoidably inscribed in the annotations. In the end, Emolex might prove to be a good representative of the emotions experienced by present-time Internet users (where e.g., the verb to cry clearly expresses negative sentiment), but not of the system of values that sustains a play by Shakespeare or a novel by Austen (where the same verb to cry can simply mean to say out loud).

For this reason, vector space models are frequently used to adapt the dictionary to a specific linguistic and cultural context, through distributional semantics ⁵⁹ and the computational technique more generally known as word embeddings ⁶⁰. Based on co-occurrences in selected corpora, words are transformed (or modeled) into multi-dimensional vectors, which encode information on semantic similarity. Starting from a selection of seed words (such as good vs. bad; or words indisputably related to basic emotions), it becomes thus possible to automatically assign a value to all words in a dictionary. This technique offers the advantage of tailoring the dictionary to a specific context, depending on the corpus that is used to generate the vectors. In this case, limitations depend primarily on the technical issues of word embeddings: for example, large corpora are required for their creation (but they are not always available, especially for historical languages) and the information encoded in the vectors does not necessarily model semantic similarity (e.g., it happens that the vectors of words such as good and bad are similar because the two words tend to appear frequently together).

⁵⁶ noted how word lists and vector space models can be combined into hybrid emotion dictionaries, which try to reach an ideal compromise between advantages and shortfalls of the two approaches.

However, a more general issue seems at stake with emotion dictionaries. As already noted, in fact, some kinds of emotions (such as the Deleuzian affects) escape linguistic formalization, thus they may be undetectable through SA approaches. This is one of the reasons why ⁵⁷ foster the use of labeled texts, where the basic unit of the emotion resource are not just words, but clauses and sentences. Ideally, in fact, even the least formalizable affects can be identified when focusing on a text span. Main issue for these kinds of emotion resources is their scarce availability for literary studies. While such material is readily available for e.g., product reviews and social media content, where the text of a review is generally accompanied by a simple rating (e.g., a number of stars) and a Tweet is supported by hashtags and emoticons (which frequently synthetize and disambiguate the emotions expressed), extensive annotation work is required for literary texts, with the above-mentioned issues of inter-annotator agreement. In any case, research in this field is rapidly moving forward, and annotated corpora such as ⁶¹ have been recently made available.

It is worth noticing how this issue is widely discussed also in literary studies, where the proposals in favor of a detectability of emotions through lexical cues are numerous: in the field of Medieval studies, for example, ⁶² bases her analysis of ancient emotional systems on words alone, while ⁶³ looks at the stylistics features that mark such aspects; the theory of emotives by ⁶⁴, then, leans on the concept of translation (intended in its wider meaning, as a process of connection between separated contexts) to build a virtuous cycle between language and feelings. Numerous original solutions have been developed also in the context of computational literary studies. ⁶⁵ developed their own emotion dictionary to study affect in Ulysses : to correctly operationalize Jameson’s theory of affect, they used it not to identify the passages dominated by such words, but those where such words did not appear. The dominance in these passages of words that pertain to the body, thus signaled the presence of (unexpressed) affects.

3.1.3. Computational methods

The final distinction in a taxonomy of SA tools pertains to the method adopted to accomplish the analysis. Here too, three main distinctions have been proposed ¹:

• M1. Simple (or advanced) wordcounts
• M2. Syntactic structure analyses
• M3. Machine learning techniques

Wordcount is evidently the easiest approach, which ignores sentence structure and word order to accomplish the most basic bag of words analysis. Given a text and an emotion dictionary, the words that appear in both are counted and their values summed to generate a final score. Such approach proves quite ineffective when dealing with short sentences or complex rhetorical structures but shows a surprising efficiency when the dimensions of the analyzed text increase. Unfortunately, no research on determining the minimum length for a reliable SA of literary texts — as done by ⁶⁶ for stylometry — exists yet. Simple wordcount can then also rely on statistics to better balance the relevance of single words (as done e.g., by ⁵): for example, if a positive word tends to appear homogeneously in multiple texts, its emotional valence might be relatively lower than that of words which appear just in a few passages. Statistics can support wordcount in even more complex ways, but when the analysis aims at fine-grained results, other approaches need to be employed.

A further step is the analysis of the syntactic structure of sentences to extract their overall meaning. This can be performed through different levels of complexity, which range from the simple identification of emotion shifters (e.g., negations, in sentences such as he was not a bad person, or it was neither sad nor boring), to a full parsing of sentences, which reconstruct their dependency trees (thus distinguishing principal from subordinate clauses, coordinating from adversative conjunctions, and so on). In theory, this approach should prove the best when aiming at high levels of precision. However, it has to deal with the multiple issues and limitations in natural language processing (NLP), especially when applied to historical languages. One of the most widely used NLP algorithms in DH, UDPipe ⁶⁷, still commits a substantial number of errors in parsing languages like Latin and Ancient Greek.

Machine learning (ML) places itself at the highest level of the taxonomy. It has recently established itself as the most effective approach to artificial intelligence, which adopts a bottom-up strategy to build a model of knowledge through a trial-and-error process ⁶⁸, where examples provided by humans (e.g., the recognition of emotions in texts) constitute the basis for a sophisticated imitation game. Some of its most advanced applications in SA (which even imitate the functioning of the human brain through artificial neural networks, also known as deep learning) are presented by ⁶⁹, ⁷⁰ and ⁷¹. Even the most complex issues in SA, such as the identification of irony ⁷² and sarcasm ⁷³, can be approached through ML. However, also ML has some fundamental limitations when applied to literary studies: main issue is that ML algorithms need human-annotated material to learn their tasks, thus they primarily depend on labeled texts (ER3), with all the related issues that were discussed above.

In conclusion, it can be stated that ML has become the dominant approach in SA. In a recent SemEval task, a total of 89 teams competed for the best SA approach in analyzing multilingual Tweets. ML approaches were the most common and successful among the participants ⁷⁴.

3.2 SA tools in literary studies

As noted at the beginning, SA approaches in literary studies are frequently many steps behind the most recent advancements in computational linguistics. This depends on the aforementioned intrinsic issues (e.g., the complexity of literary language and the unavailability of annotated corpora), but also on the tendency to adopt already-developed tools, which do not require the expertise of a computer scientist. While, on the one hand, this is a necessity in DH research (which cannot be reduced to a sub-field of computational linguistics), it may also lead to errors and misinterpretations. More subtly, as all tools bring about their implicit theories and biases, any analysis that looks just at the external outcomes without delving into the inner functioning logics, risks unintentionally supporting ideals that are not its own. This is why a critical analysis of SA tools becomes fundamental, focusing at least on the ones that are more extensively used in DH.

3.2.1 Syuzhet

Matthew Jocker’s Syuzhet is the software that originated the most recent wave of interest towards SA in literary studies. Syuzhet is probably one of the least advanced software for SA, but it efficiently combines speed and visualization power to produce effective results. When referring to the taxonomy described above, it can be labeled as:

• T1, as its default dictionary simply assigns valence to each word (even if it includes also an implementation of the NRC emotion lexicon)
• ER1, because the default dictionary was built through crowdsourcing (or wisdom-of-the-crowd)
• M1, because the analysis is run via simple wordcount

Additional feature in Syuzhet is a series of visualization algorithms, which apply multiple smoothing functions to generate elegant plot arcs. Generally, Syuzhet works as follows: (1) the analyzed text is split into sentences, (2) a series of sentiment values is produced for each sentence, and (3) the raw values are processed by the smoothing functions to generate plots. ⁷⁵ has already shown the main issues related to passages (2) and (3), such as the inability to detect negation and irony, and the distortions generated by smoothing functions like the Fourier transform. In addition, it should be noted that the speed of the algorithm is also determined by the fact that it does not count words in sentences, so the sentiment of the sentence He had a good hearth and a good mind (+1.35) is the same of the sentence He had a good hearth. Main advantage of Syuzhet is its transparency and adaptability: developed as a package for the R programming language, all its functions and resources are freely available and easily modifiable (but basic programming skills are required). It should be noted, however, that more advanced packages, such as Rsentiment and Sentimentr (the latter, developed as an expansion of Syuzhet ) are currently available for SA in R ⁷⁶. Among the most recent applications of Syuzhet in literary studies, ⁷⁷ used it to evaluate the trend of sentiments and emotions across three centuries of the English novel, while ⁷⁸ combined it with multifractal theory to analyze narrative coherence and dynamic evolution of the novel Never Let Me Go by Kazuo Ishiguro.

3.2.2 Vader

Vader ⁷⁹ is a slightly more advanced SA tool, as it moves beyond the simple word counts carried out by Syuzhet . After its first definition, it has been implemented in multiple programming languages, but one of the most used implementations is that in Python. Here, Vader has been integrated into the nltk library, which provides multiple NLP functions. In the current taxonomy, it can be classified as:

• T1, with a focus on valence alone
• ER1, with a dictionary developed through crowdsourcing
• M2, because it identifies valence shifters, intensifiers, et al.

Vader works on a sentence level and produces a numerical output (composed by four values: positive, neutral, negative, and compound, which is a normalized sum of the first three). The identification of valence shifters happens through a series of basic rules, which modify the sentiment values assigned to single words if they are preceded or followed by specific particles. For example, the sentence He had a good hearth scores a compound value of +0.44 (on a range between -1 and +1), while He had a good hearth! scores +0.49 and He had not a good hearth scores -0.34. The rules that generate these values are purely mathematical and were defined through an empirical approach. In a series of experiments with multiple annotators (e.g., asking them to evaluate the sentiment of good vs. good! and not good), numerical modifiers were assigned to single particles (e.g., 0.11 for exclamation marks and -0.69 for negations). Such a procedure determines a good balance between accuracy and computing requirements (the software is fast and quite reliable), but it also causes a relative rigidity of the model. It should be noted, in fact, that Vader was developed for the analysis of text produced in social media, thus the values of both emotion dictionary and modifiers were tailored for this specific context. In addition, it shows particular issues when dealing with irony and complex syntactic constructions: the sentence Well, he was like a potato, cited by ⁷⁵ when criticizing Syuzhet , deceives Vader too, with a compound score of +0.55. In literary studies, Vader was recently adopted by ⁸⁰ to explore the poetry of the Black Arts Movement and by ⁸¹ in a software pipeline aimed at generating visual summaries of narratives. However, the second study shows how better results are obtained by adopting a machine learning approach.

3.2.3 SentiArt

SentiArt ⁸² tries to cope with the issue of flexibility by adopting vector space models to generate its dictionary. In theory, SentiArt can be adapted to all contexts and languages (if enough training material is available). In the current taxonomy, it can be classified as:

• T1, with a focus on both valence and arousal
• ER2, with a dictionary developed through word embeddings
• M1, because it processes texts through simple wordcount

The creation of SentiArt emotion dictionaries works as follows: given a list of prototypical positive and negative words, a vector space model is used to expand it, by calculating the distance between these words and the entire dictionary. The model can be generated based on a selection of texts (taken from a specific author, genre, or period), or it can be simply downloaded from a repository of pre-trained models — e.g., FastText ⁸³. The first option is advised because it offers the possibility to tailor the dictionary to a specific context; however, it can become problematic when the training material is not enough to produce reliable vectors. Therefore, in the absence of alternatives and when working on contemporary texts, the second option constitutes a valid alternative. SentiArt has been developed in Python language, but it can also be used through the graphical interface of Orange (by installing the text add-on), which offers an easy access to its functionalities (with a limited set of pre-compiled dictionaries). The main advantage of SentiArt is that its hit rate reaches almost 100%, thus all the words in a text are given a value (while, in general, sentiment dictionaries cover just 10-20% of a text). This can constitute an issue when also function words (like articles and conjunctions) get a sentiment score: indeed, also such particles might have an impact in terms in valence/arousal (see for example the preposition Of at the beginning of Paradise Lost ); however, both semantic variance (e.g., multiple word senses) and syntactic functions (e.g., intensification and negation) are lost in the process. As a consequence, the analysis should be performed on larger text sets. ⁸² tested SentiArt on the Harry Potter novels, reaching promising results in predicting the emotion potential of text passages and in identifying the personality profile of characters.

3.2.4 SEANCE

SEANCE ⁸⁴ operationalizes the second theoretical framework, that of discrete representations of emotions. By combining basic NLP functions (such as negation detection) with multiple dictionaries, it offers the opportunity to reach a very high granularity in distinguishing discrete emotions. In the current taxonomy, it can be classified as:

• T2, with a total of 250 discrete dimensions

• ER1, as all dictionaries are provided through external resources

• M2, because it combines wordcount and basic syntactic rules

  SEANCE is distributed as a multi-platform graphical interface, that can be easily used by non-programmers. Its main potential is in the extensiveness of the vocabulary, which combines multiple resources in a single tool.⁸⁵ It includes the already-cited NRC Emotion Lexicon, but also some of the first-ever SA dictionaries, such as the General Inquirer ⁸⁶, the Affective Norms for English Words (ANEW, ⁸⁷), and many others. For each sentence or text (users have to prepare them in a plain text or tabular format) a total of 250 dimensions is measured. However, there is a significant overlap in these dimensions, as many of them encode the same phenomena (such as joy or fear), measured with different dictionaries. In addition, also multiple non-emotional, abstract concepts (such as causal, legal, and even aquatic) are measured. In literary studies, ⁸⁸ used it to compare the different editions of Wordsworth’s Prelude , tracking the change in emotional aspects and providing a quantitative confirmation to already-established critical interpretations.

3.2.5 Stanford SA

Even if developed before all the tools presented in this survey, Stanford SA ⁸⁹ is still one of the most advanced SA software currently available for digital humanists. Its main distinctive feature is the combination of ML and advanced NLP, with the ideal goal of identifying the sentiment of single sentences. In the current taxonomy, it can be classified as:

• T1, with a focus on valence alone
• ER3, as it works with human-annotated texts
• M2 and M3, because it combines parsing and ML

Stanford SA is written in the Java programming language and is part of Stanford CoreNLP , one of the most advanced NLP software suites. It can be used through command line (i.e., by typing a series of pre-formulated commands) and tested on a visually-efficient online demo. In simplified terms, Stanford SA works as follows: in a first phase, called training, the algorithm is given a series of sentences annotated by human raters. Based on these annotations, the algorithm learns how to distinguish five possible sentiments: very negative, negative, neutral, positive, and very positive. In ML terms, the output of this procedure is also called a model, intended as a formal representation of the analyzed phenomenon. At this point, the analysis of new sentences begins: for each sentence, (1) a full dependency tree is automatically built; (2) given that also ML algorithms can be structured as trees, the dependency tree is adapted to provide the structure for the ML algorithm; (3) the algorithm analyses the sentence. As evident, the success of the whole process depends heavily on the quality of the training phase. Here the possible issues are many, because annotation demands a significant amount of time and resources, and Stanford SA requires a complex annotation format, which focuses not on single words or sentences, but on all nodes in a dependency tree. When taken out-of-the box, Stanford SA performed poorly on nineteenth-century English texts, showing errors also in the reconstruction of dependency trees ⁹⁰. This depends on the fact that Stanford SA default algorithm is trained on contemporary movie reviews, thus it has substantial issues in adapting to different domains. In conclusion, while Stanford SA presents itself as one of the most sophisticated SA algorithms, its complexity in usage and requirements in training have kept digital humanists at a distance. In recent times, however, the interest of digital humanists towards ML approaches for SA — starting from isolated studies such as ⁹¹ — has increased substantially.

3.2.6 Transformers Pipelines

Transformers Pipelines represent one of the best compromises between simplicity of usage and complexity of the approach. Based on the Transformers architecture, made famous by the success of the BERT language model ⁹², Pipelines allow access to advanced ML functionalities through just a few lines of Python code. In the current taxonomy, they can be classified as:

• T1 and T2, with the possibility to switch between different models
• ER3, as models are created with human-annotated texts
• M3, because they adopt advanced ML

In the simplest implementation, through the text-classification pipeline, it is possible to calculate the valence of a sentence (accompanied by a confidence score). However, by selecting one of the many other text classification models available in the Hugging Face repository,⁹³ SA can also be accomplished in different languages and by applying many different emotion theories. One problematic aspect here is in the trustworthiness of models, which can prove efficient in accomplishing a task but can also bring about multiple biases ⁹⁴, even with ethical consequences (e.g., when implicitly modeling racist or sexist biases). While Transformers Pipelines constitute the easiest entry way for such a computational technique, it should be noted that most projects in DH try to get the best of it by using more sophisticated implementations. In fact, the possibility of fine tuning Transformers models via manual annotation stimulates the development of projects that aim at improving them further.⁹⁵ For example, ⁹⁶ fine-tuned Transformers models to recognize basic emotions in German poetry and categorize poems produced in different periods, while ⁹⁷ used a similar procedure in a project aimed at evaluating the levels of valence and arousal related to geographical entities in Swiss literature. One possible critical aspect of such an approach has been highlighted by ⁹⁸, who noted how, when dealing with traditional literary questions, Transformers do not substantially outperform simpler (wordcout-based) approaches, thus overkilling the problem with a hard-to-implement solution. However, the recent availability of high-standard online resources such as the Colab Notebooks , together with the development of research questions which require a fine-grained analysis of texts (see e.g., ⁹⁹), has made the adoption of such a solution more and more advisable in DH.

4. Conclusion

This short survey showed how the gap between state-of-the-art tools and current research in computational literary studies, while still present, seems to be gradually closing itself. And while a community-driven effort like the one in computational linguistics (embodied by phenomena such as the SemEval tasks) is still largely absent in DH, the recently growing interest (and criticisms) towards methods like SA suggests that it might be a natural outcome of the current evolution. In fact, among the most relevant acquisitions derived from the debate around ¹⁰⁰, is the importance of validation and reproducibility ¹⁰¹, i.e., the construction of a community of practice.

Still another, more theoretical issue seems to derive from a matter of modeling. When introducing bleeding edge technology in SA (as well as in all DH tools), a simple, direct connection between the phenomenon and its model seems to get lost: as shown by ¹⁰² for vector space models and by ¹⁰³ for unsupervised ML, any possible theoretical reasoning risks becoming empty or misleading when we do not know anymore the internal logic of the modeling process, or which phenomenon we are actually modeling. This is especially true for advanced ML approaches, which have been frequently criticized for their lack of transparency. SA adds a further complication to this, because of the ineluctable subjectivity that is inscribed in human emotions. A possible solution to this double conundrum can derive from the practice of annotation. In fact, as ML teaches us, the computational analysis (and prediction) of a phenomenon becomes possible only when humans have found an agreement in identifying it. By asking researchers, students, and literature lovers to annotate texts, testing existing theories and letting more general trends emerge, the dream of building a shared, community-driven hermeneutic machine ¹⁰⁴ might not be that impossible to reach.

For the moment, nothing advises against an — informed and critically aware — use of the tools that are currently available, starting perhaps from — but not limiting ourselves to — the tools presented here. Limitations are still many, starting from the fact that resources for the English language substantially outnumber those available for all other languages. However, advantages are equally significant, as in the recognition that all the tools presented here are available in the form of free, open-source, and easily modifiable software. And probably still, in the end, literary studies will continue without the need to include SA tools. In that case, no damage can be done. But if the two will find a way to connect more steadily and learn from each other, their evolution could actually become more than a simple development — and could finally be called progress.

As for the scientific validation of stylometric methods in DH, see for example the extensive body of research produced by Maciej Eder (e.g., see ^{105 66 106}) or the detailed inquiry by ¹⁰⁷. As for theoretical awareness, see for example ¹⁰⁸ and ¹⁴. A full bibliography on stylometry can be consulted on Zotero.

Note that the number of shapes is the same, but they do not correspond perfectly. All the plots generated by ⁵ can be explored interactively through an online Hedonometer.

The concept of the implied reader, derived from reception theory and intended as “a textual structure anticipating the presence of a recipient without necessarily defining him” ¹⁰⁹ has been criticized for its excessive abstraction, through which we risk losing contact with real readers ¹¹⁰. However, Hogan notices how “there are many cases in which we might wish to say that a given reader’s emotional response is misguided [too]” ¹¹¹. Thus, it seems that only a combination between the two (abstract modeling and empirical observation) might actually provide us with a reliable description of the phenomenon of reading.

SEANCE was originally conceived as an expansion of LIWC ¹¹², a widely-used (but proprietary) software, which measures more than 100 dimensions in multiple languages (but without including any syntactic rule). In literary studies, LIWC has been used by ¹¹³ to predict the fictionality of texts.

A procedure also known as transfer learning, as models which already hold a certain knowledge of human language are adapted to accomplish even more specific tasks.