Earlier this week Kathryn Schultz of the New York Times published a rather provocative, challenging, and in my opinion under-researched and over-sensationalized article about my colleague Franco Morreti’s work theorizing a mode of literary analysis that he has termed “distant-reading.” Others have already pointed out some of the errors Schultz made, and I’m fairly certain Moretti would be happy to clarify any confusion Schultz may have about his work if she were to actually interview him (i.e. before paraphrasing him). My interest here is to offer some specific thoughts and some background on “distant-reading” or what I have preferred to call “macroanalysis.”[1]

The approach to the study of literature that I call macroanalysis, instead of distant-reading (for reasons explained below), is in general ways akin to the social-science of economics or, more specifically, macroeconomics. Before the 20th century there wasn’t a defined field of “Macroeconomics.” There was, however, microeconomics, which studies the economic behavior of individual consumers and individual businesses. As such, microeconomics can be seen as analogous to the study of individual texts via “close-readings” of the material. Macroeconomics, however, is about the study of the entire economy. It tends toward enumeration and quantification and is in this sense similar to literary inquiries that are not highly theorized: bibliographic studies, biographical studies, literary history, philology, and the enumerative analysis that is the foundation of humanities computing.

By way of an analogy, we might think about interpretive close-readings as corresponding to microeconomics while quantitative macroanalysis corresponds to macroeconomics. Consider, then, that in many ways the study of literary genres or literary periods is a type of macro approach to literature. Say, for example, a scholar specializes in early 20th century poetry. Presumably, this scholar could be called upon to provide sound generalizations, or “distant-readings” about 20th century poetry based on a broad reading of individual works within that period. This would be a sort of “macro-, or distant-, reading” of the period. But this parallel falls short of approximating for literature what macroeconomics is to economics, and it is in this context that I prefer the term macroanalysis over distant-reading. The former term places the emphasis on the quantifiable methodology over the more interpretive practice of “reading.” Broad attempts to generalize about a period or about a genre are frequently just another sort of micro-analysis, in which multiple “cases” or “close-readings” of individual texts are digested before generalizations about them are drawn in very qualitative ways. Macroeconomics, on the other hand, is a more number-based discipline, one grounded in quantitative analysis not qualitative assessments. Moreover, macroeconomics employs a number of quantitative benchmarks for assessing, scrutinizing, and even forecasting the macro-economy. While there is an inherent need for understanding the economy at the micro level, in order to contextualize the macro-results, macroeconomics does not directly involve itself in the specific cases, choosing instead to see the cases in the aggregate, looking to those elements of the specific cases that can be generalized, aggregated, and quantified.

Micro-oriented approaches to literature, highly interpretive readings of literature, remain fundamentally important. Just as microeconomics offers important perspectives on the economy. It is the exact interplay between the macro and micro scale that promises a new, enhanced, and perhaps even better understanding of the literary record. The two approaches work in tandem and inform each other. Human interpretation of the “data,” whether it be mined at the macro or micro level, remains essential. While the methods of enquiry, of evidence gathering, are different, they are not antithetical, and they share the same ultimate goal of informing our understanding of the literary record, be it writ large or small. The most fundamental and important difference in the two approaches is that the macroanalytic approach reveals details about texts that are for all intents and purposes unavailable to close-readers of the texts. Writing of John Burrows’s study of Jane Austen’s oeuvre, Julia Flanders points out how Burrows’s computational study brings the most common words such as “the” and “of” into our field of view.

Flanders writes: “His [Burrows] effort, in other words, is to prove the stylistic and semantic significance of these words, to restore them to our field of view. Their absence from our field of view, their non-existence as facts for us, is precisely because they are so much there, so ubiquitous that they seem to make no difference.” (Flanders 2005)

At its most basic, the macroanalytic approach I’m advocating is simply another method of gathering information about texts, of accessing the details. The information is different from what is derived via close reading, but it not of lesser or greater value to scholars for being such.

Flanders goes on: “Burrows’ approach, although it wears its statistics prominently, foreshadows a subtle shift in the way the computer’s role vis-á-vis the detail is imagined. It foregrounds the computer not as a factual substantiator whose observations are different in kind from our own—because more trustworthy and objective—but as a device that extends the range of our perceptions to phenomena too minutely disseminated for out ordinary reading.” (Flanders 2005)

A macroanalytic approach not only helps us to see and understand the larger “literary economy” but, by means of its scope, to better see and understand the degree to which literature and the individual authors who manufacture the literature respond to or react against literary and cultural trends within their realm of experience. If authors are inevitably influenced by their predecessors, then we may even be able to chart and understand “anxieties of influence” in concrete, quantitative ways.

For historical and stylistic questions in particular, the macroanalytic approach has distinct advantages over the more traditional practice of studying literary periods and genres by means of a close study of “representative” texts. Speaking of his own efforts to provide a more encompassing view of literary history, Franco Moretti writes that “a field this large cannot be understood by stitching together separate bits of knowledge about individual cases, because it isn’t a sum of individual cases: it’s a collective system, that should be grasped as a whole . . .” (2005). To generalize about a “period” of literature based on a study of a relatively small number of books is to take a significant leap. It is less problematic, though, to consider how a macroanalytic study of several thousand texts might lead us to a better understanding of the individual texts. Until recently, we have not had the opportunity to even consider this later option, and it seems reasonable to imagine that we might, through the application of both approaches, reach a new and better informed understanding of our primary materials. This is what Juri Tynjanov imagined in 1927: “Strictly speaking”, writes Tynjanov, “one cannot study literary phenomena outside of their interrelationships.” Fortunately for me and for scholars such as Moretti, the multitude of interrelationships that overwhelmed and eluded Tynjanov and pushed the limits of close-reading lose can now be explored with the aid of computation, statistics and huge digital libraries.

My book on this subject, Literary Studies, the Digital Library, and the Inevitability of Influence, is now under contact with [Update: will be published in 2013 as Macroanalysis: Digital Methods and Literary History by University of Illinois Press.

[1] I began using the term macroanalysis in late 2003. At the time, Moretti and I were putting together plans for a co-taught course titled “Electronic Data and Literary Theory.” The course we imagined would be a research seminar in the full sense of the word and in our syllabus (dated 11/3/2003) we wrote: “the main purpose of this seminar is methodological rather than historical: learning how to use electronic search systems to analyze large quantities of data — and hence get a new, better understanding of literary and cultural history.” During the course I began work developing a text analysis toolkit that I later called CATools (for Corpus Analysis Tools). In terms of methodology, I was learning a lot at the time from work in corpus linguistics but also discovering that we (literary folks) have an entirely different set of questions. So it made sense to do at least a bit of wheel reinvention. My first experiments with the macroanalytic methodology were constructed around a corpus of Irish-American novels that I had been building since my dissertation research. I presented the first results of this work in Liverpool, at the 2004 meeting of the American Conference for Irish Studies. My paper, titled “Making and Mining a Digital Archive: the Case of the Irish-American West Project,” was part how-to and part results–I’d made one non-trivial discovery about Irish-American literary history based on this new methodology. In the spring of 2005, I offered a more detailed methodological overview of the toolkit at the inaugural meeting of the Text Analysis Developer’s Alliance. An overview of my project was documented on the TADA blog. Later that summer (2005), I presented a more generalized methodological paper titled “A Macro-Economic Model for Literary Research” at the joint meeting of the ACH and ALLC in Victoria, BC. It was there that I first articulated the economic analogy that I have come to find most useful for explaining Moretti’s idea of “distant-reading.” In 2006, while I was in residence as Research Scholar in the Digital Humanities at the Stanford Humanities Center, I spent a good deal of time thinking about macro-scale approaches to literature and then writing corpus analysis code . By the summer of 2007, I had developed a whole new toolkit and presented the first significant findings in a paper titled “Macro-Analysis (2.0)” which I delivered at the 2007 Digital Humanities meeting in Illinois. Coincidentally, this was the same conference at which Moretti presented the opening keynote lecture, a paper exploring a corpus of 19th century novel titles, which would eventually be published in Critical Inquiry. That research utilized software that I had developed in the CATools package.

Rowfont Press of Wichita, Kansas has just published a newly illustrated edition of Charles Driscoll’s memoir Kansas Irish (with my Critical Introduction). The book is available at Amazon. Kansas Irish and the two sequels that follow provide the most complete and authentic rendering of Irish life on the American prairie in the 19th Century.

Several months ago, a group of us from the Stanford Literary Lab wrote and sent out for review the article that now appears in Pamphlet 1 of the Lab. The article, titled “Quantitative Formalism: an Experiment” was submitted, peer-reviewed, and approved for publication in a prestigious literary journal. There was, however, a catch. The editors of the journal asked that we trim the number of charts in the article and that we alter the tone and character of the article to make it less of a narrative. In other words, the article was of a style and content that the editors found to be too foreign to their traditions.

Rather than revise the article in ways we felt would misrepresent its function and intent, we turned to that most traditional of literary forms, the pamphlet. Considering the largely quantitative and digital methodology employed in our research, a pamphlet was a seemingly ironic choice. The most obvious venue would seem to have been the web. After all, aren’t the blog and the web site, the pamphlet forms of the digital age (see, for example, Pamphleteers and Web Sites)? We certainly considered an electronic format, and we have posted a pdf version of the essay on our web site, but we decided to print.

Why print the pamphlet? As a literary form, the pamphlet has a long tradition of going against the grain; it’s an alternative form that is malleable. In the pamphlet, George Orwell wrote, “one has complete freedom of expression, including, if one chooses, the freedom to be scurrilous, abusive, and seditious; or, on the other hand, to be more detailed, serious and ‘high-brow’ than is ever possible in a newspaper or in most kinds of periodicals.” It has been used in campaigning and marketing, but most famously, the pamphlet has been employed by political, religious, and social “provocateurs” and “radicals.” Many of these objects have been lost, but the best have been preserved, such as those crafted by talented satirists (Swift) sober thinkers (Paine) and social critics (Voltaire). And they are preserved because they are historical objects, “ephemera” as the librarians say. The pamphlet is an ephemeral object, an object “lasting only for a day.” As “an experiment” our “quantitative formalism” pamphlet is a middle point, a hash mark on the line of time, not an end point or destination not even a beginning.

It’s interesting to consider how the preferred citation style of literary scholarship, the MLA Style, places emphasis on page reference over time, over moment of publication. With some obvious exceptions, the basic logic here is that what someone says about Shakespeare today has the same validity, the same scholarly purchase, as something said fifty years ago. As a discipline, our “results” tend to be qualitative and interpretive, not bound in time or subject to revision based on the introduction of new evidence. They are, of course, subject to reinterpretation, but that is fundamentally different from the changes wrought by new discoveries. The date-based citation style, on the other hand, places emphasis on points in time and acknowledges the fast-paced and ephemeral nature of certain fields of research. This is most obvious in the sciences, in medicine, for example, where new experiments and new discoveries are constantly and quickly changing the field. One need only eavesdrop on the the Digital Humanities twittersphere for a few hours to note the similarities; the pace of change is rapid.

So why not a pamphlet? Why not recognize in both form, title, and narrative style that ours is an experiment? It is a bit of research that is useful for today but also something we entirely expect to change. Indeed, we are already working on the next iteration(s), the next experiments. Unlike the neatly closed arguments of our traditional work, these experiments of our Literary Lab open as many doors as they close. In fact, in the course of our research on novel genres, it became apparent that we could, must, go on forever. Each test led us to some new idea, some new direction to explore. There are some discoveries to be sure, and some of our results will likely, hopefully, stand the test of time. But my co-authors and I understand, or perhaps simply “believe,” that there is still much, much more work to be done. If it reads more like a lab report than a traditional essay, it’s because it is a lab report and self-consciously so, intentionally so.

Readers wishing to experience the full pleasure of a touchable paper pamphlet may contact me with their name and address. No charge, while supplies last:-)

I’ve been hearing a lot about “collaboration,” especially in the digital humanities. Lisa Spiro at Rice University has written a very informative post about Collaborative Authorship in the Humanities as well as another post providing Examples of Collaborative Digital Humanities Projects. Both of these posts are worth reading, and Spiro offers some well-thought out and researched perspectives.

My own experiences with collaboration include both research and authorship. I have seen first hand how fruitful collaboration, especially interdisciplinary collaboration, can be. It is safe to say that I’m a believer. In fact the course I have been teaching for the last two years, Literary Studies and the Digital Library, is designed entirely around collaborative research projects. And yet I have to say that I’m am entirely suspicious of the current rage for “collaboration.”

No doubt the current popularity of collaboration, at least in the humanities, is a natural extension of the movement toward interdisciplinary studies. Through collaboration with people outside our individual disciplines has led to fruitful work, there seems to be an unnatural desire on the part of some administrators and even some colleagues to “foster collaboration,” as if collaboration were something that occurs in a petri dish, something that needs only to be “fostered” in order to evolve.

But collaboration does not arise out of a petri dish, it arises out of need. Sure, there are serendipitous collaborations that arise out of proximity: X bumps into Y at the water cooler and they get to talking . . . but more often successful collaboration arises out of need: X wishes to investigate a topic but requires the skills of Y in order to do a good job.

Failed collaborations, on the other hand, are all too often the result of good intentioned but overly forced attempts to bring people together. I attended a seminar a couple of years ago on the subject of “fostering collaboration in the humanities.” The organizers of the meeting certainly understood the promise of new knowledge that might be derived through interaction, but they entirely miscalculated when it came to individual motivation to collaborate. It’s a classic case of putting the cart before the horse. In my experience, fruitful collaboration evolves organically and is motivated by the underlying research questions, questions that are always too big and too complex to be addressed by a single researcher.

Those who do corpus level computational text analysis are always hungry for more and more texts to analyze. Though we’ve become adept at locating texts from a wide range of sources (our own institutional repositories as well as a number of other places including Google Books, the Internet Archive, and Project Gutenberg), we still face a number of preprocessing tasks to bring those various files into some standard format. The texts found at these resources are not always in a format friendly to the tools we use for processing those texts. For example, I’ve developed lots of processing scripts that are designed to leverage the metadata that is frequently encoded into TEI-based xml. A text from Project Gutenberg, however, is not only just plain text, but it has a lot of boilerplate text at the beginning and end of each file that needs to be removed prior to text analysis.

I’m currently building a corpus of 19th century novels and discovered that many of the texts I would like to include have already been digitized by Project Gutenberg. This, of course, was great news. But, the system I have developed for ingesting texts into my corpus assumes that the texts will all be in TEI-XML with markup indicating such important things as “author,” “title”, and “date” of publication. I downloaded about 100 novels and was about to begin opening them up one by one and adding the metadata. . .eek! I quickly realized the mundanity of the task and thought, “hmm, I bet someone has written a nice regex script for doing this sort of thing.” A quick trolling of the web led me to the web page of Michiel Overtoom who had developed some python scripts for downloading and cleaning up (“beautifying” in his language) Dutch Gutenberg texts for his eBook Reader. Overtoom’s process is mainly designed to strip out the boilerplate and then rename the files with naming conventions that reflect the author and title of the books.

With Overtoom’s script as a base, I reengineered the code to convert a Gutenberg text into a minimally encoded and TEI-compliant XML file. The script builds a teiHeader that includes the author and title of the work (unfortunately, Project Gutenberg texts do not include publication dates, why?) and then adds “text”, “body”, div, and all the p tags. The final result is a document that meets basic TEI requirements. The script is copied below, but since the all important python spacing may be destroyed by this posting, it’s better to download it here and then change the file extension from .txt. to “.py”. Enjoy!