By Mario Biagioli

Emergent Conversation 16

This essay is part of the series PoLAR Online Emergent Conversation on Peer Review as Intellectual Accompaniment

I believe that, rooted in the traditional peer review model, the conceptualization of dialogue informing this collection may in fact constrain the ambitious rethinking of the process of editorial review we have been encouraged to pursue here. After suggesting ways to enhance the dialogical dimensions of the traditional peer review model, I will propose a substantially different conceptualization of dialogue, one framed by the more recent practice of post-publication review.

Michael Eisen, a founder of PLoS—a journal that has been crucial to promoting open access—has argued that one of the key functions of peer review is aimed at people who will never read that publication.[1] From deans to the general public, there are many who want to know if a publication was well researched, properly argued, and deserving publication, but may not have the skills or the time to actually read it. The focus on peer review’s function for non-readers seems radically alien to the concerns behind Čarna Brković’s and Jennifer Curtis’ suggestion for our Emergent Conversation: peer review’s potential for staging dialogues between authors and reviewers, reframing the review process not as “working against” the submission but “with it.” What I find interesting is that Eisen, Brković, and Curtis are all right, for different reasons. The practice of peer review does range between quality control and dialogue—between “working against” and “working with.” But while based on the assessment of peers, it is simultaneously aimed at the needs of other peer readers as well as of outsiders who will treat it as a blackboxed “knowledge product” bearing a peculiar certification mark: “peer reviewed.” Modern peer review has been introduced with both communities in mind—the scientists and the public.

The effects of peer review radiate out in concentric circles from the place and moment of editorial decision, where and when the unpublished manuscript metamorphoses (or not) into a published article or book. At the center of those circles, authors, editors and referees engage with the text, often with substantial care, but the level of engagement diminishes rapidly as soon one moves to more distant circles. There one finds users who will trust that publication (without necessarily reading it carefully) precisely because it has been read and “audited” by professionals. And then, further away from the center, there will be those who will not read but simply count that publication to rank a university based on its output of peer-reviewed articles. This kind of “managed alienation”—from close engagement to non-reading—is made possible by peer review, which requires dialogue among peers but also enables distancing by means of certification.

As we think about ways to foster and value the conversations staged by and through editorial peer review, we should not forget that that is only a snapshot of the beginning of the story. Those texts that do not make it through the filter are unlikely to become part of further conversations, while those that do may then feed future debates and disputes between authors and readers. If all goes well, there will be many more and larger dialogues after editorial peer review. At the same time, some of those conversations might in fact be foreclosed by the perception that peer reviewed has certified a text, turning it into a product to be used rather than an argument one should keep “working with.” I would not go as far as to mis-paraphrase Barthes into saying that the death of the referee is the birth of the reader.[2] The point is that texts should have the longest active post-publication lives, that readers could and should provide authors with much more (and perhaps better) feedback than the two or three referees that happened to be on the editor’s rolodex, and that, ideally, all publications should (have the option to) evolve as a result of those conversations. As we focus on how to enhance the dialogues that take place around the “birth” of a publication, we should not lose sight of the long game, and how that could be both enhanced and constrained by the way we arrange editorial peer review to begin with.

I have spent a few decades thinking about peer review as an author, a referee, and a scholar, keeping an eye on its contemporary developments and empirical studies of its performance, while also engaging its history, especially its intriguing connection to book censorship.[3] I also take reviewing very seriously. I stopped writing book reviews long ago, believing that my time could be better spent helping the publication of new research rather than helping academic presses market the books they had already published. And while I am notoriously late as a reviewer, I put much care in the reports I write. Substantial experience and some knowledge have not brought clarity, however. I still do not know for sure if peer review is good, bad, or the least bad tool we have to guide the publication and evaluation process. While I believe peer review can prevent the publication of clearly sub-standard submissions (but not necessarily catch fraud), I cannot tell how much it actually improves (or merely modifies) what was deemed publishable “with revisions.” Editor-ordered revisions certainly makes articles look more polished, with fewer gaps or jumps and more references and more nuanced claims to ruffle fewer feathers. But would not some careful old-style copyediting have taken care of much of that? How many of the revisions are truly substantive rather than more or less cosmetic?  And how much do they really help readers? Can’t they figure out a publication’s obvious limitations and strong points themselves? Would it not be enough for peer review to spot and correct empirical errors, leaving matters of interpretation to the readers?

While I can make a strong argument for peer review (at least some revised and improved version of it), it would be equally possible to write a systematic critique of both its well-documented empirical shortcomings and its function as an apparatus of academic disciplining, or mere polishing. Rather than a problem, however, I find this predicament remarkably interesting precisely because of the tensions it evidences between the perceived unnegotiable need for peer review and the simultaneous acknowledgment of its spotty performance and dubious foundations. Unable to engage these issues in any detail here, I will present two vignettes, one outlining possible improvements, the other sketching out some more radical changes. They are meant as food for thought, not policy suggestions.

The journal peer review process is hierarchical and has an unquestionable filtering role. As much as we would like to reframe it as “working with” the submission, the dialogue between authors, editors, and reviewers will always have a “working against” component. There are, however, ways to make the process more dialogical, starting, I suggest, by ending the editors’ practice of asking referees to judge the submission by placing it in standard boxes—accept, reject, accept with revisions, revise and resubmit, etc. For the interaction between referees and authors to be more dialogic, it would help that authors knew that, while referees express candid opinions about the text, they are not making final recommendations.

The dialogue between authors and reviewers could also be made more interactive by allowing the two to connect (anonymously, if so wished) through an email link provided by the journal. All the communications would be recorded on the journal’s server and could be monitored and curated by the editor, but the discussants could talk as they wish for as long as they wish about the reviewer’s report and the author’s proposed changes. When the conversation reaches its end, the editor could read the transcripts of the exchanges and make a decision. (That record, I believe, would be more informative than a standard referee report followed by the author’s response, which is usually defensive, being perceived as an exercise in compromise rather than dialogue). In any case, the technical details are not crucial here. The point is that the kind of dialogue we have now based on the exchange of series of comments and counter-comments once or twice over a few months does not look or sound like dialogue, and that there should be better ways to make it more interactive.

It would also be important and motivating for reviewers to know not only what the outcome of the process was (which they rarely hear from social sciences and humanities journals), but also what the other reviewers’ opinions were, and how the authors responded to them.  Science journals provide that information almost routinely, which allows one reviewer to learn about (if not quite dialogue with) the other referees’ reports. Personally, I am intrigued to see the often different opinions of other reviewers, and then watch editors craft “consensus” from them—a very interesting opportunity for brief ethnographic observations of knowledge-in-the-making through the editorial process.[4]

While I am not sure that paying referees for their work would be a good idea, I believe they should be clearly credited for their work in the article, maybe in a second byline for reviewers rather than being either thanked as “the anonymous reviewer who provided insightful comments” or collectively listed in alphabetical order in the last issue of the year, as some science journals do. Being clearly listed and credited could also make referees more accountable. Additionally, the editor should offer to publish appropriately edited versions of the reports if they are deemed to be informative to the readers, bringing them into the dialogue that had started during the peer review process.  (While in some cases referees may have contributed more to an article than some of the official co-authors, I do not think it will be appropriate to have referees become co-authors as that could produce serious conflicts of interest).

None of these suggestions would turn peer review into a real dialogue. No matter what, the author has indefinitely more skin in the game that the referee, and this is bound to skew the conversation. Still, by clearly freeing the reviewer’s role from that of providing explicit advice for acceptance or rejection, by making the conversation between authors and reviewers more interactive, and by incentivizing the reviewer to write his/her reports in a form that is thoughtful and articulate enough to be attached as an appendix to the article (which should not be difficult to publish electronically), we could achieve a non-trivial approximation of a dialogue. And it could be more interesting for the readers as well, who would not have much trouble jumping in and continuing the conversation started during the editorial peer review process, even (or perhaps even more so) if the author and the reviewers ended their dialogue in disagreement.

That said, it is also worth engaging the very different perspective taken by the proponents of post-publication review. Starting from the perception of traditional editorial peer review as part of the problem, not the solution, they have proposed a radically different conceptualization of review, one that de-centers both the review and publication process, questioning the traditional sequence: submission → review → editorial decision → publication. Post-publication review, I believe, could dramatically reframe the conversation we are having here.

Central to post-publication review is questioning the assumption that a text should go through a life-or-death judgment by two reviewers. (Juries are a lot bigger than that). In addition, no matter how competent and thoughtful, those two or three people cannot be representative of the current state of a discipline today and, a fortiori, in the future. (That may have been less of a problem hundred years ago when fields and curricula were more stable, but such stability is certainly gone from the “university of excellence,” which seems to constantly adopt new fields and tweak others). And no matter how up-to-date and future-oriented they might be, these few referees will not be able to judge the importance of the article and the “impact” it could have in the future as the disciplinary landscape of its possible relevance will evolve, or perhaps turn into a desert. As a result, if the referees do not find the publication important or relevant and the editors follow that advice, they may end up “killing” something that might have become valuable in the future, foreclosing the possibility of new and possibly important conversations. (We may be professionally inclined to focus on the costs of poor publications and information overload, but post-publication review is asking us to think seriously about the present and especially future costs of not publishing). And while nobody can predict the future, having only a couple of people making that call is particularly risky, more so when we all know from personal experience that the quality of referees can be rather uneven. Transferring much of the review function to present and future readers would thus seem a better wager.

The argument for post-publication review may look anti-hierarchical—down with the reviewers!—but is in fact framed by the perceived opportunity costs of peer review, and the publications it forecloses without having the proper knowledge to do so. The figure of the dialogue between authors and reviewers is a very attractive one as it captures much of what we scholars hold dear—engaged conversation across different positions—but in this case the participants may be just too few, with too much authority. Post-publication review is not just about adding voices to editorial dialogues in the present but to increase the chance that there will be conversations in the future, possibly in very different discursive landscapes. Just think about how fruitful the rediscovery of Gabriel Tarde has been in the last twenty years, in the post-Durkheimian age. The chances of such rediscoveries happening more frequently depend on letting those publications happen, which in turn may generate comments and reviews—the kind of materials that will either enable a rediscovery (or reinforce a well-earned oblivion).

It is not that peer review has nothing to contribute to this new framework. Journals that use post-publication review (but also depositories like ArXiv, BiorXiv and others) do not publish any submission they receive. (The screening done by depositories is minimal but has been tightened up during Covid to control the avalanche of dubious science about the pandemic).[5] Unlike preprint depositories, PLoS relies on traditional peer review, but explicitly asks reviewers not to evaluate the submission based on its perceived importance or likely impact. To the proponents of post-publication review, that kind of judgment is either impossible or “ideological.” Reviewers are just asked to report on whether the experiment is well-planned and executed, and that the data analysis meets accepted standards. The referees are asked to filter out manuscripts that are sloppy or worse, but it will be up to present and future readers to determine whether they find that text important or suggestive enough to use and connect it to discursive networks that will exist or be constituted at that time. Post-publication review has emerged from debates – simultaneously very important and utterly mundane – over the future of scientific journals.  Behind it, however, we see a radically dialogical view of knowledge-making that is not limited to science but in fact bears uncanny resonances with Bakhtin’s classic analysis of dialogue.[6]

There would be many ways to incorporate post-publication conversations into the text, if one is willing to partially give up on the notion of “version of record.” Readers could post their comments (that could be curated by the editors), and the authors could revise and re-post their texts—a common practice on depositories. (Protocols about how to constructively manage these public comments are being developed as we speak).[7] And comments could be posted also after the author has passed, creating an inter-generational record of the conversation for future readers to engage.

Compared to traditional peer review, the dialogue between authors and reviewers is displaced in time and space, spreading anywhere on a map unstructured by concentric circles. And as the figure of the reviewer merges with the general reader, both the role of the editor and the certification function of peer review are reduced. (Post-publication review does not “certify” the text for readers and users but rather enriches their reading by enveloping the text in critical commentary). In sum, peer review no longer takes place between authors, editors, and referees in those few months prior to publication, in a dialogue staged by the editor, but will follow from the dissemination of the publication itself, which will be more likely to happen if the review standards are limited to catching sloppiness of the submission rather than judge its importance or relevance. Finally, post-publication review may be less personal because reviewers will not be likely to “put themselves in the authors’ shoes,” given that the publication has already happened. This is rather different from the image of intense collaborative knowledge-making scenarios that started our conversation. But would the loss of the “aura” of that image be balanced by the opportunity of enabling many more conversations?

Mario Biagioli is Distinguished Professor of Law and Communication at UCLA. His current research is at the intersection of STS and intellectual property law. He has edited, most recently, From Russia with Code (Duke 2019) (with Vincent Lepinay), Gaming the Metrics (MIT 2020) (with Alexandra Lippman), and Academic Brands (Cambridge 2022) (with Madhavi Sunder). His “Ghosts, Brands, and Influencers: Emergent Trends in Scientific Authorship” has just appeared in Social Studies of Science.

Notes

[1] Michael Eisen, EMBL Forum Science and Society Seminar, August 9, 2018, at: https://www.youtube.com/watch?v=0XbdMOcMB4E, at 57’.

[2] Roland Barthes, “The Death of the Author”, Image/Music/Text (New York: Hill and Wang, 1977), pp. 142-148.

[3] Mario Biagioli, “From Book Censorship to Academic Peer Review” Emergences 12 (2002): 11-45.

[4] See a partial analysis of the review of grant proposals as discussed in Donald Brenneis, “Discourse and Discipline at the National Research Council” Cultural Anthropology 9 (1994): 23– 36.

[5] ArXiv focuses on mathematics, physics, and computer science, rejecting only 2% of the submissions (Daniel Garisto, “ArXiv.org Reaches a Milestone and a Reckoning”, Scientific American, January 10, 2022). The rejection rates are significantly higher (but still much lower by journals’ standards) in biomedical depositories like BiorXiv and especially MedRxiv (Richard Sever, John Inglis, Theodora Bloom, Claire Rawlinson, Harlan Krumholz, Joseph Ross, “Pandemic preprints—a duty of responsible stewardship” thebmjopinion April 27, 2021, at: https://blogs.bmj.com/bmj/2021/04/27/pandemic-preprints-a-duty-of-responsible-stewardship/.

[6] “There is neither a first word nor a last word. The contexts of dialogue are without limit. They extend into the deepest past and the most distant future. Even meanings born in dialogues of the remotest past will never be finally grasped once and for all, for they will always be renewed in later dialogue. At any present moment of the dialogue there are great masses of forgotten meanings, but these will be recalled again at a given moment in the dialogue’s later course when it will be given new life” quoted in Katerina Clark and Michael Holquist in their book Mikhail Bakhtin, (Cambridge, Mass.: Belknap Press of Harvard University Press, 1984), pp. 349-50. I thank Jennifer Curtis for this point, and the quote.

[7] Sandra Franco Iborra, Jessica Polka, Iratxe Puebla, “Point of View: Promoting Constructive Feedback on Preprints with the FAST Principles”, eLife, April 27, 2022, (DOI: 10.7554/eLife.78424), at: https://elifesciences.org/articles/78424.