RESOURCES FOR ASSESSING DIGITAL SCHOLARSHIP

Listed below are a series of  resources and tools for the assessment of digital scholarship.  These resources include Learned Society models of assessment, models in practice at the departmental level, research projects employing these technologies, and resources posted by scholarly journals.  This list also includes a brief discussion of ways to define digital scholarship in the humanities and questions to consider as part of the assessment process.

1. Learned Society models for digital scholarship assessment

Professional Guidelines for assessing format, design and scholarly content have generally been  initiated by learned societies, with universities following.  See below for key learned society models.

MLA (Modern Language Association)   www.mla.org/guidelines_evaluation_digital

AHA (American Historical Association) www.historians.org/perspectives/issues/2001/0110/0110pro1.cfm  (focuses specifically on evaluation for tenure and promotion)

http://historians.org/teaching-and-learning/digital-history-resources/evaluation-of-digital-scholarship-in-history/guidelines-for-the-evaluation-of-digital-scholarship-in-history (focuses on general scholarly evaluation)

NINES (Nineteenth Century Scholarship Online)  — scholarly organization dedicated to providing standards for rigorous peer review.

http://institutes.nines.org/docs/2011-documents/guidelines-for-promotion-and-tenure-committees-in-judging-digital-work/

ACLS (American Council of Learned Societies)  see 2006 Our Cultural Commonwealth: The Report of the ACLS Commission on Cyberinfrastructure for the Humanities and Social Sciences.

http://www.acls.org/programs/cyberinfrastructure/

Additional Resources:

—NITLE (National Institute for Technology in Liberal Education)

—http://www.slideshare.net/nitle/byerly-nitle-digital-scholarship

Presentation on Evaluating Digital Scholarship

—MLA Guidelines for Editors of Scholarly Editions https://www.mla.org/Resources/Research/Surveys-Reports-and-Other-Documents/Publishing-and-Scholarship/Reports-from-the-MLA-Committee-on-Scholarly-Editions/Guidelines-for-Editors-of-Scholarly-Editions

—18th Connect   http://www.18thconnect.org/about/scholarship/peer-review/

—18th century scholarship online (partner of NINES)

—Like NINES, 18th Connect offers peer-review based on two key areas of critique:  Is the content important and does it advance the field? Is it presented in a way that is accessible, sustainable, well-organize and well-documented?

2. Specific Models in Practice at Universities 

The examples listed below represent both evaluation procedures that have been established at institutions, as well as guidelines and ongoing investigations of the best ways to incorporate evaluation of digital scholarship into tenure and promotion practices.

—http://digitalhumanities.unc.edu/resources/valuing-evaluating-dh-practice/ from UNC Chapel Hill

—Promotion & Tenure Criteria for Assessing Digital Research in the Humanities from University of Nebraska-Lincoln’s Center for Digital Research in the Humanities

—Evaluating Digital Scholarship from the University of Virginia’s College and Graduate School of Arts and Sciences

—How to Evaluate Digital Scholarship (for Academic Review Committees, Deans, Provosts, and Chairs) by Todd Presner at UCLA

—Memorandum Regarding the Evaluation of Digital Scholarship from the Emory College Humanities Council

—http://newmedia.umaine.edu/interarchive/promotion_tenure_redefinitions.html  from the
University of Maine, New Media Department

Example A: English Department, College of Humanities and Social Sciences, NC State University

In evaluating scholarly, creative, or journalistic work in electronic forms, including websites, archives, tools, and other resources, the Department follows the recommendations of the MLA’s Committee on Computers and Emerging Technologies as set forth in “Guidelines for Evaluating Work in Digital Humanities and Digital Media” (2012) as well as recommendations from a subcommittee of the department voting faculty regarding digital media and digital humanities projects.  The document (under “Faculty Resources” on the departmental website) recognizes that not all Digital Humanities and Digital Media projects are practicable to review prior to distribution/publication and not all will be distributed via 3rd parties. assured.

To be considered refereed, publications should undergo field-specific peer or editorial review or accreditation, a process that may include consideration of:

• — Who edits the source of publication and who serves on the review board
• —The collaboration or community out of which a DH or DM project develops or is incubated (e.g., NEH workshops).
• —Grant funding (i.e., successful proposals for grant funding should be considered equivalent to a favorable peer review)
• —The project host (e.g., university sponsored, commercial, etc.)
• —Project availability, reach, and impact (i.e., what audiences does the project reach and how effectively?)
• Project affiliations (i.e., connections to other projects and conversations)
• —Refereed electronic publications will be considered equivalent to print publications.
• —For the purpose of RPT, a book, monograph, article, or edited text will fulfill departmental standards only when it is either published (either in traditional or electronic form) or has proceeded through a stage of review at which publication is assured.

3. Research Projects Employing Digital Humanities methodologies
Example B: Livingston Online

—*Livingstone Online: http://www.livingstoneonline.org/

—Collaboration of historians of medicine and science and scholars in English from various universities around the world.

—Involves historical research, digital curation and archival work, critical commentary and sophisticated web design.

—“The aim of Livingstone Online is to use the potential of electronic publishing to make available an online edition of the medical and scientific correspondence of David Livingstone. We have now produced transcriptions and high-quality colour reproductions of the letters from Livingstone held by several major libraries in the United Kingdom and elsewhere as well as letters held in smaller collections.”

—Received NEH support.

Example C. Visualizing Emancipation

*Visualizing Emancipation: http://dsl.richmond.edu/emancipation/

—A project of the Digital Scholarship Lab at University of Richmond.

—“Visualizing Emancipation organizes documentary evidence about when, where, and how slavery fell apart during the American Civil War. Funded by a We the People grant from the National Endowment for the Humanities, it shows how emancipation occurred unevenly across the South, beginning before the first major battles and ending after the end of the Confederacy. It shows the complex interactions between federal policies, armies in the field, and the actions of enslaved men and women on countless farms and city blocks.

—Visualizing Emancipation maps a messy story, in which patterns partially visible today remained hidden to many involved. It presents a history of emancipation where brutality is sometimes easier to see than generosity and where the costs of war and freedom fell disproportionately on the most vulnerable in the South. The war that brought freedom to millions brought unmatched destruction and disruption. If emancipation was a process, it must have seemed a chaotic, directionless one to many caught up in it. Visualizing Emancipation shows a war in which alliances between enslaved people and union soldiers were uneasy and often tested, but which yielded, somehow, the end of slavery.”

—Received NEH support.

4. Ways to Define Digital Scholarship in the Humanities

Given the varied nature of digital humanities research, any definition of scholarship employing methodologies of the digital humanities needs to be broad in scope.

Not a single field or discipline, but an array of practices:

(1) may be data focused or object focused

(2) may incorporate visualization for spatial analysis

(3) may be collaborative, open-access and non-proprietary

(4) may be more process-oriented than product-oriented
(from http://www.slideshare.net/nitle/byerly-nitle-digital-scholarship)

5. Questions to Consider for Assessment

• What assessment standards can be put in place to enable campus-wide evaluation procedures for digital scholarship, and how can these standards effectively evaluate research which is typically more process oriented?
• Do assessment standards need to be established at the level of the College or separately by departments and divisions?
• What weight should be given to digital scholarship as part of the evaluative process during tenure and promotion?
• How should digital scholarship and digital humanities scholarship be defined?
• In what ways can an exploration of guidelines and procedures introduced by learned societies and best practices at other institutions help Wake Forest establish its own guidelines and standards?

6.  Additional Resources Posted by Scholarly Journals

Journal of American History Guidelines for Reviews

—Digital History http://www.journalofamericanhistory.org/submit/digitalhistoryreviews.html

—Specific questions reviewers are expected to ask include:

—– Content (Is the scholarship sound and current? What is the interpretation or point of view?)

—– Form (Is it clear? Easy to navigate? Does it function effectively? Does it have a clear, effective, and original design? Does it have a coherent structure?)

—– Audience/Use (Is it directed at a clear audience? Will it serve the needs of that audience?)

—– New Media (Does it make effective use of new media and new technology? Does it do something that could not be done in other media–print, exhibition, film?)

—Engaging Digital Scholarship: Thoughts on Evaluating Multimedia Scholarship by Steve Anderson and Tara McPherson in the Journal of the Modern Language Association

—Short Guide To Evaluation Of Digital Work by Geoffrey Rockwell in the Journal of Digital Humanities