How Current Scientific Cultures and Metrics Reproduce Privilege

At the Privileged Logics 2024 conference, there was broad agreement that metrics currently used are unfair and reproduce privilege. Reasons include adherence to a competitive scientific culture, perceptions that these metrics are objective, their usefulness in appealing to outsiders for engagement and recruitment, and inertia and/or mistaken notions of rigor from those who have been at the institution for a long time and don’t want to change the way things are done.

Examples of metrics that have affected outcomes for individuals due to privilege included:

• The demand for novelty and transformation (although there can also be an anti-innovation bias denying merit to delivery modes such as podcasts as well as to core expanding substantive areas and a focus on community impact) over replication and incremental gains
• Statistical significance as a measure for worthwhile research
• Tenure as a status attached to financial stability
• Grades as measures of learning and achievement
• Differential treatment based on your field or major
• The demand for grants and differential value attached to different kinds of grants (e.g., federal), and different methodologies (e.g., community-based research takes more time, and this is not accounted for when counting publications).

We also talked about “memorable messages”: E.g., “Don’t focus too much on teaching,” which undergird the existing hierarchies of advantage. 

Strategies for mitigating negative outcomes of metrics because of privilege included:

• Flexible appointment scheduling for research participants
• Appropriate financial compensation for research participants
• Workshops for informal peer review of methodology that assigns value before publication
• Expansion of promotion and tenure criteria (e.g., categories for community engagement and DEIJ work); recognition of integration of teaching/service/research; recognizing planning, engagement and other types of grants as worthwhile deliverables
• Equity audits

Among the unquestioned assumptions that maintain privilege are the superior “rigor” of quantitative versus qualitative designs — science is even thought to be “by its nature” quantitative — hence we set our review standards similarly. Similarly, there is a conservative bias in that those in power get to define quality (what “counts”) as well as when and how and who gets to be involved in those normative conversations. Questions fostering inclusion and change would push toward more holistic and expansive metrics; ex., success of students/faculty/staff whom we have mentored; contributions to the institution and community (can we elevate service more?). 

A two-tiered system (research vs teaching faculty) perpetuates the hierarchy even while ostensibly recognizing contributions that are not traditional. These hierarchies of merit and reward, including status and financial rewards, are present in hiring, promotion, funding agencies, publishers (e.g., journal rankings) and most of the peer review systems surrounding those arenas.