When promoting professors, research productivity matters—at some universities more than others

From ScienceMag:

After 10 years working as a chief technical officer in industry, Boon Han Lim decided it was time to go back to academia. That had always been his dream. But he soon found his research wasn’t advancing as quickly as he’d hoped. One reason, he suspected, was that he was under constant pressure to meet specific benchmarks, such as publishing in high-impact journals.

Lim, now an associate professor at the University Tunku Abdul Rahman, wished that evaluations of his job performance took a more holistic view of his work. And he’s not alone. Researchers around the world have voiced complaints about having to prove their worth through simplistic measures of their productivity. It’s an issue that particularly impacts lower income countries, according to a new study published this week in Nature.

“Unfortunately, quantitative … evaluation metrics continue to play a predominant role in developing countries,” says Laura Rovelli, an independent researcher at Argentina’s National Scientific and Technical Research Council (CONICET) who has studied research evaluation practices. Still, crafting more diverse and inclusive ways to evaluate research can be challenging, she says, as there’s no standard way to do it.

Lim co-led the study after joining the Global Young Academy, a society seeking to give early-career scientists throughout the world a voice. After speaking with other members, he realized he wasn’t the only one frustrated with academia’s assessment process; many wanted to see a broader range of contributions, such as societal impacts of their research, be counted. So he joined the academy’s group that was exploring how different countries evaluate researchers for promotion and led the study from 2018 to 2020, after which Yensi Flores Bueso, a researcher at University College Cork, took charge. “I said, ‘Right, I have a problem. So I also think I should contribute a little bit of my effort to [solve] it.’”

The study, which was authored by researchers around the globe, analyzed policies for determining whether a scholar should be promoted to full professor at 190 academic institutions and 58 government agencies in 121 countries. Overall, the team found that 97% of policies mentioned research outputs. Most also mentioned a scholar’s teaching (93%), funding (79%), mentoring (75%), awards (69%), and community service (63%). But the policies varied widely between universities, and some put more emphasis on specific metrics than others.      

The findings also revealed a link between a country’s average per capita income and how institutions decide what researchers get promoted. Policies in high-income countries put more emphasis on a scholar’s visibility, including engagement with the academic community and the general public as well as the number of awards they received. Low-income and upper middle–income countries, in contrast, focused more on publication metrics, such as how many papers a researcher published and how many times those papers were cited.

It’s possible that some institutions rely on quantitative assessments of research productivity to an even greater degree than was detected in the study. Lim’s team was limited to looking at the public policies that were available; they didn’t review the evaluation processes themselves, so they couldn’t tell how much evaluators—often other professors at a university—were relying on publication metrics in the background to make their decision. “Even if the policy is silent about the use of metrics, the external reviewers may refer to them, or not, depending on the disciplinary norms,” says Lisa Wolf-Wendel, associate dean for research and graduate studies in the School of Education & Human Sciences at the University of Kansas.

Rovelli hopes the new study will start a dialogue about differences between countries and how institutions should think about developing new practices. She notes that in 2022, CONICET added a requirement that scientists working for the agency submit a narrative resume during their promotion evaluation process, which gives them an opportunity to discuss their professional trajectory and the contributions of their research. The Coordination for the Improvement of Higher Education Personnel in Brazil––the agency responsible for training higher education staff in the country––also recently added an “impact on society” category to its promotion evaluation process to evaluate the social, environmental, and economic benefits of a researcher’s work.

Despite the persistence of policies focused on publications, Lim says he reminds scientists around the world, especially those starting out, to stop thinking of themselves as paper generators. “This to me is not that healthy,” he says. He decided not to rush in his career to achieve high-productivity metrics during evaluations. “I have several projects which have been carried out for more than 5 years but yet to publish,” he says, “because I want to come out with more solid contributions.” He often reminds his colleagues and students: “Go do what you are interested in … and what you feel can impact the world.”

Read More