Great piece by jeffreybrainard.bsky.social with comments from others www.science.org/content/arti...
Can AI help identify high-volume, low-quality, “questionable” scientific journals (which some, controversially, call #predatoryjournals )? Authors of this new study emphasize aiding not replacing human evaluators of these journals. @science.org www.science.org/content/arti...
Take a look at the full paper:
- science.org/doi/10.1126/...
I started this project 5 years ago. Paper with my former students Han Zhuang and Lizhen Liang
Disclaimer: I am the founder of
ReviewerZero AI (www.reviewerzero.ai),
where we help with research integrity issues.
A few takeaways:
- the AI of course still makes mistakes, but we believe those mistakes are worth it
- predictions should be part of a triage system with experts. we make our predictions "interpretable", aligning with DOAJ guidelines
- But AI for integrity is here to stay
Applied to journals that haven't been vetted before , we predict more than 1000 new potentially questionable journals.
They have collectively published over 500K articles, cited millions of times, and acknowledged major funders in US, China, and Japan
We tried several methods, including simple regression, random forest and deep learning.
- Bibliometric features alone: PRC AUC ≈0.64
- Combined model: PRC AUC ≈0.79
Agreement with DOAJ guidelines and expert reviewers was strong, though false positives remain.
We trained models on ~15,000 journals labeled by DOAJ (12,869 legitimate vs 2,536 removed). Features included:
- Website content (editorial boards, policies)
- Website design (HTML structure, screenshots)
- Bibliometrics (citations, author metrics)
🚨New paper🚨
Open access has expanded science’s reach but also fueled the rise of "questionable" journals. Manual vetting can’t keep pace with thousands of titles and bad actors who adapt quickly. Wrong incentives too strong.
In a new Science Advances paper we ask: can AI help?
that makes sense! 🧠
Who are these hundreds of people who started following me recently? I mean, hi 👋, but how did you find me without any advertisement from my part? 😅 This place is starting to look great.
NotebookLM podcast version of the paper. Pretty good! notebooklm.google.com/notebook/d75... (need a Google account)
Excited to share a new article with Han Zhuang in Quantitative Studies of Science! "Incorporating costs and benefits to the evaluation of uncertain research results," we propose a unified framework for deciding when to continue research, even if it might be wrong 🤯. direct.mit.edu/qss/article/...
Tenure/tenure-track faculty positions in NLP in CS. Looking for people in science of science! Super interdisciplinary dept, great place to work and grow, amazing location. Learn more jobs.colorado.edu/jobs/JobDeta... Feel free to reach out and ask about the position and details.
