Aaron Tay

For SR what i mean isn't the actual expertise in scoping, crafting search strategies etc that requires a lot of practice and experience. I mean the concept of SR, the general conducting standards from Cochrane, etc. I am not even a SR librarian and I spot dozens of "SR" that are poorly done.

There *is* hype yes but we need to really study such things and not fall to confirmation bias. And I find when a new good academic ai tools emerges it spreads through the academic grapevine very quickly... we look out of touch when we keep ranting about chatgpt etc

SR is an edge case where most academics can't reborthered to master and if they were serious its not hard for them. Anyway in my context I was talking about using deep research for narrative LR where they are as expert as anyone including librarians.

They are literally telling me they have utility in THEIR field. Eg doing real research not just literature review. Most of them do narrative LR. And let's be frank literature review isn't hard just troublesome

I can understand concerns about environment, equitabllity, copyright. The part that annoys me is people saying its useless or people using it are falling for hype

Semantic Scholar | AI-Powered Research Tool Semantic Scholar uses groundbreaking AI and engineering to understand the semantics of scientific literature to help Scholars discover relevant research.

As a first approximation go to www.semanticscholar.org & search. Are results okay? Or do known item search to see if they are indexed. If it is okay, elicit, undermind, Consensus, Scispace etc etc will do ok (2)

Yes the main issue with these tools is source not so much algo. Pretty much all of them including elicit, undermind draw from the same open sources eg Semantic Scholar. Preprints isnt a issue, they got them. Is paywall stuff in domains that dont do preprints, monograph heavy fields etc . (1)

Everyone from Anthropic, Google, xai/grok has a premium 200++ tier now. Most people? it's just a waste of money since we can't tell the difference between the 20 and 200 model. But if you need hard math problem..coding professionally..? Yeah 200 might be worth it

Me? I only have o3 not o3 pro. I basically couldn't tell the difference really because I don't have hard enough problems

Math and coding. Eg you a economics or law guy doing theoretical modelling. The frontier models o3 pro/gemini 2.5 pro deep think are at the level where they can help. Of course they are still not perfect but good enough that they have been helpful to even some research level work

The 200pm isn't for literature review. That one a 20 per month sub in elicit, undermind or similar specialized tool will do :)

Just to add by academic deep research i dont mean openai/gemini/perplexity deep research. I wouldn't use those either and most faculty i know wouldn't either after trying

All I can say is if your experience of deep research is only via general DR like openai/gemini/perplexity I can't blame you for being skeptical.

Again let's pretend these professors from the top universities in the world are dumb and spend money based on faith. My experience? They are ruthlessly pragmatic. If it doesn't help them they won't use it as it wastes time much less pay so much for it

There is a digital divide yes. If you are a assistant prof in a top 50 world ranked uni, 200 a month is just a drop in the bucket from their research funds. The fact they doing it is a strong signal, it is useful in STEM etc. Or are we pretending these people are dumb and just caught in the hype?

I would say the best ones are at the stage they can cut down the time you spend orientating yourself in a new area. You are not supposed to copy and paste it to your LR! People who do that are not serious.

I would say up to this year I dont know of any serious solution thar was really meant to automate SR. They only started emerging this year.

Elicit: The AI Research Assistant Use AI to search, summarize, extract data from, and chat with over 125 million papers. Used by over 2 million researchers in academia and industry.

What is working seems to be either elicit style brute force screen top 500 results or agentic deep search style where LLM steers search with iterative search but even then.. none of the general deep research Gemini/openai/perplexity deep research are even meant for acad lit review much less SR(5)

What didn't work as of now is using chatgpt to formulate boolean. They just dont know enough about MeSH and don't "reason" well enough & end of producing high precision low recall strategies (4)

What did work and was promising is that even with gpt3.5 they were promising as screeners. And a studies now suggest the latest 4.1 might even be pretty much human level. (3)

I am so amused by so much "research" thinking the single shot embedding search could find all the papers in a serious gold standard SR. How is that "fair" when the human searcher gets to pilot and test search queries and then screens for hundreds of hours & you think ONE magic embedding will work(2)

I really should write a long blog post addressing this misconception. None of the off-the-shelf tools until this year were designed to do SR. What people did was to consider if embeddings search could complement boolean search (1)

The notebookLM is either finetuned or has a system prompt to make it very faithful to the source and avoid relying on pretrained knowledge. But this means it might be less "smart" in general . So one simple test, upload a pdf into NotebookLM and Ai google studio and ask it to critique(2)

attended a workshop where a researcher showed his workflow for Notebook LM. It is well known NotebookLM LLM is extremely faithful the source (my own tests shows example where no other LLM not even Gemini 2.5 pro i tested can pass except NotebookLM ). But i suspect there is a trade-off (1)

I can tell you, it goes beyond literature review, many are using frontier LLMs ($200++ per month models) to help with research. They tell me they hope this idea that LLMs are useless is the prevailing opinion among their peers because this gives them a competitive advantage :)

Undermind & academic deep research tools are heavily used by Prof & serious Phd students. Highly skeptical & serious people who don't have time for nonsense. Maybe they are all clueless? After all Librarians who barely use them already decided on principle "stochastic parrots" cant be useful? sigh

I guess what's im saying is using LLM answers as just one source of evidence to try to triangulate with other sources is probably fine. The LLM might be not fully correct or just wrong but you run similar risks reading blog posts from me.

I probably misunderstanding stuff even now (me from 2023 misunderstood even more so is like the 2027 me will laugh at 2025 me) but it is hard to blame LLMs for that because I likely would have misunderstood the same if not more anyway if I just restricted myself to usual sources without LLM

Truth is i use LLMs primarily o3 to explain information retrieval concepts. It's not the only thing I use of course. I watch YouTube sessions by professors, read industry blogs, arxiv etc but I also supplement by asking o3 things i am fuzzy about and it helps a lot. Is this problematic?

Eg under limitations "this thing we did? Probably doesn't measure what we wanted cos x" if they truly embarrassed this would try hide that in last paragraph:)