@MattBruenig

It's worse than that. They are taking my money!

I've been following this project for awhile with some interest. I think the real heavy lift here is creating the subject matter database, and making it usable enough that the llm can effectively follow citations and generate hyperlinks. It's very cool!

I've approved you

let's find someone who knows about the NLRA and have them prepare a set of questions and the answers they expect, and then I will run the questions through the bot and see if they match (this is how I have evaluated it of course since I know a lot about the NLRA)

let's find someone who knows about the NLRA and have them prepare a set of questions and the answers they expect, and then I will run the questions through the bot and see if they match (this is how I have evaluated it of course since I know a lot about the NLRA)

You should give it a try and tell me what you think

When this happens, what it actually does is say that some precedent says X while other precedent says Y. It is also able to pick up overturning of precedent over time and reports that accurately as well. I've seen it!

Yes it's a research tool. Indeed, it will only produce legal memos, not any other kind of document, like a brief. And it's absolutely the best way to start any particular research you want to get into. You can use the memo as a jumping off point if you want to do more.

the subagent writes a python script to find the exact quotation string in the .jsonl file, i watched it do it! what is the other point about propositions? You are saying i need to run it after the memo not before?

Since all the sources are already downloaded in advance and stores locally in .jsonl files, it is very straightforward to have it go through them

Yeah I've already got it working it looks like. Before it writes the memo, it writes this JSON file that lists every quotation and proposition and corresponding citation. Then it spawns a subagent with clean context window to verify each one (quotes deterministically, propositions in its LLM way).

Quotes can be verified deterministically because when it queries the database, it stores the JSON search results (which includes full text documents) locally. So that part will be easy. Proposition checks more fuzzy but will run it through them anyways.

Ok I am implementing this now. The thing already does a "citation audit" before it writes the memo. But that is to check that the links work. I will have it create a .json file with citation->proposition pairs and then spawn subagent to check those after the citation audit.

Yes I did something like this for my book to sort through the text after it was written.

To be clear, what my tool does is produce a research memo. Totally different situation if what you are doing is drafting something for submission to the NLRB or a court. This would never be submitted anywhere. If you were to write something based on it, you'd click through to the sources.

Sad to see people not understand the difference between asking an LLM to answer a question from its training data and using an LLM to agentically search a database to gather materials that it uses to answer the question. But it's also whatever. People who use it see what it does.

screenshot of claude

Need to answer a question about labor law?

@mattbruenig.bsky.social built a Claude skill that writes a legal memo on any NLRA/NLRB-related topic with specific links to decisions, court opinions, manuals, and agency memos

YouTube video by Matt Bruenig My Fully Automated Labor Law Research Tool Is Finally Here

now we're talking. a claude skill by @mattbruenig.bsky.social built on top of stuff that required a bunch of coding, but which itself has nothing to do with coding. expert-level (at least) labor law research, roughly for free, built on top of nlrbresearch.com/NLRB/NLRB_DB

This is very good from Matt and it's probably the best non-software use case rn because he's

- self-supervised
- able to build a strong database with clear ground truth
- an expert who can check his own work
- doing something with an obvious end state

That is really what I was looking for with my spot checking and did not find any instances of it. But I'd bet there is at least one in there somewhere. Subsequently I have thought maybe I should have another LLM like GPT check all the citations for accurate descriptions. Maybe for future versions.

They weren't random. I was just kind of skimming around. I actually don't worry much at all about that particular error. I have been doing these summaries for so long. I know they are very accurate. Only thing I really thing could be an issue is Sonnet citing case 1 for something case 2 said.

My e-commerce "app" (if you can call it that) is a python file that listens for a stripe webhook and then once received calls the resend API to e-mail the three documents. It's like 50 lines of code. No customer data is stored on the server. Stripe and Resend have it.

My e-commerce "app" (if you can call it that) is a python file that listens for a stripe webhook and then once received calls the resend API to e-mail the three documents. It's like 50 lines of code. No customer data is stored on the server. Stripe and Resend have it.

When I was in college, a friend preparing for a trip to East Africa wanted to learn Swahili. Instead of using a textbook, he downloaded the entire Swahili-language Wikipedia and wrote a program to find every unique word, count their occurrences, and sort them by frequency. This approach allowed him to focus his studies on the most common words first. This strategy, while limited for in-depth language study, suited his practical goal: to quickly acquire the most useful Swahili for everyday conversation and immersion. In this book, I use a similar method to provide an introduction to private sector labor law as governed by the National Labor Relations Act (NLRA) and administered by the National Labor Relations Board (NLRB). Using data from the NLRB Research database, I determined which NLRB and Supreme Court cases have been cited the most since the NLRA’s enactment in 1935. By combining citation frequency with my own judgment as an experienced labor lawyer, I have created this text, which I hope can help individuals—especially law students, union staff, and rank-and-file workers—gain an initial footing in this area of law.

The National Labor Relations Book, a $10 introduction to private-sector labor law under the NLRA

Interesting example of something that wouldn't exist without LLMs: @mattbruenig.bsky.social used Gemini & Claude to summarize each of the 100 most-cited cases based on their 100 most recent citing cases

So for NLRB Edge, a lot of what you are having to do is hope the LLM figures out what parts of the case are important and what parts it can leave out. With this project, I tell the LLM exactly what is important (how they case applies a specific target case) and to only focus on that.

As Steve points out, I have been summarizing cases like this for years for NLRB Edge. The kind of summaries I asked Gemini/Sonnet to do for this project were *simpler* than what they do for NLRB Edge because they are just about a single aspect of a case, not the entire case.

So for NLRB Edge, a lot of what you are having to do is hope the LLM figures out what parts of the case are important and what parts it can leave out. With this project, I tell the LLM exactly what is important (how they case applies a specific target case) and to only focus on that.

As Steve points out, I have been summarizing cases like this for years for NLRB Edge. The kind of summaries I asked Gemini/Sonnet to do for this project were *simpler* than what they do for NLRB Edge because they are just about a single aspect of a case, not the entire case.

3. You can spot check by clicking the link and reading the skimming the underlying case to make sure it says what it is being cited for. I did this 100 times and found 0 errors.

4. Finally, the reader is able to read the underlying document as well before they rely upon anything in the text.

2. You can use code to follow the hyperlink of each citation and make sure the citation in the text matches the database entry it is linking to. There were ~10 errors out of 1,233, which I manually fixed.