Expected Parrot

Let us know if you need help getting started and any feature requests! www.expectedparrot.com/getting-star...

Here's a quick demo:

We've been adding collaboration features that let teams work on AI+human surveys in code and interactively at once.

Eg, you can create a survey with our open-source package EDSL or using our no-code builder, and invite collaborators to contribute using whichever approach works for them:

Bring your own keys or mix and match A key feature of Expected Parrot is that you can always use your own keys for LLMs and human participant platforms, or you can use your Expected Parrot key for unified access.

A key feature of our platform is that you can choose whether to use your own API keys for LLMs and human participant platforms, use an Expected Parrot key for unified access, or mix and match keys as desired:
blog.expectedparrot.com/p/bring-your...

We've made some big improvements for running LLM experiments and hybrid AI + human surveys on our platform! Update your EDSL version to get all the latest features:

Validate your LLM answers with real respondents Here's a quick example of methods for generating a web-based version of your LLM survey and analyzing human and LLM responses together at your workspace.

Sometimes you need to check your LLM answers with real respondents. Our tools make it easy to launch a web-based version your LLM survey, combine human and LLM results, and analyze them at your dashboard or workspace. Here's a quick example: blog.expectedparrot.com/p/validate-y...

When you conduct LLM experiments at Expected Parrot 🦜 your prompts and responses are automatically cached, so you--and everyone you share your research with 🌎--can retrieve them for free 💸 and verify that you ran them ✅.

How to retrieve, combine & query your saved results with SQL Quick methods for accessing and analyzing your automatically stored results from anywhere.

New blog post: Quick methods for retrieving, combining & analyzing results
blog.expectedparrot.com/p/how-to-ret...

3. Share your survey. When anyone reruns your prompts they can retrieve your cached responses for free.

2. Run it at Expected Parrot. Your prompts and responses are automatically cached. www.expectedparrot.com/getting-star...

1. Create a survey in EDSL, our open-source Python package for simulating experiments with AI agents and LLMs. github.com/expectedparr...

It's a free shared resource that we created because research should be easy to share and replicate.

How it works:

🚀 Over 1 million Q&A cached and counting The Universal Remote Cache now has over a million unique LLM prompts and responses. You can retrieve them at no cost whenever you replicate research at Expected Parrot.

🚀 Over 1 million Q&A cached and counting!

The Universal Remote Cache now contains over a million unique LLM prompts and responses. You can retrieve them at no cost whenever you replicate research at Expected Parrot.
blog.expectedparrot.com/p/over-1-mil...

Your account comes with $25 in credits for API calls to LLMs for getting started! www.expectedparrot.com

You can check current model pricing and performance daily:

* Methods for estimating tokens and costs for every prompt
* Dashboard pages for tracking actual costs for every model and survey job

New features for monitoring your AI research costs Our tools are designed to make it easy to understand and monitor your LLM usage. Learn more about our latest features for precisely tracking your costs.

📢📢📢 New features for monitoring your AI research costs

Our tools are designed to make it easy to understand and monitor your LLM usage. Learn more about our latest features for precisely tracking your costs: blog.expectedparrot.com/p/new-featur...

* Unified access to language model providers with a single API key and universal methods—no software engineering required

* Automatically formatted datasets of responses eliminate manual data cleaning work

* Built-in methods for analyzing, reproducing and exporting results

I want to see how OpenAI's o3 compares to... EDSL is an open-source tool that lets you readily compare performance for many language models at once.

Several key features of EDSL make it a convenient tool for comparing the performance of language models:

blog.expectedparrot.com/p/i-want-to-...

Have an idea for an LLM experiment? Here are some reasons to run it at Expected Parrot, and easy ways to get started.

Have an idea for an LLM experiment?
Here are some reasons to run it at Expected Parrot, and easy ways to get started.
blog.expectedparrot.com/p/have-an-id...

What are my options for running EDSL surveys? Here's a quick recap of features for specifying where and how to run your LLM-based surveys in EDSL.

What are my options for running EDSL surveys?

Here's a quick recap of features for specifying where and how to run your LLM-based surveys in EDSL. You can bring your own API keys for LLMs or use an Expected Parrot key to access all available models at once.
blog.expectedparrot.com/p/what-are-m...

Show prompts, please! EDSL makes it easy to tinker with prompts, compare responses from different language models, and replicate experiments by automatically storing all your prompts, responses and code to reproduce them.

Show prompts, please!
EDSL makes it easy to tinker with prompts, compare responses from different language models, and replicate experiments by automatically storing all your prompts, responses and code to reproduce them.
blog.expectedparrot.com/p/show-promp...

We launched a GPT for generating EDSL survey code - please give it a spin and let us know what you think!
chatgpt.com/g/g-67d17648...

How do I control AI agent memory when answering my questions? EDSL provides a variety of features for adding context of prior questions to new questions, piping answers into other questions, and asking multiple questions at once in a structured way.

How do I control AI agent memory when answering my questions?
EDSL provides a variety of features for controlling the context of information presented at each step of your experiment.
blog.expectedparrot.com/p/how-do-i-c...

"Surveys" sounds niche and not what I need Totally fair, but many research tasks and experiments can be designed as survey-like interactions.

"'Surveys' sounds niche and not what I need"
Totally fair! But many research tasks and experiments can be designed as survey-like interactions. Please let us try to convince you: blog.expectedparrot.com/p/surveys-so...

How to use a language model to label your data I have a large collection of [texts, images, PDFs, CSVs, tables, transcripts ... ] that I need to [sort, tag, extract, clean, analyze, synthesize, summarize ... ].

Whether or not your ultimate research goal has anything to do with language models, using a model to perform data labeling tasks can be very helpful to your process. Here's a quick example of how to do it in EDSL: open.substack.com/pub/expected...

Do you owe someone an expense report?

Here's a notebook for reading a statement, identifying reimburseables, and drafting a memo, easily modified for your own content and purposes: www.expectedparrot.com/content/Robi...

[6/6] Check out our docs for downloadable examples that you can modify for your own data labeling tasks: docs.expectedparrot.com

[5/6] When you share your survey, others can retrieve your responses and reproduce your results at no cost. Our universal remote cache of stored responses is free and available to all users. Learn more about how it works: docs.expectedparrot.com/en/latest/re...