Common Crawl Foundation

Common Crawl - Blog - IETF 123 Report A report on IETF 123 in Madrid, including sessions on AI content preferences, bot authentication, and web measurement.

A report on IETF 123 in Madrid, including sessions on AI content preferences, bot authentication, and web measurement.

commoncrawl.org/blog/ietf-12...

Common Crawl - Blog - Host- and Domain-Level Web Graphs May, June, and July 2025 Our Web Graph release for July 2025 is now available, consisting of 481.6 million nodes and 3.4 billion edges at the host level, and 209.5 million nodes and 2.6 billion edges at the domain level.

Our Web Graph release for July 2025 is now available, consisting of 481.6 million nodes and 3.4 billion edges at the host level, and 209.5 million nodes and 2.6 billion edges at the domain level.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - July 2025 Crawl Archive Now Available The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content.

The crawl archive for July 2025 is now available. Crawled between July 7th and July 21st, the data contains 2.42 billion web pages, or 419 TiB of uncompressed content.

commoncrawl.org/blog/july-20...

Common Crawl - Blog - WMDQS Shared Task on Language Identification The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identi...

The Common Crawl Foundation, MLCommons, EleutherAI, and John Hopkins' Center for Language and Speech Processing have the pleasure of inviting you to register for the 1st shared task on Language Identification for web data.

commoncrawl.org/blog/wmdqs-s...

Unlocking data to advance European commerce and culture - Microsoft On the Issues Microsoft launches 2 initiatives to open Europe’s languages and culture, building on AI, cloud, and digital sovereignty commitments.

"MOIC will also partner with Common Crawl, one of the largest free and open repositories of web crawled data. MOIC will fund work at Common Crawl, leveraging native speakers to annotate and seed European language data in the publicly available Common Crawl data set."

Common Crawl - Blog - The First WMDQS-Masakhane LangID Hackathon In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotation...

In June 2025 the Common Crawl Foundation, MLCommons, and EleutherAI had the pleasure of hosting a virtual hackathon in partnership with Masakhane in order to collect language identification annotations for African languages.

commoncrawl.org/blog/the-fir...

Common Crawl - Blog - Host- and Domain-Level Web Graphs April, May, and June 2025 We are pleased to announce that the Web Graph for June 2025 is now available. The graph consists of 371.6 million nodes and 3.1 billion edges at the host level, and 161.8 million nodes and 2.2 billion...

We are pleased to announce that the Web Graph for June 2025 is now available. The graph consists of 371.6 million nodes and 3.1 billion edges at the host level, and 161.8 million nodes and 2.2 billion edges at the domain level.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - Common Crawl at the United Nations Open Source Week, June 2025 The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open s...

The Common Crawl Foundation team took part in the United Nations Open Source Week in New York City this June, meeting with global developers, researchers, and policymakers to discuss all things open source and AI.

commoncrawl.org/blog/common-...

Common Crawl - Blog - June 2025 Crawl Archive Now Available We are pleased to announce that the crawl archive for June 2025 is now available.

We are pleased to announce that the crawl archive for June 2025 is now available.

www.commoncrawl.org/blog/june-20...

Common Crawl - Blog - May/June 2025 Newsletter We're happy to share our newsletter for May/June 2025 with updates from our team.

We're happy to share our newsletter for May/June 2025 with updates from our team.

commoncrawl.org/blog/may-jun...

The deadline for paper submissions has been extended!

The new deadline is July 3, 2025. AoE.

For more information, please visit: wmdqs.org

The AI Alliance Forms Non-profit AI Lab and AI Technology & Advocacy Association to Scale Open-Source Innovation /PRNewswire/ -- Today, the AI Alliance, a global collaboration of more than 180 organizations committed to AI open innovation, announced it has incorporated...

The AI Alliance Forms Non-profit AI Lab and AI Technology & Advocacy Association to Scale Open-Source Innovation

www.prnewswire.com/news-release...

Common Crawl - Blog - Announcing the Whirlwind Tour of Common Crawl's Datasets using Python Announcing a refreshed version of the Whirlwind Tour in Python. Get to know how to make the most of our crawl data.

Announcing a refreshed version of the Whirlwind Tour in Python. Get to know how to make the most of our crawl data.

commoncrawl.org/blog/announc...

AI Alliance @ IBM One Madison (UN Open Source Week 2025) · Luma This year’s UN Open Source Week 2025, June 16-20) will once again bring together a global “who is who” of Open Source leaders. As part of the official…

The Common Crawl Foundation, together with IBM, the AI Alliance, and BrightQuery will be hosting an "UN Conference" at IBM's new flagship NYC HQ at One Madison Avenue on Friday, June 20, from 12:30-5pm.

If you are in NYC, it would be great to see you there!

lu.ma/p0a1scde

Common Crawl - Blog - Host- and Domain-Level Web Graphs March, April, and May 2025 We are pleased to announce that the Web Graph for May 2025 is now available. The graph consists of 326.8 million nodes and 2.9 billion edges at the host level, and 156.1 million nodes and 2.1 billion ...

We are pleased to announce that the Web Graph for May 2025 is now available. The graph consists of 326.8 million nodes and 2.9 billion edges at the host level, and 156.1 million nodes and 2.1 billion edges at the domain level.

commoncrawl.org/blog/host--a...

Common Crawl - Blog - May 2025 Crawl Archive Now Available We are pleased to announce that the crawl archive for May 2025 is now available. The data was crawled between May 11th and May 25th, and contains 2.47 billion web pages, or 429 TiB of uncompressed con...

We are pleased to announce that the crawl archive for May 2025 is now available. The data was crawled between May 11th and May 25th, and contains 2.47 billion web pages, or 429 TiB of uncompressed content.

commoncrawl.org/blog/may-202...

Call for papers!
We are organising the 1st Workshop on Multilingual Data Quality Signals with @mlcommons.org and @eleutherai.bsky.social, held in tandem with @colmweb.org. Submit your research on multilingual data quality!

Submission deadline is 23 June, more info: wmdqs.org

The #iipcGA25 Research Working Group meeting was packed–co-chair Ben O’Brien of @nationallibrarynz.bsky.social discussed Whole-of-Domain, Magnus Birkenes and Marie Roald of @nettarkivet.bsky.social explained their WebData Project, and Sebastian Nagel presented @commoncrawl.bsky.social’s datasets!

We would like to welcome all of our attending members to Oslo, with a special welcome to two of our newest members, the Publications Office of the European Union and @commoncrawl.bsky.social!

@nettarkivet.bsky.social | #iipcGA25 | #webarchiving

Introducing GovArchive.us & Mirroring Entire Sites with Web Archives • Webrecorder Blog Introducing GovArchive.us and tooling to mirror web sites using web archives.

Our friends at Webrecorder have announced the launch of GovArchive.us, a dedicated site for exploring their US Government Web Archive on Browsertrix. More details in their blog post: