Repositories list

• web-languages

  Public
  Crowd-sourced lists of urls to help Common Crawl crawl under-resourced languages. See https://github.com/commoncrawl/web-languages-code/ for the code

  crawlinglanguage-detectiondataset

  crawlinglanguage-detectiondataset

  8888 forks•6868 stars•33 issues•11 pull request•Updated May 1, 2026May 1, 2026

• cc-index-table

  Public
  Index Common Crawl archives in tabular format

  sqlsparkcolumnar-storage+ 3

  sqlsparkcolumnar-storageaws-athenaapache-parquetcommoncrawl

  Java

  •

  Apache License 2.0

  •1515 forks•128128 stars•66 issues•77 pull requests•Updated Apr 30, 2026Apr 30, 2026

• cc-webgraph-statistics

  Public
  Statistics of Common Crawl monthly Web Graphs

  JavaScript

  •

  Apache License 2.0

  •11 fork•55 stars•00 issues•00 pull requests•Updated Apr 30, 2026Apr 30, 2026

• crawl-openathena

  Public
  00 forks•00 stars•00 issues•00 pull requests•Updated Apr 28, 2026Apr 28, 2026

• cc-crawl-statistics

  Public
  Statistics of Common Crawl monthly archives mined from URL index files

  statisticscommoncrawlcommon-crawl

  statisticscommoncrawlcommon-crawl

  Python

  •

  Apache License 2.0

  •1616 forks•219219 stars•33 issues•11 pull request•Updated Apr 27, 2026Apr 27, 2026

• cc-downloader

  Public
  A polite and user-friendly downloader for Common Crawl data

  rustdownloadercommoncrawl

  rustdownloadercommoncrawl

  Rust

  •

  Apache License 2.0

  •55 forks•8080 stars•99 issues•22 pull requests•Updated Apr 24, 2026Apr 24, 2026

• cc-warcinfo-index-builder

  Public
  Code to build an index that maps warcinfo-id to crawl / warc

  Python

  •00 forks•00 stars•00 issues•00 pull requests•Updated Apr 24, 2026Apr 24, 2026

• news-crawl

  Public
  News crawling with StormCrawler - stores content as WARC

  crawlernewsweb-crawler+ 5

  crawlernewsweb-crawlerapache-stormwarccommoncrawlcommon-crawlstorm-crawler

  Java

  •

  Apache License 2.0

  •4141 forks•366366 stars•1414 issues•44 pull requests•Updated Apr 21, 2026Apr 21, 2026

• whirlwind-java

  Public
  A whirlwind tour of Common Crawl's data using Java

  Java

  •

  Apache License 2.0

  •00 forks•33 stars•11 issue•00 pull requests•Updated Apr 20, 2026Apr 20, 2026

• nutch

  Public
  Common Crawl fork of Apache Nutch

  javabig-datahadoop+ 2

  javabig-datahadoopweb-crawlercommoncrawl

  Java

  •

  Apache License 2.0

  •1.3k1.3k forks•4141 stars•1111 issues•33 pull requests•Updated Apr 20, 2026Apr 20, 2026

• robotstxt-experiments

  Public
  How is the Robots Exclusion Protocol (robots.txt) used in the WWW? This projects tries to get some insights mining Common Crawl's robots.txt captures of the yea…

  robots-txtrobotstxt

  robots-txtrobotstxt

  Jupyter Notebook

  •

  MIT License

  •00 forks•00 stars•00 issues•00 pull requests•Updated Apr 17, 2026Apr 17, 2026

• cc-quick-scripts

  Public
  Scripts to verify Common Crawl segments and WARC/WET/WAT files

  internal-tools

  internal-tools

  Python

  •

  MIT License

  •44 forks•22 stars•00 issues•00 pull requests•Updated Apr 14, 2026Apr 14, 2026

• cc-host-index

  Public
  Tools for working with the host index

  Python

  •44 forks•1313 stars•11 issue•11 pull request•Updated Apr 14, 2026Apr 14, 2026

• whirlwind-python

  Public
  A whirlwind tour of Common Crawl's data using Python

  pythontutorialarchive+ 1

  pythontutorialarchivewarc

  Python

  •

  Apache License 2.0

  •99 forks•4444 stars•00 issues•11 pull request•Updated Apr 13, 2026Apr 13, 2026

• cc-index-annotations

  Public
  Example code to join an annotation to a host or url index

  Python

  •00 forks•11 star•00 issues•00 pull requests•Updated Apr 13, 2026Apr 13, 2026

• crawler-commons

  Public
  A set of reusable Java components that implement functionality common to any web crawler

  Java

  •

  Apache License 2.0

  •9090 forks•22 stars•00 issues•22 pull requests•Updated Apr 7, 2026Apr 7, 2026

• eot2020-host-index

  Public
  Tools to work with the preliminary End of Term Archive host index

  Python

  •00 forks•00 stars•00 issues•00 pull requests•Updated Mar 30, 2026Mar 30, 2026

• cc-webgraph

  Public
  Tools to construct and process Common Crawl webgraphs

  pagerankwebgraphcommoncrawl+ 3

  pagerankwebgraphcommoncrawlcommon-crawlcentrality-measureswebgraph-framework

  Java

  •

  Apache License 2.0

  •66 forks•108108 stars•88 issues•00 pull requests•Updated Mar 26, 2026Mar 26, 2026

• cc-pyspark

  Public
  Process Common Crawl data with Python and Spark

  sparkpysparksparksql+ 5

  sparkpysparksparksqlwetcommoncrawlcommon-crawlwarc-fileswat-files

  Python

  •

  MIT License

  •9494 forks•454454 stars•44 issues•22 pull requests•Updated Mar 26, 2026Mar 26, 2026

• cdx_toolkit

  Public
  A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine

  pythonwarcweb-archiving+ 4

  pythonwarcweb-archivingcdxweb-archivescommoncrawlcdx-api

  Python

  •

  Apache License 2.0

  •3434 forks•204204 stars•66 issues•44 pull requests•Updated Mar 23, 2026Mar 23, 2026

• ipv6-analysis

  Public
  A survey of IPv6 support among the 100k top ranked hosts on the web

  Python

  •

  Apache License 2.0

  •00 forks•11 star•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• warcio-s3

  Public
  Streaming WARC/ARC library for fast web archive IO

  Python

  •

  Apache License 2.0

  •6969 forks•00 stars•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• cc-citations

  Public
  Scientific articles using or citing Common Crawl data

  bibtexbibliographyopendata

  bibtexbibliographyopendata

  Jupyter Notebook

  •44 forks•2828 stars•00 issues•00 pull requests•Updated Mar 19, 2026Mar 19, 2026

• cc-nutch-example

  Public
  Apache Nutch example project to archive content in WARC files

  Shell

  •

  Apache License 2.0

  •22 forks•33 stars•00 issues•00 pull requests•Updated Mar 17, 2026Mar 17, 2026

• awesome-low-resource-languages

  Public
  This list provides resources useful for documenting, conserving, developing, preserving, or working with endangered and low resource languages.

  00 forks•00 stars•00 issues•00 pull requests•Updated Mar 16, 2026Mar 16, 2026

• cc-web-graph-neo4j

  Public
  Instructions and code for using the Common Crawl Web Graph in Neo4j format

  Shell

  •00 forks•22 stars•00 issues•00 pull requests•Updated Mar 16, 2026Mar 16, 2026

• cc-warc-examples

  Public
  CommonCrawl WARC/WET/WAT examples and processing code for Java + Hadoop

  javahadoopmapreduce+ 1

  javahadoopmapreducecommoncrawl

  Java

  •

  MIT License

  •4545 forks•3838 stars•00 issues•00 pull requests•Updated Mar 12, 2026Mar 12, 2026

• ia-web-commons

  Public
  Web archiving utility library

  cdx-fileswarc-fileswat-files

  cdx-fileswarc-fileswat-files

  Java

  •

  Apache License 2.0

  •7676 forks•1111 stars•44 issues•11 pull request•Updated Mar 11, 2026Mar 11, 2026

• language-detection-cld2

  Public
  Natural language detection, Java bindings for CLD2

  natural-languagelanguage-detectionlanguage-identification

  natural-languagelanguage-detectionlanguage-identification

  Java

  •

  Apache License 2.0

  •22 forks•1717 stars•33 issues•11 pull request•Updated Feb 26, 2026Feb 26, 2026

• cc-citations-paper-explorer

  Public
  A visual paper explorer based on cc-citations. https://huggingface.co/spaces/commoncrawl/cc-citations

  embeddingsldacommoncrawl+ 1

  embeddingsldacommoncrawlscincl

  Python

  •

  Apache License 2.0

  •00 forks•00 stars•00 issues•00 pull requests•Updated Feb 19, 2026Feb 19, 2026