A corpus of monthly n-grams generated from more than two billion English tweets

# A corpus of monthly n-grams generated from more than two billion English tweets

<br/>Elso Dittfeld,¹ [Robert Jäschke](https://hu.berlin/RJ/),¹ Dimitar Dimitrov²

¹ Humboldt-Universität zu Berlin<br/>² GESIS – Leibniz-Institute for the Social Sciences

March 18, 2026

<a rel="license"
href="http://creativecommons.org/licenses/by-sa/4.0/"><img
alt="Creative Commons Lizenzvertrag"
style="border-width:0;width:88px;margin:0;" src="images/cc.png"/></a>
<small style="width: 24em">
This work is licensed under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">Creative Commons Attribution-ShareAlike 4.0 International</a>.
</small>

Note:
- dataset result of Bachelor thesis of Elso Dittfeld
- joint work with Dimitar Dimitrov from GESIS

---

# Introduction

![a female scientist with a lab coat and a test
tube](images/woman-scientist-svgrepo-com.svg) 
![a balance to represent the legal system](images/balance-scale-svgrepo-com.svg)

<small>graphics by <a href="https://www.svgrepo.com" target="_blank">SVG Repo</a></small>

Note:
- Twitter/X interesting resource for researchers:
  - language evolution on social media
  - trends & topics
  - polarization of society
  - effects of algorithms (e.g., echo chambers)
- 🖱️  but legal constraints:
  - terms of service
  - copyright, licensing etc.
  - personal (privacy)

![Datasets with "tweets" on Zenodo](images/zenodo_tweets.png)

## Existing Approaches

### rehydration

is dead <em class="tonedown">([Assenmacher et al. 2023]((https://doi.org/10.36190/2023.56)))</em>

<br/>

### limited parts of data

- URLs <em class="tonedown">(e.g., doi:[10.5281/zenodo.154583](https://doi.org/10.5281/zenodo.154583))</em>
- @mentions <em class="tonedown">(e.g., doi:[10.5281/zenodo.3634154](https://doi.org/10.5281/zenodo.3634154))</em>
- #hashtags <em class="tonedown">(e.g., doi:[10.5281/zenodo.17082606](https://doi.org/10.5281/zenodo.17082606))</em>
- entities and sentiment scores <em class="tonedown">(e.g., [TweetsKB](https://data.gesis.org/tweetskb/), [Fafalios et al. 2018](https://doi.org/10.1007/978-3-319-93417-4_12))</em>
- ...

<br/>

→ rarely textual content

Note:
- astoundingly many "tweet" datasets on Zenodo and elsewhere
- often: just tweet ids
  - 🖱️  rehydration: get actual tweets from Twitter's API
	- 🖱️  since 2023 practically useless
- 🖱️  alternatives: limited parts of data

## Derived Text Formats

<em class="tonedown">([Schöch et al. 2020](https://doi.org/10.17175/2020_006))</em>

![animation of topic detection on a term-document matrix](https://upload.wikimedia.org/wikipedia/commons/6/6e/Topic_detection_in_a_document-word_matrix.gif)

<small>© [Christoph Carl Kling](https://commons.wikimedia.org/wiki/File:Topic_detection_in_a_document-word_matrix.gif) / [CC-BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/deed.en)</small>

</div>

- transform texts such that original can not be reconstructed
  - document-term matrices
  - n-grams <em class="tonedown">(e.g., [Google Books n-grams](http://commondatastorage.googleapis.com/books/syntactic-ngrams/index.html))</em>
  - word embeddings
- transformed texts useful for (some) research
  - e.g., topic modelling, authorship classification

→ [Text+ working group](https://text-plus.org/en/themen-dokumentation/atf/)

Note:
- idea: share information *about* texts without sharing the texts
- animation shows topic detection on documents for which only a
  document-term matrix is available

--
## n-Grams

> This is a simple sentence.

- 1-grams: This, is, a, simple, sentence
- 2-grams: This is, is a, a simple, simple sentence
- 3-grams: This is a, is a simple, a simple sentence

<br/>

→ prior datasets, for example, [Herdağdelen 2013](https://doi.org/10.1007/s10579-013-9227-2)

</div>

Note:
- consecutive sequence of n tokens
- here: tokens = words, but could also be characters
- 🖱️  prior dataset based on 74,579,965 tweets from 2010 only
  - ... and no longer available

---

# Approach

## Dataset

```json
{
  "created_at": "Mon Mar 23 14:16:46 +0000 2020",
  "id": 1242092932179677187,
  "id_str": "1242092932179677187",
  "text": "Digitale Medien: Wie die Digitalisierung das \"Gedächtnis der Nation\" verändert | @th_held hat uns für die @dpa in L… https://t.co/f3OEiJnxJV",
  "source": "<a href=\"http://itunes.apple.com/us/app/twitter/id409789998?mt=12\" rel=\"nofollow\">Twitter for Mac</a>",
  "truncated": true,
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "in_reply_to_screen_name": null,
  "user": {
	"id": 714103813,
	"id_str": "714103813",
	"name": "Deutsche Nationalbibliothek",
	"screen_name": "DNB_Aktuelles",
	"location": "Leipzig und Frankfurt am Main",
	"url": "http://www.dnb.de",
	"description": "Hier twittert Stephan Jockel, Pressesprecher der Deutschen Nationalbibliothek in Leipzig und Frankfurt am Main. Impressum: http://www.dnb.de/impressum",
	"translator_type": "none",
	"protected": false,
	"verified": false,
	"followers_count": 4513,
	"friends_count": 161,
	"listed_count": 168,
	"favourites_count": 1999,
	"statuses_count": 3201,
	"created_at": "Tue Jul 24 11:01:43 +0000 2012",
	"utc_offset": null,
	"time_zone": null,
	"geo_enabled": true,
	"lang": null,
	"contributors_enabled": false,
	"is_translator": false,
	"profile_background_color": "FFFFFF",
	"profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
	"profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
	"profile_background_tile": false,
	"profile_link_color": "0084B4",
	"profile_sidebar_border_color": "C0DEED",
	"profile_sidebar_fill_color": "DDEEF6",
	"profile_text_color": "333333",
	"profile_use_background_image": true,
	"profile_image_url": "http://pbs.twimg.com/profile_images/969240640700801024/Kb3ck0ch_normal.jpg",
	"profile_image_url_https": "https://pbs.twimg.com/profile_images/969240640700801024/Kb3ck0ch_normal.jpg",
	"profile_banner_url": "https://pbs.twimg.com/profile_banners/714103813/1398238577",
	"default_profile": false,
	"default_profile_image": false,
	"following": null,
	"follow_request_sent": null,
	"notifications": null
  },
  "geo": null,
  "coordinates": null,
  "place": null,
  "contributors": null,
  "is_quote_status": false,
  "extended_tweet": {
	"full_text": "Digitale Medien: Wie die Digitalisierung das \"Gedächtnis der Nation\" verändert | @th_held hat uns für die @dpa in Leipzig besucht und mit unserem Direktor Michael Fernau und mit @kayj_k von den Digitalen Diensten gesprochen. | heise online hat den Beitrag: https://t.co/IOXQc2buE4",
	"display_text_range": [
	  0,
	  280
	],
	"entities": {
	  "hashtags": [],
	  "urls": [
		{
		  "url": "https://t.co/IOXQc2buE4",
		  "expanded_url": "https://www.heise.de/newsticker/meldung/Digitale-Medien-Wie-die-Digitalisierung-das-Gedaechtnis-der-Nation-veraendert-4687855.html",
		  "display_url": "heise.de/newsticker/mel…",
		  "indices": [
			257,
			280
		  ]
		}
	  ],
	  "user_mentions": [
		{
		  "screen_name": "th_held",
		  "name": "Theresa Held",
		  "id": 916642015,
		  "id_str": "916642015",
		  "indices": [
			81,
			89
		  ]
		},
		{
		  "screen_name": "dpa",
		  "name": "dpa",
		  "id": 40227292,
		  "id_str": "40227292",
		  "indices": [
			106,
			110
		  ]
		},
		{
		  "screen_name": "kayj_k",
		  "name": "Kathrin Jockel",
		  "id": 2591854492,
		  "id_str": "2591854492",
		  "indices": [
			178,
			185
		  ]
		}
	  ],
	  "symbols": []
	}
  },
  "quote_count": 0,
  "reply_count": 0,
  "retweet_count": 0,
  "favorite_count": 0,
  "entities": {
	"hashtags": [],
	"urls": [
	  {
		"url": "https://t.co/f3OEiJnxJV",
		"expanded_url": "https://twitter.com/i/web/status/1242092932179677187",
		"display_url": "twitter.com/i/web/status/1…",
		"indices": [
		  117,
		  140
		]
	  }
	],
	"user_mentions": [
	  {
		"screen_name": "th_held",
		"name": "Theresa Held",
		"id": 916642015,
		"id_str": "916642015",
		"indices": [
		  81,
		  89
		]
	  },
	  {
		"screen_name": "dpa",
		"name": "dpa",
		"id": 40227292,
		"id_str": "40227292",
		"indices": [
		  106,
		  110
		]
	  }
	],
	"symbols": []
  },
  "favorited": false,
  "retweeted": false,
  "possibly_sensitive": false,
  "filter_level": "low",
  "lang": "de",
  "timestamp_ms": "1584973006659"
}
```

- collection of the random 1% sample of the Twitter streaming API
  between January 2013 and June 2023 <em class="tonedown">(subset of those used for [TweetsKB](https://data.gesis.org/tweetskb/))</em>
  - 3,700 gzip-compressed [JSONL](https://en.wikipedia.org/wiki/JSON_streaming#JSONL) files
  - approx. 8.4 Terabyte
  - 14,160,083,429 valid tweets
- [some details on the collection process](https://slides.igada.de/2023-07-07-fontane_archiv/)

Note:
- so processing this data is not trivial
- for instance, we had to filter ... 🖱️

## Filtering Tweets

| count              | type                      |
| -------------:     | -----------               |
| 14,160,083,429     | valid                     |
| -9,260,435,685     | non-English               |
| -1,182,713,840     | deleted                   |
| -1,606,843,572     | retweets                  |
| -2,718,439         | duplicates                |
| **=2,107,371,893** | **unique English tweets** |

Notes:
- **invalid**: e.g., incomplete due to disconnect, empty tweets (no text)
- **English**:
- **deleted**: deletion information contained in our dataset
- **retweets**: avoid distortions/spam (but could be interesting as an extension)
- **duplicates**: due to redundant collection

## Processing

- removed URLs and @mentions
- tokenized with twokenize <em class="tonedown">([Ritter et al. 2011](https://aclanthology.org/D11-1141/))</em>
- optional: normalization to lower case
- extracted 1-, 2-, 3-grams

```text
what’s
don’t
&
06/12/2023
€3.50
#OMG
@user
30°C
;D
>.<
123
45.67
1,000
🔥
```

> what’s don’t & 06/12/2023 €3.50 #OMG @user 30°C ;D >.< 123 45.67 1,000 🔥

Note:
- URLs/@mentions: privacy/personal rights; URLs: not our focus
- tokenization: separating individual tokens
  - typically words, numbers etc. but tweets very rich:
	- 🖱️  idiosyncrasies typical of social media, such as creative
	  orthography, meta-information (e.g., hashtags, mentions), and
	  emoticons
  - simple / standard tokenizer not sufficient
	- e.g., separates emoticons or currency symbols (or units like °C)
	- thus: **twokenize** 🖱️  *(works like a charm)*
- as can be seen: **we keep punctuation**

--
## Dataset

- two parts: *original case* and *lower case*
  - each part: 11 TAR files, one for each year (2013-2023)
	- each TAR: up to 12 gzip-compressed TSV files, one for each month
- → overall: 125 TSV files
  - each line represents an n-gram with the following columns: 
	- `ngram_type`: 1, 2, or 3
	- `count`: frequency <em class="tonedown">(i.e., number of appearances per month)</em>
	- `ngram`: the n-gram itself

## Examples

<div class="container">
<div class="col fragment">
<small>
first twenty rows

| ngram_type |  count | ngram |
| ---------: | -----: | :---- |
|          1 | 195891 | .     |
|          1 | 145089 | the   |
|          1 | 142295 | ,     |
|          1 | 134689 | to    |
|          1 | 111336 | a     |
|          1 | 106765 | I     |
|          1 |  98259 | …     |
|          1 |  85359 | and   |
|          1 |  77553 | you   |
|          1 |  73279 | of    |
|          1 |  69851 | !     |
|          1 |  68355 | is    |
|          1 |  68253 | in    |
|          1 |  67921 | for   |
|          1 |  58991 | :     |
|          1 |  47744 | on    |
|          1 |  44847 | ?     |
|          1 |  40520 | my    |
|          1 |  39672 | it    |
|          1 |  37573 | that  |

</small>
</div>
<div class="col fragment">
<small>
rows around 'library'

| ngram_type | count | ngram       |
| ---------: | ----: | :---------  |
|          1 |   150 | POTUS       |
|          1 |   150 | Seeing      |
|          1 |   150 | TX          |
|          1 |   150 | Unknown     |
|          1 |   150 | apartment   |
|          1 |   150 | audience    |
|          1 |   150 | concern     |
|          1 |   150 | friendship  |
|          1 |   150 | hehe        |
|          1 |   150 | hottest     |
|          1 |   150 | **library** |
|          1 |   150 | limit       |
|          1 |   150 | messages    |
|          1 |   150 | pleased     |
|          1 |   150 | print       |
|          1 |   150 | properly    |
|          1 |   150 | racism      |
|          1 |   150 | 💞          |
|          1 |   149 | #free       |
|          1 |   149 | Casino      |

</small>
</div>
<div class="col fragment">
<small>
3-grams with 'library'

| ngram_type | count | ngram                           |
| ---------: | ----: | :---------                      |
|          3 |    21 | library ! #tvtime               |
|          3 |    21 | my library !                    |
|          3 |    21 | to my library                   |
|          3 |    19 | in the library                  |
|          3 |    10 | at the library                  |
|          3 |     7 | to the library                  |
|          3 |     5 | the library .                   |
|          3 |     5 | the library and                 |
|          3 |     4 | library . I                     |
|          3 |     3 | from the library                |
|          3 |     3 | the library but                 |
|          3 |     3 | visit the library               |
|          3 |     2 | a library book                  |
|          3 |     2 | library is closed               |
|          3 |     2 | the library ,                   |
|          3 |     2 | the library i                   |
|          3 |     2 | the library in                  |
|          3 |     2 | the library on                  |
|          3 |     2 | the library this                |
|          3 |     2 | the library tomorrow            |

</small>
</div>
</div>

Note:
- frequent English words
- punctuation
- mixed (original) case
- hashtags (#free) and emojis (💞)

## Statistics

![Monthly numbers of 1-, 2-, and 3-grams](images/stats.svg)

Note:
- avg. 600k distinct 1-grams per month
- avg. 3.3M distinct 2-grams per month
- avg. 6M distinct 3-grams per month
- gaps: server fault / API v1 shutdown in 2022-12
- trends: decline until 2020 → resurgence during pandemic

## Published Dataset

![Dataset on Zenodo](images/zenodo_dataset.png)

doi:[10.5281/zenodo.15736201](https://doi.org/10.5281/zenodo.15736201)

## Limitations

- gaps & errors
- timeframe
- post-collection deletions
- sampling <em class="tonedown">([Morstatter et al. 2013](https://doi.org/10.1609/icwsm.v7i1.14401))

Note:
- gaps: outages (crawler, Twitter API), API v1 shutdown (→ 2022-12 missing)
- timeframe: 01/2013 and 06/2023 incomplete, 07/2023++ missing → years incomplete
- sampling:
  - started from 1%, then filtered → not 1% of English tweets

---

# Discussion

## Summary

- monthly 1-, 2-, 3-grams based on more than 2 billion English tweets
  - 01/2013 – 06/2023
  - including frequency information
- publicly available at doi:[10.5281/zenodo.15736201](https://doi.org/10.5281/zenodo.15736201)

![Timeline of the hashtag '#covid19' and the terms 'trump' and 'biden'](images/example_timeline.png)

Note:
- effortless re-use
  - no need for complex filtering and pre-processing
  - handling raw data is cumbersome!
  - it's quite simple to produce analyses like this plot 🖱️

## Outlook

- online n-gram viewer: https://data.gesis.org/twitterngram (test phase)
- sharing annotations of copyrighted texts
  - masking and alignment <em class="tonedown">([Rehm et al. 2007](https://nbn-resolving.org/urn:nbn:de:bsz:mh39-45145), [Arnold and Jäschke 2025](https://doi.org/10.1007/978-3-032-06136-2_13))</em>
  - [Sharing Is Caring: Wie das Teilen von Annotationen urheberrechtlich geschützter Texte gelingen kann](https://bibliocon2026.abstractserver.com/program/#/details/presentations/248) @ [BiblioCon 2026](https://2026.bibliocon.de/)

Note:
- limitations also in other areas of research, e.g., sharing of
  annotated copyrighted texts

<!--

qrencode -s15 -o images/qrcode.png "https://slides.igada.de/2026-03-18-smad/"

-->

[![Link zu den Folien](images/qrcode.png)](https://slides.igada.de/2024-09-13-hu/)

[https://slides.igada.de/2026-03-18-smad/](https://slides.igada.de/2026-03-18-smad/)

doi:[10.5281/zenodo.15736201](https://doi.org/10.5281/zenodo.15736201)

[![www.bibsonomy.org](images/postit.png)](https://www.bibsonomy.org/)

[![weltliteratur.net](images/logo_weltliteratur_953x265.png)](http://weltliteratur.net/)

</div>

![XKCD: Questions?](images/xkcd_1256.png)

<small>© [XKCD, Randall Munroe](https://xkcd.com/1256) / [CC BY-NC 2.5](https://creativecommons.org/licenses/by-nc/2.5/)</small>

-  <small>thanks to Asmelash Teka Hadgu, Ernesto Diaz-Aviles, Erdal Baran, Stefan Dietze, Dimitar Dimitrov, Pavlos Fafalios, Vasileios Iosifidis, Ran Yu, Xiaofei Zhu, Matthäus Zloch, Sebastian Schellhammer</small>
- 
  [robert.jaeschke@hu-berlin.de](mailto:robert.jaeschke@hu-berlin.de)
- 
  [https://hu.berlin/RJ](https://hu.berlin/RJ)

## References

- Frederik Arnold and Robert Jäschke. “**Sharing is Caring: A Text
  Alignment Approach for Sharing Annotations of Copyrighted Texts**.”
  In: *New Trends in Theory and Practice of Digital Libraries: TPDL
  2025 Short Papers and Workshops*, pp. 135–145, 2025.
  doi:[10.1007/978-3-032-06136-2_13](https://doi.org/10.1007/978-3-032-06136-2_13)
- Dennis Assenmacher et al. “**The end of the rehydration era – the
  problem of sharing harmful Twitter research data**.” In: *2nd
  Workshop on Novel Evaluation Approaches for Text Classification
  Systems (NEATCLasS)*, 2023.
  doi:[10.36190/2023.56](https://doi.org/10.36190/2023.56).
- Pavlos Fafalios et al. “**TweetsKB: A Public and Large-Scale RDF
  Corpus of Annotated Tweets**.” In: *The Semantic Web*,
  pp. 177–190, 2018.
  doi:[10.1007/978-3-319-93417-4_12](https://doi.org/10.1007/978-3-319-93417-4_12).
- Amaç Herdağdelen. “**Twitter n-gram corpus with demographic
  metadata**.” In: *Language resources and evaluation*, 47.4,
  pp. 1127-1147, 2013.
  doi:[10.1007/s10579-013-9227-2](https://doi.org/10.1007/s10579-013-9227-2)
- Georg Rehm et al. “**Corpus masking: Legally bypassing licensing
  restrictions for the free distribution of text collections**.” In:
  *Digital Humanities 2007. Conference Abstracts*, pp. 166-170, 2007.
  urn:[nbn:de:bsz:mh39-45145](https://nbn-resolving.org/urn:nbn:de:bsz:mh39-45145)
- Christof Schöch et al. “**Abgeleitete Textformate: Text und Data
  Mining mit urheberrechtlich geschützten Textbeständen**.” In:
  *Zeitschrift für digitale Geisteswissenschaften* 5, 2020.
  doi:[10.17175/2020_006](https://doi.org/10.17175/2020_006).
- Fred Morstatter et al. “**Is the Sample Good Enough? Comparing Data
  from Twitter's Streaming API with Twitter's Firehose.**.” In:
  *Proceedings of the Seventh International AAAI Conference on Weblogs
  and Social Media*, pp. 400-408, 2013.
  doi:[10.1609/icwsm.v7i1.14401](https://doi.org/10.1609/icwsm.v7i1.14401)
- Alan Ritter et al. “**Named Entity Recognition in Tweets: An
  Experimental Study**.” In: *Proceedings of the 2011 Conference on
  Empirical Methods in Natural Language Processing*,
  pp. 1524–1534, 2011.
  url:[https://aclanthology.org/D11-1141/](https://aclanthology.org/D11-1141/)