Exploring historical trends in the community archive
Originally published in the CA github wiki in Nov 2024.
One of the first apps we built on top of the Twitter Community Archive is a "Google trends" like app. Try it here. Source code.
Below is a few interesting use cases & insights people have found so far using this tool. (If you find anything novel/interesting please let us know!)
Visualizing how ideas spread
Most of the people who have contributed their twitter data so far belong to a community called "tpot" (this part of twitter). It didn't always have this name, some people called it "postrat" or "ingroup" before it got its current name. We can see this happening in the graph below:
Sept 2021, the word "tpot" first appeared
By March 2022, its usage grew to match the other competing names
Right after that inflection point, "postrat" & "ingroup" usage died, the community had accepted the new name
We can select a region of this graph to filter tweets by that time range.
The individual tweets you find here all link back to twitter, so you can read the context of the discussion even though we only have a very small subset of twitter data. This is the real power of integration over data silos: we don't need all the data to find useful insights. Individual communities can create these open source "views" into their own data & use it side by side with the global search tools.
The really exciting thing about this being open source / open data is that we can recreate this search across communities. In the future we could plug in Mastodon/Blue sky data, or even blog posts on LessWrong / Substack. You could track how ideas spread across these channels (take the most popular blog posts, how many of them originate as discussions on twitter. Or vice versa: which blogs trigger the most discussion & generate new vocabulary for the community?)
Comparing a community's trends with mainstream Google trends
We can see ways in which specific communities are "ahead of the mainstream discourse" by comparing these two views. I plugged in "ChatGPT" and "Claude" into the twitter trends app (left), and into Google trends (right).
This becomes extremely valuable when you filter it by specific subsets of people. A lot of trends are too small to show up in the global Google trends, but make huge waves in specific communities (like "science twitter", "software devs twitter", or just "the list of people of I trust on topic X").