Learn how to build a panoramic observatory app for monitoring and collecting datasets from major social-media platforms and online communities—no prior programming experience required!
Whether you’re a security researcher, a journalist, an activist, or a scholar, this book is an essential blueprint for building your window into online worlds.Analyzing the billions of social media posts published daily online provides essential data for modern researchers—and this book gives you the tools to build a panoramic view of the digital-community discourse out there waiting to be discovered. Social Media Observatory is an in-depth beginner’s guide to coding web applications that monitor commentary on the major platforms in real time, giving you unparalleled insights into the cultural and political conversations of our time, then empowering you to reach your own conclusions.
You’ll first set up a working observatory for public channels on the instant messaging app Telegram, then incorporate Instagram and Twitter with help from the Wayback Machine. You’ll detect Twitter bots via account birth charts, extract text-embedded URLs for analysis, plot interactive Telegram networks to enumerate media bubbles, and map the structure of a social movement on Instagram. Finally, you’ll deploy your observatory to a remote server to collaborate with other investigators. By the end of the book, you’ll have a working website that allows team members to log in, request data collection, and perform analysis of collected data, via an intuitive and user-friendly interface.
Along the way, you’ll learn to:
- Use Python libraries to retrieve Telegram data from a remote API
- Analyze the data you’ve retrieved with PostgreSQL, RabbitMQ, and other tools
- Store data in a relational database
- Create and securely deploy an interactive dashboard for data collection and analysis in the cloud
- Modify your observatory’s functionality to suit your needs
Whether you’re a security researcher, a journalist, an activist, or a scholar, Social Media Observatory is an essential blueprint for building your window into online worlds.
Foreword
Acknowledgments
Introduction
PART I: Foundations
Chapter 1: Preparing Your Development Workstation
Chapter 2: Coding a Skeletal Observatory
Chapter 3: Adding a Web Interface
Chapter 4: Retrieving and Analyzing Content
Chapter 5: Mapping Networks
PART II: Adversarial Archiving
Chapter 6: Scraping Twitter via the Wayback Machine
Chapter 7: Scraping Instagram with Playwright
Chapter 8: Scraping TikTok on Android
PART III: Going Public
Chapter 9: Writing an API
Chapter 10: Deploying to the Cloud
Alexei Sisulu Abrahams leads the digital trace team at the Canadian Media Ecosystem Observatory, where he and his colleagues monitor political discourse on social media. He holds a doctorate in economics from Brown University, and has over a decade of experience leading research investigations at Harvard, Princeton, Citizen Lab, and the World Bank. A social scientist by training, he learned to code by taking classes, online courses, and reading No Starch Press books.
