Back to Blog

Building Movie & TV Offline Database: Unifying 1.29M+ Titles Across 6 Media Platforms

•Emmanouil Athanasopoulos•2 min read•325 words
Node.jsData EngineeringMedia APIsBig DataGitHub Actions

The Motivation Behind the Project

Media server hobbyists, mobile tracker developers, and data scientists constantly struggle with fragmented media metadata: IMDb has comprehensive vote tallies but no open cross-platform links, TMDB has great posters and season breakdowns but different IDs, Letterboxd has film community culture, and TVDB excels at episodic broadcasts. Movie & TV Offline Database was created to solve this fragmentation by providing an open-source, daily-updated offline dataset unifying over 1.29 million movies and series across 6 major platforms with no vote thresholds.

Core Features and Design Goals

  • 1.29M+ Movies & TV Catalog: Comprehensive coverage of the entire cinematic and television catalog with zero vote count minimums.
  • Universal 6-Platform ID Map: Direct 1:1 bidirectional mapping across IMDb (tt*), TMDB, TheTVDB, Letterboxd, Trakt, and Wikidata.
  • 1.7M+ Ratings & Votes Aggregator: Full IMDb ratings matrix integrated directly into each title record.
  • Multi-Tier Asset Distribution: Daily distribution of minified production JSON, human-readable indented JSON, key-value ratings, and index cross-reference maps.
  • Constant-Memory Streaming Pipeline: Node.js streaming transform architecture that parses multi-gigabyte TSV files in under 400MB of RAM.

Deep Dive: How It Works Under the Hood

Processing over 1.29 million entries with complex cross-platform links in a memory-constrained CI environment (GitHub Actions runners with 7GB RAM) required abandoning in-memory array manipulation in favor of Node.js streaming pipelines. Multi-gigabyte IMDb dump archives are decompressed and piped line-by-line through custom streaming parsers that filter, normalize, and match entries against external ID databases in a single deterministic pass.

Technical Implementation

  • Built streaming TSV parsers handling millions of rows with minimal GC pressure.
  • Implemented bidirectional index builders for O(1) lookups by IMDb ID or TMDB ID.
  • Integrated automated nightly GitHub Actions workflows publishing four distinct dataset formats to GitHub Releases.

The Technology Stack

Node.js streaming architecture, TypeScript, zlib, and GitHub Actions CI/CD.

Final Reflections

High-volume data pipelines don't require heavy database infrastructure. Streaming transforms and flat-file JSON releases can power thousands of client applications and self-hosted servers with rock-solid reliability.