Back to Blog

Engineering Greece Transit Scraper: Automated Multi-Modal ETL Pipeline & Vector Datasets

•Emmanouil Athanasopoulos•2 min read•345 words
PythonETLGISGitHub ActionsGeoJSONPublic TransitAutomation

The Motivation Behind the Project

Public transit telematics across Greece have historically remained locked in separate, legacy municipal systems: Athens operates under the OASA Telematics portal, while Thessaloniki relies on OASTH endpoints. Both feature non-standard JSON payloads, unpredictable rate-limit throttling, and zero geographical vector geometry for rail and metro systems. The Greece Transit Scraper was engineered to bridge this gap: an autonomous Python and GitHub Actions ETL pipeline that extracts, enriches, validates, compresses, and publishes daily vector transit datasets and departure schedules directly to GitHub Releases.

Core Features and Design Goals

  • Multi-City Unified Extraction: Scrapes raw line, route, and stop definitions from both Athens (OASA) and Thessaloniki (OASTH) telematics gateways into clean, normalized JSON schemas.
  • Rail & Tram Vector Geometries: Integrates precise geographic GeoJSON LineStrings and point coordinates for Athens Metro (Lines 1, 2, 3), Tram lines, Suburban Rail, and the newly launched Thessaloniki Metro.
  • Wikimedia Station Photography Ingestion: Enriches 58 rail and metro stations with high-resolution Wikimedia Commons station photography and responsive thumbnail URLs.
  • Automated Compression & Bundling: Serializes transit networks into minified JSON and zstandard/gzip compressed archives, reducing multi-megabyte datasets to sub-50KB payloads for low-bandwidth mobile consumption.
  • Autonomous CI/CD Releases: Scheduled GitHub Actions workflow runs daily to verify API integrity, package datasets, and publish version-tagged GitHub Releases with zero manual intervention.

Technical Implementation

  • Engineered scraper.py with exponential backoff retries and connection pooling to reliably query legacy government telematics endpoints without triggering rate-limit bans.
  • Built seed_master_data.py to compile master station coordinates, interchange mappings, and Wikipedia image assets into validated transit graph structures.
  • Authored package_dataset.py to generate both minified JSON and compressed artifacts, writing semantic release notes with item counts and data hashes.
  • Configured a GitHub Actions workflow (.github/workflows/update-transit.yml) using gh release create to automate release tagging and asset uploading.

The Technology Stack

Built with Python 3.11, Requests, and Zstandard. The automation pipeline runs on Ubuntu runners via GitHub Actions, publishing tagged release assets for mobile clients to consume with atomic cache invalidation.

Final Reflections

Automating the dataset pipeline turns fragile real-time mobile apps into resilient, offline-first applications with guaranteed vector map fidelity.