Back to Blog

Building FuelGR Pipeline: Scraping 4,700+ Gas Stations with Geospatial Grids & Zstandard Compression

•Emmanouil Athanasopoulos•3 min read•585 words
Node.jsPythonGeospatialZstandardGitHub ActionsData Engineering

The Motivation Behind the Project

The official Greek fuel observatory provides real-time pricing for thousands of service stations across the country, but its public web interfaces are guarded by Google reCAPTCHA v2 and restrictive 30-result geospatial API caps. Single-city query approaches consistently missed over 95% of active stations in the country. To power the Fuel Greece open-source mobile application, I designed an automated, high-throughput daily extraction and distribution pipeline that discovers all 4,700+ operating stations nationwide in under 45 seconds, packages split-file historical archives, and distributes compressed payloads through GitHub Releases CDN without recurring database hosting expenses.

Core Features and Design Goals

  • 1,107-Point Geospatial Scan Grid: A mathematically tailored coordinate lattice encompassing all 54 Greek prefectural units, national highway corridors, Aegean and Ionian islands, and dense metropolitan clusters.
  • 25-Worker HTTP Keep-Alive Pool: High-concurrency Node.js fetch architecture utilizing persistent TCP sockets to query 1,107 coordinates in just 44 seconds without browser automation overhead.
  • Ministry PDF Daily Bulletin Parser: Python-based PDF layout extractor that parses daily Ministry of Development price decrees using font metadata and regex heuristics to compute regional averages for Unleaded 95, Unleaded 100, Diesel, and LPG.
  • Split-File Mobile Architecture: Redesigned output into an ultra-compact daily master (stations_latest.min.json) with embedded 7-day deltas and 14-day sparklines, coupled with on-demand station history files (history/{id}.json).
  • 90% Zstandard (.zst) Compression: Shrinks the 1.84MB station dataset down to 179KB using dictionary-less modern Zstandard compression, dramatically reducing mobile data usage.
  • Zero-Cost GitHub Releases CDN: Fully serverless release lifecycle triggered daily at 14:00 EEST via GitHub Actions, publishing immutable release tags and clobbering latest assets.

Deep Dive: How It Works Under the Hood

The breakthrough in discovering all 4,700+ gas stations lay in bypassing the government web portal's CAPTCHA by directly querying the official mobile backend feed with a carefully constructed geospatial grid. Because the backend limits queries to a 30-station radius, querying only major city centers yields under 200 stations. By generating a 1,107-coordinate bounding lattice mapped along the real road network and island perimeters with overlapping 25km radii, every operating station in Greece is intersected at least once. Concurrent HTTP keep-alive connections ensure the entire country is queried in parallel without triggering rate limits.

Technical Implementation

  • Engineered a dual-fuel scanning algorithm (Unleaded 95 & Autogas/LPG) ensuring dual-license stations are thoroughly captured.
  • Designed a Python dataset packaging pipeline with zstandard compression reducing JSON transport size by ~90%.
  • Structured per-station on-demand history archives containing chronological price records and all-time low/high statistics.
  • Automated daily GitHub Actions execution with automated release tagging, changelog generation, and GitHub CLI distribution.

Architecture Overview

The pipeline runs entirely within GitHub Actions runners. First, scraper.py extracts regional averages from government PDF bulletins. Next, scraper.js executes the 1,107-point coordinate scan pool. Finally, package_dataset.py computes 7-day deltas, generates 14-day sparklines, minifies JSON, compresses .zst binaries, and publishes them to GitHub Releases via gh release create.

Challenges I Faced Along the Way

Overcoming the CAPTCHA barrier without paid solver APIs or heavy headless Chromium browsers was the chief engineering constraint. Discovering the direct mobile API endpoints and modeling the geospatial lattice to cover remote mountain roads and island ports was essential to reaching 100% station discovery.

The Technology Stack

Node.js with native fetch and HTTP Keep-Alive agents, Python 3.12 with pypdf and zstandard, and GitHub Actions for automated orchestration.

Final Reflections

By combining smart geospatial grid sampling with modern compression algorithms like Zstandard and leveraging GitHub Releases as a globally distributed CDN, you can build production-grade data pipelines that serve mobile apps with zero operational infrastructure costs.