Back to Blog

Building PharmaFinder Scraper: Crawling Official Greek Syndicates with Self-Healing Registries & Zstandard CDN

•Emmanouil Athanasopoulos•3 min read•654 words
PythonReact NativeScrapingGeospatialZstandardGitHub ActionsData Engineering

The Motivation Behind the Project

Locating open emergency pharmacies in Greece during nights, weekends, and holidays has historically suffered from fragmented municipal syndicates, proprietary mobile APIs, and brittle reverse-engineered proxies. To power the PharmaFinder Greece React Native mobile application, I designed an autonomous multi-source data pipeline that eliminates third-party dependencies by crawling official Greek pharmaceutical syndicate portals (ΦΣΑ Attica and the 30+ regional prefectures of the ITeQ network). The pipeline matches active duty shifts against a 9,103-store national master registry using phone normalization and Haversine proximity heuristics, self-heals as new stores emerge, and distributes ultra-compact Zstandard (.zst) releases 4 times daily via GitHub Actions.

Core Features and Design Goals

  • Direct Primary Syndicate Ingestion: Ingests live rota data directly from official statutory pharmaceutical syndicate servers—ΦΣΑ (Athens/Attica basin) and 30+ regional prefectures across Greece (Crete, Thessaloniki, Patras, Larissa, Aegean & Ionian islands)—with zero third-party proxy dependencies.
  • Self-Healing Master Catalog: Maintains a permanent 9,103-store national baseline in data/pharmacies_master.json. When syndicates list previously uncataloged or relocated stores, the pipeline dynamically extracts their phone numbers, computes high-precision GPS coordinates, and automatically upserts them into the catalog.
  • Multi-Strategy Entity Matching: Matches active syndicate rota cards to master catalog pharmacies using normalized 10-digit phone lookup, fallback Haversine distance (≤75m) proximity testing, and Greek token overlap heuristics.
  • Clean State Isolation: Resets all stores to inactive status at the start of each execution before applying fresh scrape results, ensuring shift rotas never bleed or accumulate across calendar days.
  • 88% Zstandard (.zst) Compression: Compresses the uncompressed 3.7MB nationwide JSON payload down to ~449KB using dictionary-less level-19 Zstandard compression for instantaneous, low-bandwidth mobile hydration.
  • 4x Daily Statutory Automation: Synchronized with Greek pharmacy statutory shift rollover windows (07:17, 13:47, 19:23, 00:17 Athens time) via GitHub Actions cron, publishing immutable releases with GitHub CLI.

Deep Dive: How It Works Under the Hood

Initially, the pipeline relied on scraping a proprietary web app's Next.js BFF proxy using AES-CBC token decryption. However, this introduced a single point of failure and missed rural regions. Investigation into the Greek pharmaceutical IT infrastructure revealed two primary sources: ΦΣΑ Attica (fsa-efimeries.gr), which serves all on-duty pharmacies for the capital in a single structured HTMX-compatible endpoint with exact map coordinates, and the ITeQ regional network (*.efhmeries.gr), powering 30+ prefectural syndicates. By querying these syndicates in parallel with polite backoff delays and matching entries against our nationwide coordinate database, we achieved complete nationwide coverage directly from the legal authorities.

Technical Implementation

  • Implemented a parallel crawler querying 30 regional ITeQ subdomains with exponential backoff and timeout safeguards to respect syndicate web servers.
  • Built robust Greek text normalization and 10-digit phone sanitization (handling 2xxx landline and 69xxx mobile prefixes) for accurate cross-source entity deduplication.
  • Integrated Haversine spherical distance calculation (threshold ≤75m) for geocoded stores without matching phone records.
  • Configured automated GitHub Actions workflows with pip dependency caching and GitHub Release CDN deployment.

Architecture Overview

The pipeline runs on GitHub Actions runners 4 times daily. multi_source_scraper.py queries ΦΣΑ Attica and 30 regional syndicate endpoints, normalizes duty shifts, self-heals data/pharmacies_master.json, and writes data/duties_raw.json. Next, package_dataset.py performs clean state reset, merges active shifts into a 1-to-1 schema, produces minified pharmacies_latest.min.json, compresses it to .zst, and tags an official GitHub Release via gh release create.

Challenges I Faced Along the Way

The main engineering challenge was heterogeneous formatting across 30+ regional syndicates. Some syndicate sites format hours as full ranges (08:00 – 14:00 & 17:30 – 21:00), while others display overnight flags or split shift markers. Building resilient regex parsers and Greek diacritic-insensitive entity resolution ensured 100% extraction accuracy without dropping stores.

The Technology Stack

Python 3.12 with urllib and zstandard, GitHub Actions for automated cron orchestration, and GitHub Releases as a zero-cost serverless CDN.

Final Reflections

By replacing fragile reverse-engineered proxies with direct queries to primary syndicate sources and coupling them with a self-healing master catalog and Zstandard compression, you can deliver high-availability, zero-maintenance datasets directly to mobile apps without recurring backend infrastructure costs.