LinkedIn Job Scraping & Data Export Pipeline

Situation

I built this during my M.Sc. while job hunting and trying to decide which skills to prioritize before graduating. I kept checking LinkedIn manually, but that doesn't scale across cities, roles, and time. There's no public jobs API, so the choices were clear: keep copy-pasting, or build a scraper that runs daily, keeps a queryable history, and actually answers the question.

Task

The goal was practical: a daily data feed I could query to track which skills were trending. That meant unattended runs, graceful handling when LinkedIn's markup changed instead of crashes, a queryable run history, and automatic alerts when a run came back empty, all without me having to check.

Action

• Designed the core scraping and persistence logic to be callable two ways: an Argparse CLI for one-off runs and an importable library API for scripted use, both driving the same underlying code.
• Scraped LinkedIn's guest surface via paginated card search and per-job detail calls, with rotating User-Agents, exponential backoff, and per-field null-checks so markup shifts degrade to 'N/A' instead of crashing. Optional OpenAI enrichment adds structured tags (summary, tech stack, seniority) before persisting to SQLite across four tables.
• Automated delivery on GitHub Actions: a daily cron commits scrape output and dated CSV exports to an orphan data branch, auto-files an issue on any empty run, and auto-closes it on recovery.

How it works

Click to zoom

Three entry points (CLI, library call, scheduled cron) all route into the same pipeline core. Inside, each job passes through scraping, field cleaning, and title classification before landing in SQLite. The dashed OpenAI branch is opt-in: if an API key is set, each record gets a summary, a detected tech stack, and a seniority level added before persistence.

Result

Shipped to PyPI as LinkedInWebScraper v1.1.1. The CI schedule has been running every day since March 22, 2026, each committing output and dated CSV exports to the data branch. A multi-city analysis across Mexico City, Monterrey, and Guadalajara covered 285 deduplicated postings: Python dominated skill requirements at roughly 80% of listings, mid-senior roles made up half the market with entry level at 38%, and the work-scheme split was nearly even: on-site 41%, hybrid 30%, remote 29%.

Learning

Unofficial scrapers have no deprecation notices, so resilience has to be the starting assumption rather than a patch added after the first breakage. The retry policy was the clearest example: a typed, frozen dataclass that models retryable vs. permanent failures is not just safer than ad-hoc retries, it is a contract you can test in isolation. That same thinking scales: null-check every field at parse time, and make the system alert you when it fails rather than silently returning empty.