Web Scraping & Data Mining Developer

We are seeking a contract web scraping and data mining developer to reliably collect and structure data from a wide range of public online sources. The ideal candidate delivers work quickly, prioritizes accuracy, and can independently take a source from initial assessment to clean, usable data.

Responsibilities:

• Build and maintain scrapers and crawlers to extract structured data from websites, APIs, and public records, including sites with pagination, dynamic JavaScript content, and anti-bot measures.
• Work with a variety of source types and formats (HTML, JSON, PDFs, scanned
• documents, non-English pages) and deliver clean, well-structured output (CSV/JSON, or written directly into our pipelines and databases).
• Develop reusable scripts and utilities to automate recurring collection and monitoring tasks.
• Diagnose and repair scrapers when source websites change, and adapt to new sources as needed.
• Document all work clearly so that other members of the team can run and maintain it.

Required skills and experience

• Strong proficiency in Python, with hands-on experience using Scrapy, Selenium (or Playwright), BeautifulSoup, and requests.
• Solid experience working with APIs (REST, authentication, rate limits) and SQL.
• Demonstrated ability to address common anti-scraping challenges, including rotating proxies, headers and session management, CAPTCHAs, and rate limiting.
• Experience handling large volumes of unstructured or inconsistent data, with a consistent record of accurate, verified results.
• Ability to work independently through the full development cycle: scoping a source, building the solution, and delivering usable data.
• Clear communication and dependable turnaround.

Preferred qualifications

• Additional programming languages (Go, Perl, Java, JavaScript) and reverse-engineering experience for complex sources.
• Experience with non-English-language sources and OCR.
• Familiarity with cloud- and queue-based scraping architectures (e.g., AWS, containerized workers, commercial proxy services).
• Experience with public records, corporate registries, court filings, or similar structured and semi-structured sources.
• Interest in supporting investigative, transparency, or public-interest work.

Attributes of a strong candidate

• A commitment to accuracy and quality.
• Strong problem-solving skills and the ability to learn new technologies quickly.
• Proactive communication and reliable adherence to deadlines.
• Discretion and reliability when handling sensitive projects.