Bypassing the Scrape: How Public ATS Feeds Are Transforming Tech Hiring Data and Job Market Analysis

By Tech & Data Insights Reporter

Main Facts: The End of Traditional Web Scraping for Job Openings

For anyone building a modern job board, running high-volume recruitment outreach campaigns, or tracking hiring trends as a macroeconomic market signal, the core bottleneck has always been the same: acquiring a clean, structured, and up-to-date list of open job roles.

For years, the default path involved heavy web scraping architectures targeting mainstream platforms like LinkedIn, Indeed, or Naukri. Developers and data scientists became all too familiar with the friction-filled ecosystem of rotating proxies, aggressive CAPTCHAs, dynamic DOM mutations, duplicate listings, and the constant threat of IP bans.

However, a simpler, more robust data source has been hiding in plain sight. Most modern product-led companies—from nimble startups to multinational tech giants—do not host their own bespoke applicant-tracking infrastructure. Instead, they rely on standardized Applicant Tracking Systems (ATS) such as Greenhouse, Lever, Ashby, and SmartRecruiters. Crucially, these vendors publish public JSON job-board feeds designed precisely to allow companies to effortlessly embed their live job openings onto corporate careers pages or third-party web properties. No authentication, no logins, and no session cookies are required to access them.

Capitalizing on this transparent architecture, developer Nidhi (@RealNidhibhai on X, GitHub: Nidhi-bhai) has engineered two specialized Apify actors—the India Tech Jobs Scraper and the AI & Startup Jobs Scraper—designed to ingest these direct ATS endpoints at scale. Rather than battling anti-scraping defenses on public job aggregator sites, these tools query the primary source of truth, returning structured, normalized, and real-time hiring data in seconds.

Chronology: The Evolution of ATS Data Extraction and the Birth of the Scrapers

The methodology of pulling data directly from ATS endpoints represents a philosophical shift in how recruitment data is gathered. Understanding its trajectory highlights why this approach is gaining rapid traction among developers and data analysts.

Phase 1: The Era of Aggregator Scraping (2010–2020)

During this decade, job market analytics relied heavily on scraping job aggregators. Because platforms like LinkedIn consolidated millions of listings, companies attempted to harvest data at scale. This sparked an ongoing technological arms race between scrapers (using headless browsers like Puppeteer or Selenium) and anti-bot systems (utilizing advanced browser fingerprinting and behavioral analysis). The result was fragile codebases that broke frequently whenever an aggregator updated its frontend user interface.

Phase 2: Discovery of Native JSON Endpoints (2021–2024)

As enterprise companies standardized around SaaS ATS providers (Greenhouse, Lever, Ashby), developers realized that these platforms exposed lightweight, public-facing API endpoints. These JSON feeds were intentionally left accessible to ensure fast page loads for candidate-facing careers portals. Engineers began writing bespoke scripts to query individual company slugs (e.g., api.greenhouse.io/v1/boards/company/jobs). While effective for single companies, scaling this approach across hundreds of disparate organizations required complex, custom-built orchestration layers.

Phase 3: Centralized Automation via Apify (2025–Present)

The release of generalized and sector-specific Apify actors systematized this process. By packaging the discovery logic, directory curation, and data normalization into ready-to-run cloud components, developers can now fetch tens of thousands of live tech jobs globally with a single function call. In benchmark tests conducted in late October 2026, these actors successfully harvested over 23,000 active listings across dozens of platforms in under a minute, signaling a mature shift toward API-driven recruitment intelligence.

Supporting Data: Inside the Actors and Their Capabilities

To understand the scale and efficiency of ATS-feed scraping, it helps to examine the two primary tools built by Nidhi and evaluate how they parse, clean, and deliver recruitment data.

1. The India Tech Jobs Scraper

• Built-in Directory: Features a curated list of 57 prominent employers actively hiring engineering and product talent in India. Examples include global and regional heavyweights like Stripe, Databricks, Okta, MongoDB, Zscaler, Swiggy, CRED, Meesho, Freshworks, Groww, Sarvam AI, and InMobi.
• Normalization Engine: Automatically tags every record with boolean flags like isIndia and standardized string fields for indiaCity. It resolves common formatting inconsistencies, seamlessly unifying variations such as "Bangalore" and "Bengaluru," or "Gurgaon" and "Gurugram."
• Performance Benchmark: In a test execution on October 27, 2026, the actor successfully extracted 1,431 live India-based job openings drawn from 61 company boards in a mere 51 seconds.

2. The AI & Startup Jobs Scraper

• Directory Scope: Encompasses 144 high-growth companies categorized by distinct industry sectors: ai, fintech, devtools, security, crypto, health, autonomy_space_defense, and consumer_gaming. The directory features bleeding-edge AI labs and tech leaders including OpenAI, Anthropic, Perplexity, ElevenLabs, Cursor (Anysphere), Stripe, Databricks, and SpaceX.
• Volume and Distribution: A comprehensive run on October 27, 2026, yielded 23,192 total jobs across 144 boards, broken down by ATS provider:
  • Greenhouse: 16,881 jobs
  • Ashby: 5,506 jobs
  • Lever: 805 jobs
  • Out of these, 4,303 roles were explicitly designated as remote.
• Flexibility: Beyond the pre-populated directory, both actors accept custom inputs. Users can ingest any company running on supported ATS platforms by passing a simple careers URL (https://jobs.lever.co/meesho), an explicit provider-slug pair (greenhouse:stripe), or a bare company slug.

Sample Input Schemas in Action

To demonstrate how these scrapers operate in practice, consider two common querying scenarios:

Example A: Filtering Data and ML Roles in Bengaluru (India Actor Schema)

  "useIndiaDirectory": true,
  "companies": ["https://jobs.lever.co/meesho", "greenhouse:stripe"],
  "indiaOnly": true,
  "includeGlobalRemote": false,
  "cities": ["Bengaluru", "Bangalore"],
  "keywords": ["data", "machine learning"],
  "excludeKeywords": ["intern"],
  "remoteOnly": false,
  "postedWithinDays": 7,
  "includeDescription": false,
  "maxItems": 500

Example B: Querying AI-Lab Engineering Roles in San Francisco or Remote (AI & Startup Actor Schema)

  "useDirectory": true,
  "sectors": ["ai"],
  "companies": [],
  "locations": ["San Francisco", "Remote"],
  "keywords": ["engineer"],
  "excludeKeywords": ["intern", "manager"],
  "remoteOnly": false,
  "postedWithinDays": 14,
  "includeDescription": false,
  "maxItems": 2000

Note: Leaving the sectors array empty forces the actor to sweep all 144 directory-listed companies indiscriminately.

Technical Integration: Python, JavaScript, and AI Agents

Integrating these job feeds into custom software pipelines is remarkably straightforward. Because the underlying actors are hosted on Apify, developers can interact with them using standard SDKs, command-line interfaces, or modern AI agent protocols.

Python Integration Example

Using the official apify-client package, developers can trigger an extraction run and stream the resulting items directly into their analytics pipelines:

from apify_client import ApifyClient

# Initialize the Apify client with your API token
client = ApifyClient("<YOUR_APIFY_TOKEN>")

# Define the execution parameters
run_input = 
    "useDirectory": True,
    "sectors": ["ai"],
    "locations": ["San Francisco", "Remote"],
    "keywords": ["engineer"],
    "postedWithinDays": 14,
    "maxItems": 2000,


# Run the AI & Startup Jobs Scraper actor
run = client.actor("prospect_antechamber/ai-startup-jobs-scraper").call(run_input=run_input)

# Iterate over the resulting dataset items
for job in client.dataset(run.default_dataset_id).iterate_items():
    print(job["company"], " | ", job["title"], " | ", job["location"], " | ", job["jobUrl"])

Multi-Platform Ecosystem Support

• JavaScript/TypeScript: Fully supported via the apify-client library available on npm.
• Command Line: Executable natively via the Apify CLI (apify call ... --output-dataset).
• AI Agents & LLMs: Can be plugged directly into AI workflows as an external tool using the Apify Model Context Protocol (MCP) server endpoint (https://mcp.apify.com/?tools=prospect_antechamber/ai-startup-jobs-scraper).

Standardized Output Schema

Regardless of whether the source is Greenhouse, Lever, Ashby, or SmartRecruiters, every scraped row is normalized into a uniform flat JSON schema:

  "ats": "ashby",
  "companySlug": "atlan",
  "company": "Atlan",
  "jobId": "20fe5046-b287-40f6-990f-842ad75526e5",
  "title": "Staff Software Engineer - Infrastructure (AI Native)",
  "location": "India",
  "allLocations": ["India"],
  "department": "Engineering",
  "team": "Engineering",
  "employmentType": "FullTime",
  "workplaceType": "Remote",
  "postedAt": "2026-10-23T14:44:45.243000+00:00",
  "updatedAt": null,
  "jobUrl": "https://jobs.ashbyhq.com/atlan/20fe5046-b287-40f6-990f-842ad75526e5",
  "applyUrl": "https://jobs.ashbyhq.com/atlan/20fe5046-b287-40f6-990f-842ad75526e5/application",
  "salary": null,
  "descriptionText": null,
  "isIndia": "true",
  "indiaCity": null,
  "isRemote": true,
  "scrapedAt": "2026-10-27T07:26:28.940092+00:00"

(Note: The AI & Startup actor appends an additional sectors array to categorize each record).

Implications: Building a Daily Hiring-Intent Intelligence Pipeline

The availability of structured, programmatic ATS feeds unlocks powerful downstream workflows for entrepreneurs, data analysts, and recruiters. A proven architecture for leveraging this data is the automated daily "new roles" feed.

 [ Scheduled Cron Trigger ] 
            │
            ▼
   [ Apify Actor Run ] ──(Queries Public ATS Feeds)
            │
            ▼
   [ JSON / CSV Dataset ] 
            │
      ┌─────┴─────┐
      ▼           ▼
[ Google Sheets / CRM ]   [ Webhook / Make / Zapier ]
      │           │
      └─────┬─────┘
            ▼
   [ Daily Newsletter / Hiring Intelligence Dashboard ]

Key Business and Analytical Benefits:

1. Automated Content Generation: Power niche job boards or curate specialized tech newsletters (e.g., "Remote AI Engineering Roles") with zero manual curation overhead.
2. Macroeconomic Market Signals: Track corporate expansion and contraction trends over time. A sudden surge in open infrastructure engineering roles at a specific fintech firm serves as an immediate, leading indicator of geographic expansion, new product launches, or capital influx.
3. Sales and Recruitment Prospecting: B2B sales teams can monitor when target client accounts begin aggressively hiring within specific technical divisions, signaling a budget-allocation phase ideal for outreach.

Official Responses, Pricing, and Limitations

While the methodology is efficient, practitioners must account for structural pricing models and inherent platform limitations.

Economic Model

Both actors operate on Apify’s transparent pay-per-event pricing model:

• India Tech Jobs Scraper: Charged at $1 per 1,000 jobs successfully returned.
• AI & Startup Jobs Scraper: Charged at $1.50 per 1,000 jobs returned.

Inherent Limitations

• ATS Dependency: The actors are strictly limited to companies utilizing Greenhouse, Lever, Ashby, or SmartRecruiters. Organizations using legacy enterprise systems (e.g., Workday, SuccessFactors) or custom-coded recruitment portals cannot be parsed through these standardized JSON endpoints.
• Missing Descriptions: To optimize speed and minimize data transfer payloads, full job descriptions are frequently excluded by default (includeDescription: false), requiring secondary fetching if deep semantic analysis of job text is necessary.
• Data Latency: While ATS endpoints reflect real-time updates made by corporate recruiters, companies occasionally fail to close filled positions promptly, leading to minor legacy overhang in listings.

Compliance and Legality

A critical differentiator of ATS-feed scraping compared to traditional web scraping is its passive nature. These tools do not bypass authentication walls, violate terms of service via aggressive headless browser automation, bypass CAPTCHAs, or harvest personal candidate data. They merely read the public, unauthenticated JSON endpoints explicitly published by ATS vendors for distribution. However, end-users remain legally responsible for ensuring that their downstream utilization and storage of the data comply with regional data protection regulations (such as GDPR or CCPA) and the specific terms of each source organization.

For developers seeking to expand coverage, maintainers actively accept community requests to integrate additional corporate careers pages into the core directories, marking a collaborative step forward in open recruitment intelligence.