Start with introspection if it is enabled, then verify everything against real browser traffic, and use JavaScript analysis when the schema is hidden or queries are persisted. That order saves time, reduces guesswork, and keeps your scraper closer to how the site actually works.
TLDR: GraphQL reverse web scraping is fastest when you combine three sources: introspection, public or leaked API documentation, and JavaScript bundle analysis. For example, on a retail site with 50 product pages, introspection might reveal product fields in under a minute, while JavaScript analysis may uncover the exact persisted query hash used by the browser. In one common workflow, teams spend 70% less time mapping endpoints when introspection is available, but still need browser traffic to confirm filters, headers, and pagination.
Why GraphQL scraping feels different
REST scraping is often a hunt for many endpoints. GraphQL scraping is usually a hunt for one endpoint and many operations. That sounds easier. Sometimes it is. Other times, the endpoint accepts only signed requests, persisted hashes, strict headers, or variables that are buried deep inside minified JavaScript.
With GraphQL, the browser may call something like:
POST /graphql
{
"operationName": "ProductSearch",
"variables": {
"query": "laptop",
"page": 2
},
"query": "query ProductSearch(...) { ... }"
}The endpoint stays the same, but the query controls the data. That is why reverse web scraping GraphQL is less about finding URLs and more about understanding schema, operations, variables, headers, and client behavior.
Method 1: GraphQL introspection
GraphQL introspection is the cleanest starting point. It asks the server to describe its own types, fields, arguments, enums, and relationships. If enabled, it can expose a full map of the API.
A minimal JavaScript introspection check looks like this:
const response = await fetch("https://example.com/graphql", {
method: "POST",
headers: {
"content-type": "application/json"
},
body: JSON.stringify({
query: `{
__schema {
queryType { name }
types { name kind }
}
}`
})
});
console.log(await response.json());If the response includes types such as Product, Price, Review, or SearchResult, you have a strong lead. You can then query deeper with tools such as GraphiQL, Altair, Insomnia, or Postman.
The catch is that many production APIs disable introspection. You may get errors like GraphQL introspection is not allowed. Annoying, yes. Also common. Some teams disable it for security, while others rely on persisted queries and never send full query text from the browser.
What introspection gives you
- Type names: Useful for finding products, users, stores, orders, or reviews.
- Field lists: Shows available data, not just what the page displays.
- Arguments: Reveals filters such as category, sort, locale, page size, or date range.
- Enums: Helps avoid invalid values in scrape requests.
- Relationships: Shows how objects connect, such as products to sellers or reviews.
Method 2: API documentation
API documentation is more human friendly than introspection. It may explain authentication, rate limits, error codes, pagination rules, and request examples. That context matters. A schema can tell you that first is an argument. Docs may tell you that the maximum value is 100 and anything higher gets silently capped.
Docs show up in many places. Check developer portals, OpenAPI files that mention GraphQL, public GitHub repositories, mobile app docs, SDK packages, and old staging domains. Also inspect npm packages from the same company. It drives me crazy that useful request examples are sometimes hidden in a forgotten SDK while the official docs say almost nothing.
Documentation is still not truth. Browser traffic is truth. Docs drift. Fields get renamed. Examples use old auth formats. A documented query may fail because the frontend now sends extra headers such as x-client-name, apollographql-client-version, or a locale cookie.
Introspection vs documentation
| Source | Best for | Main weakness |
|---|---|---|
| Introspection | Finding exact schema, fields, types, arguments | Often disabled in production |
| API docs | Understanding auth, limits, examples, business rules | May be stale or incomplete |
| JavaScript analysis | Recovering real operations, variables, hashes, headers | Minified code can be painful |
Method 3: JavaScript analysis tools
When introspection fails and docs are weak, inspect the frontend. The browser already knows how to talk to the GraphQL API. Your job is to observe and copy the relevant behavior without copying fragile noise.
Start with Chrome DevTools or Firefox Developer Tools. Open the Network tab, filter by graphql, api, or fetch, then reload the page. Look for POST requests with JSON bodies. Save the request as cURL. Convert it to JavaScript fetch if needed.
Next, inspect the bundled JavaScript. Search for:
operationNamegql`graphqlpersistedQuerysha256Hash__typename- Field names seen on the page, such as
salePriceoravailability
Useful tools include ripgrep for fast text search, Prettier for formatting bundles, source-map-explorer for readable source maps, and @babel/parser for AST-based extraction. For proxied traffic, mitmproxy, Charles Proxy, and HTTP Toolkit can be handy.
Persisted queries: the common headache
Many GraphQL clients send only a hash instead of the full query. You might see this:
{
"operationName": "ProductSearch",
"variables": { "query": "desk" },
"extensions": {
"persistedQuery": {
"version": 1,
"sha256Hash": "b4f1..."
}
}
}Honestly, it feels like the API is taunting you. The query exists somewhere, but not always in the request. Sometimes it is inside a JavaScript bundle. Sometimes it is downloaded from a manifest file. Sometimes it is generated during build time and mapped by operation name.
Search the bundles for the hash, the operation name, or unique field names shown in the response. If source maps are exposed, grab them quickly. They can turn unreadable chunks into source files with folder names, component names, and GraphQL documents.
A practical JavaScript workflow
- Record real traffic. Load several pages and capture GraphQL requests.
- Group by operationName. Separate search, product detail, reviews, stock, and recommendations.
- Test introspection. If enabled, export the schema.
- Search documentation. Check official docs, SDKs, npm packages, and public repos.
- Analyze bundles. Find queries, hashes, headers, and variable builders.
- Replay with fetch. Start with copied browser headers, then remove extras one by one.
- Stabilize pagination. Confirm cursors, limits, sorting, and duplicate handling.
- Add polite limits. Use backoff, caching, and request spacing.
Ethics, rate limits, and fragile assumptions
Only scrape data you are allowed to access. Respect robots policies, terms, account rules, and rate limits. Do not bypass paywalls, private data controls, or authentication systems. If an endpoint exposes more data than the page shows, treat that as sensitive until you have permission.
GraphQL scrapers also break in subtle ways. A frontend release can rename an operation, rotate persisted hashes, change cursor rules, or require a new client header. Build monitoring around response shape, error rates, and record counts. If yesterday’s scraper collected 10,000 items and today it gets 600, stop and inspect before trusting the output.
Best choice for each situation
- Use introspection when it is enabled and you need a fast schema map.
- Use documentation when you need rules, limits, and supported usage patterns.
- Use JavaScript analysis when the browser works but the API is poorly documented or uses persisted queries.
The strongest approach is not choosing one method. It is combining all three. Introspection shows what exists. Documentation explains what should happen. JavaScript analysis proves what the site actually sends. For reverse web scraping GraphQL with JavaScript, that mix gives you the best shot at clean data without wasting hours on blind guesses.