Scrapeless CLI

Overview

The Scrapeless CLI provides a single scrapeless command for web extraction, website crawling, structured Google Search results, AI answers, proxy URL generation, and cloud browser operations. Run it in a terminal or invoke it from your automation scripts.

Features

FeatureCommand entry pointPurpose
Web Unlockerscrapeless unlock / scrapeless scrapeExtract a page in HTML, Markdown, or other supported formats.
Crawlscrapeless crawlCreate a website crawl task and retrieve its results.
AI Scraperscrapeless aiCollect answers from supported AI platforms.
Google Searchscrapeless scraping-api google-searchRetrieve structured Google Search results with language, location, and pagination options.
Proxiesscrapeless proxy createGenerate an authenticated proxy URL locally.
Cloud browserscrapeless browserRun the upstream cloud browser CLI.

api is an alias for scraping-api. For example, scrapeless api google-search and scrapeless scraping-api google-search run the same command.

Get Started

Quickstart

Requirements

Use Node.js 22 or later and npm. Node.js is the CLI runtime; you do not need to write JavaScript or TypeScript. A Scrapeless account and API key are required for authenticated services.

1. Install the CLI

npm install -g scrapeless
scrapeless --version
scrapeless --help

2. Configure Your API Key

Copy your API key from the Scrapeless dashboard and replace YOUR_API_TOKEN:

  1. Open Settings in the dashboard.
  2. Select API Key Management.
  3. Use an available key, or choose Create API Key if you need one. Copy the key for the configuration command below.
scrapeless config set api_token YOUR_API_TOKEN
scrapeless auth status

auth status checks authentication and account balance. Cloud requests and proxy traffic consume account credits.

3. Extract a Page

scrapeless unlock https://example.com --type markdown

The command prints the page’s extracted Markdown.

Inspect the Full Response

Add --json to inspect the full response, including its code and page content:

scrapeless unlock https://example.com --type markdown --json

4. Search Google

Run a search with a query, language, and country:

scrapeless scraping-api google-search --q "coffee" --hl en --gl us

The command prints the complete result JSON. To save it to a file:

scrapeless scraping-api google-search --q "coffee" --hl en --gl us -o search.json

If your installed CLI does not recognize scraping-api, update it and check the command help:

npm install -g scrapeless@latest
scrapeless scraping-api google-search --help

Usage Examples

Install the CLI and configure your API key before running these examples.

Web Unlocker

Extract a selected region as HTML, or save a full JSON response:

scrapeless unlock https://example.com --type html --selector body
scrapeless unlock https://example.com --type markdown --json -o page.json

scrape is an alias for unlock. The default output is HTML, with JavaScript rendering enabled.

Crawl

scrapeless crawl https://example.com --limit 3 --max-depth 2 --formats markdown --json -o crawl.json

--formats markdown selects page content format; the overall task result remains JSON. Scope a crawl with --include-paths and --exclude-paths.

AI Scraper

scrapeless ai platforms
scrapeless ai chatgpt "Explain what Scrapeless is used for" -o answer.md
scrapeless ai perplexity "What web scraping techniques are available?" --web-search true --json -o answer.json

Successful tasks return Markdown by default. Options vary by platform; use ai platforms for the current list. English prompts are examples: the CLI accepts Unicode text, and language availability depends on the selected service.

Use scraping-api google-search, or its shorter alias api google-search, to retrieve structured search results.

# Search with a Google search operator and a specific location
scrapeless api google-search \
  --q "site:example.com coffee" \
  --location "Seattle,Washington,United States"
 
# Request results starting at offset 10 and save the JSON
scrapeless api google-search \
  --q "coffee" --hl en --gl us --start 10 \
  -o search.json
 
# Search Google News
scrapeless api google-search --q "artificial intelligence" --tbm nws

Common options:

OptionDescription
--qRequired search query. Supports Google search operators. Can also be supplied as input.q through --data.
--hl, --glSearch language and country codes, such as en and us.
--location, --uuleCanonical location or encoded Google location. Use only one.
--google-domainGoogle domain, such as google.com.
--tbmSearch type, such as nws for news, isch for images, or lcl for local results.
--startNon-negative result offset. For --tbm lcl, use multiples of 20.
--safeSafeSearch setting: active or off.
--deviceCurrently accepts desktop only.
--country, --proxy-urlProxy country or custom proxy URL.
--timeout, --intervalPolling timeout and interval, in seconds. Defaults: 300 and 5.
-o, --outputSave the result JSON to a file.

--gl controls the search country; --country configures the proxy country. These are separate settings. A non-empty --proxy-url takes precedence over the proxy country.

For additional request fields, pass a complete JSON body. Save the following as search-request.json:

{
  "actor": "scraper.google.search",
  "input": {
    "q": "coffee",
    "hl": "en",
    "gl": "us",
    "as_eq": "instant"
  },
  "proxy": {
    "country": "US"
  }
}

Run the request:

scrapeless api google-search --data @search-request.json -o search.json

Explicit flags override corresponding body fields. The command always sets the actor to scraper.google.search.

Run scrapeless scraping-api google-search --help for all CLI options. See the Google Search documentation for API details.

Proxies

scrapeless proxy gateways
scrapeless proxy create --type residential --country US --gateway us

URL generation happens locally without testing connectivity. Supported types are residential, ipv6, and datacenter; gateways accept us, eu, ap, or hostname:port. The URL contains credentials; keep it private.

Cloud Browser

Replace SESSION_ID with the session ID returned by new-session:

scrapeless browser new-session --name demo --ttl 300 --json
scrapeless browser --session-id SESSION_ID open https://example.com
scrapeless browser --session-id SESSION_ID snapshot -i
scrapeless browser --session-id SESSION_ID screenshot page.png
scrapeless browser --session-id SESSION_ID close

For click @e1 or fill @e2 "hello world", use actual interactive references from a fresh snapshot while the session is open. By default, first use, including browser --help, installs the upstream CLI; see Browser Requirements in Configuration and Tasks.

For command options, run scrapeless unlock --help, scrapeless crawl --help, scrapeless ai --help, scrapeless scraping-api google-search --help, or scrapeless proxy create --help. The first-use installation behavior of browser --help is described above.

Full command reference and source-build instructions are in the CLI repository. AI service behavior and billing are covered in the AI Scraper documentation.

Configuration and Tasks

Credentials

The default file is ~/.scrapeless/config.json. Key precedence is configuration file → SCRAPELESS_API_TOKEN → SCRAPELESS_API_KEY. To use an environment variable:

export SCRAPELESS_API_KEY="YOUR_API_TOKEN"

A saved key takes precedence. The CLI does not load .env automatically. This is POSIX syntax; PowerShell uses $env:SCRAPELESS_API_KEY="YOUR_API_TOKEN".

VariablePurpose
SCRAPELESS_API_KEYAPI key; also supported by Browser.
SCRAPELESS_API_TOKENAlternative key variable for unified commands.
SCRAPELESS_CONFIG_PATHCustom configuration path for unified commands only.
SCRAPELESS_API_BASE_URLUnified HTTP API override; defaults to https://api.scrapeless.com.
SCRAPELESS_CACHE_DIRBrowser cache root; defaults to ~/.cache/scrapeless.
SCRAPELESS_BROWSER_BINAbsolute path to a compatible executable; skips automatic installation.

A custom config path does not change Browser’s default file. The wrapper passes its resolved key via an environment variable, but Browser’s default saved key can override it. Use the shared default file to keep authentication consistent.

Output and Request Bodies

HTTP commands support --json for the full response and -o, --output <file> to save output. By default, results go to stdout; with -o, they are saved to the file instead. Progress, task-ID notices during polling, and errors go to stderr. For Crawl and AI creation, --no-wait includes the creation response and its task ID in the result output.

Unlocker, Crawl, AI, and Scraping API creation commands accept --data '<JSON>' or --data @request.json. Precedence is explicit flags → body fields → CLI defaults; nested objects merge and arrays replace.

Google Search always returns the complete result JSON, even when --json is omitted. Use -o, --output <file> to save it. Task-ID notices and errors go to stderr.

Task Handling

Crawl and AI creation poll every 5 seconds for up to 300 seconds by default. Add --no-wait to return after creation. Query an existing task with its returned ID:

scrapeless crawl result TASK_ID --wait --timeout 300 --interval 5
scrapeless ai result TASK_ID --platform chatgpt --wait --timeout 300 --interval 5

Use the ID for the corresponding service. A polling timeout does not cancel the server task. Check completed for Crawl or success for AI and inspect actual content; exit code 0 alone is insufficient.

--timeout and --interval use seconds; scraping options --wait-for and --scrape-timeout use milliseconds. Each HTTP request has a separate 60-second timeout. Task-creation POSTs are not retried automatically.

Google Search Tasks

Google Search returns results immediately on HTTP 200. If the API responds with HTTP 201 and a taskId, the CLI automatically polls for the result every 5 seconds for up to 300 seconds. Adjust these values with --interval and --timeout.

To resume checking an existing task without creating another request:

scrapeless scraping-api result TASK_ID \
  --timeout 300 --interval 5 \
  -o search.json

The result command polls automatically; no --wait flag is needed.

HTTP 4xx or 5xx responses and polling timeouts exit with code 1. On a polling timeout, stderr includes the task ID, last response, and a command to continue checking. The timeout does not cancel the server task. Task-creation POST requests are not retried automatically.

Browser Requirements

The wrapper needs npm, access to npm and GitHub Releases, and cache write permissions. Sessions run in the cloud; the wrapper runs locally without a separate Chromium install.

The pinned release supports macOS x64/arm64, Linux x64/arm64, and Windows x64; no Windows arm64 binary is bundled.

Troubleshooting

IssueWhat to check
Missing key or HTTP 401/403Check the saved key, account permissions, and balance; run scrapeless auth status.
Crawl or AI polling timeoutKeep the task ID and use the result ... --wait commands above; inspect status and actual content.
Browser installation failsCheck npm, network access, cache write permissions, and the platform requirements above.
Unknown scraping-api commandUpdate with npm install -g scrapeless@latest, then run scrapeless scraping-api google-search --help.
Google Search polling timeoutUse the reported task ID with scrapeless scraping-api result TASK_ID. Increase --timeout if needed.
Google Search parameter validation failsProvide a non-empty query, use only one of --location and --uule, and use a non-negative --start. Local search requires offsets in multiples of 20.