Scrapeless CLI
Overview
The Scrapeless CLI provides a single scrapeless command for web extraction, website crawling, structured Google Search results, AI answers, proxy URL generation, and cloud browser operations. Run it in a terminal or invoke it from your automation scripts.
Features
| Feature | Command entry point | Purpose |
|---|---|---|
| Web Unlocker | scrapeless unlock / scrapeless scrape | Extract a page in HTML, Markdown, or other supported formats. |
| Crawl | scrapeless crawl | Create a website crawl task and retrieve its results. |
| AI Scraper | scrapeless ai | Collect answers from supported AI platforms. |
| Google Search | scrapeless scraping-api google-search | Retrieve structured Google Search results with language, location, and pagination options. |
| Proxies | scrapeless proxy create | Generate an authenticated proxy URL locally. |
| Cloud browser | scrapeless browser | Run the upstream cloud browser CLI. |
api is an alias for scraping-api. For example, scrapeless api google-search and scrapeless scraping-api google-search run the same command.
Get Started
- Quickstart — install, configure your key, and make your first request.
- Usage Examples — commands for each feature.
- Configuration and Tasks — credentials, output, polling, and browser requirements.
Quickstart
Requirements
Use Node.js 22 or later and npm. Node.js is the CLI runtime; you do not need to write JavaScript or TypeScript. A Scrapeless account and API key are required for authenticated services.
1. Install the CLI
npm install -g scrapeless
scrapeless --version
scrapeless --help2. Configure Your API Key
Copy your API key from the Scrapeless dashboard and replace YOUR_API_TOKEN:
- Open Settings in the dashboard.
- Select API Key Management.
- Use an available key, or choose Create API Key if you need one. Copy the key for the configuration command below.
scrapeless config set api_token YOUR_API_TOKEN
scrapeless auth statusauth status checks authentication and account balance. Cloud requests and proxy traffic consume account credits.
3. Extract a Page
scrapeless unlock https://example.com --type markdownThe command prints the page’s extracted Markdown.
Inspect the Full Response
Add --json to inspect the full response, including its code and page content:
scrapeless unlock https://example.com --type markdown --json4. Search Google
Run a search with a query, language, and country:
scrapeless scraping-api google-search --q "coffee" --hl en --gl usThe command prints the complete result JSON. To save it to a file:
scrapeless scraping-api google-search --q "coffee" --hl en --gl us -o search.jsonIf your installed CLI does not recognize scraping-api, update it and check the command help:
npm install -g scrapeless@latest
scrapeless scraping-api google-search --helpUsage Examples
Install the CLI and configure your API key before running these examples.
Web Unlocker
Extract a selected region as HTML, or save a full JSON response:
scrapeless unlock https://example.com --type html --selector body
scrapeless unlock https://example.com --type markdown --json -o page.jsonscrape is an alias for unlock. The default output is HTML, with JavaScript rendering enabled.
Crawl
scrapeless crawl https://example.com --limit 3 --max-depth 2 --formats markdown --json -o crawl.json--formats markdown selects page content format; the overall task result remains JSON. Scope a crawl with --include-paths and --exclude-paths.
AI Scraper
scrapeless ai platforms
scrapeless ai chatgpt "Explain what Scrapeless is used for" -o answer.md
scrapeless ai perplexity "What web scraping techniques are available?" --web-search true --json -o answer.jsonSuccessful tasks return Markdown by default. Options vary by platform; use ai platforms for the current list. English prompts are examples: the CLI accepts Unicode text, and language availability depends on the selected service.
Google Search
Use scraping-api google-search, or its shorter alias api google-search, to retrieve structured search results.
# Search with a Google search operator and a specific location
scrapeless api google-search \
--q "site:example.com coffee" \
--location "Seattle,Washington,United States"
# Request results starting at offset 10 and save the JSON
scrapeless api google-search \
--q "coffee" --hl en --gl us --start 10 \
-o search.json
# Search Google News
scrapeless api google-search --q "artificial intelligence" --tbm nwsCommon options:
| Option | Description |
|---|---|
--q | Required search query. Supports Google search operators. Can also be supplied as input.q through --data. |
--hl, --gl | Search language and country codes, such as en and us. |
--location, --uule | Canonical location or encoded Google location. Use only one. |
--google-domain | Google domain, such as google.com. |
--tbm | Search type, such as nws for news, isch for images, or lcl for local results. |
--start | Non-negative result offset. For --tbm lcl, use multiples of 20. |
--safe | SafeSearch setting: active or off. |
--device | Currently accepts desktop only. |
--country, --proxy-url | Proxy country or custom proxy URL. |
--timeout, --interval | Polling timeout and interval, in seconds. Defaults: 300 and 5. |
-o, --output | Save the result JSON to a file. |
--gl controls the search country; --country configures the proxy country. These are separate settings. A non-empty --proxy-url takes precedence over the proxy country.
For additional request fields, pass a complete JSON body. Save the following as search-request.json:
{
"actor": "scraper.google.search",
"input": {
"q": "coffee",
"hl": "en",
"gl": "us",
"as_eq": "instant"
},
"proxy": {
"country": "US"
}
}Run the request:
scrapeless api google-search --data @search-request.json -o search.jsonExplicit flags override corresponding body fields. The command always sets the actor to scraper.google.search.
Run scrapeless scraping-api google-search --help for all CLI options. See the Google Search documentation for API details.
Proxies
scrapeless proxy gateways
scrapeless proxy create --type residential --country US --gateway usURL generation happens locally without testing connectivity. Supported types are residential, ipv6, and datacenter; gateways accept us, eu, ap, or hostname:port. The URL contains credentials; keep it private.
Cloud Browser
Replace SESSION_ID with the session ID returned by new-session:
scrapeless browser new-session --name demo --ttl 300 --json
scrapeless browser --session-id SESSION_ID open https://example.com
scrapeless browser --session-id SESSION_ID snapshot -i
scrapeless browser --session-id SESSION_ID screenshot page.png
scrapeless browser --session-id SESSION_ID closeFor click @e1 or fill @e2 "hello world", use actual interactive references from a fresh snapshot while the session is open. By default, first use, including browser --help, installs the upstream CLI; see Browser Requirements in Configuration and Tasks.
For command options, run scrapeless unlock --help, scrapeless crawl --help, scrapeless ai --help, scrapeless scraping-api google-search --help, or scrapeless proxy create --help. The first-use installation behavior of browser --help is described above.
Full command reference and source-build instructions are in the CLI repository. AI service behavior and billing are covered in the AI Scraper documentation.
Configuration and Tasks
Credentials
The default file is ~/.scrapeless/config.json. Key precedence is configuration file → SCRAPELESS_API_TOKEN → SCRAPELESS_API_KEY. To use an environment variable:
export SCRAPELESS_API_KEY="YOUR_API_TOKEN"A saved key takes precedence. The CLI does not load .env automatically. This is POSIX syntax; PowerShell uses $env:SCRAPELESS_API_KEY="YOUR_API_TOKEN".
| Variable | Purpose |
|---|---|
SCRAPELESS_API_KEY | API key; also supported by Browser. |
SCRAPELESS_API_TOKEN | Alternative key variable for unified commands. |
SCRAPELESS_CONFIG_PATH | Custom configuration path for unified commands only. |
SCRAPELESS_API_BASE_URL | Unified HTTP API override; defaults to https://api.scrapeless.com. |
SCRAPELESS_CACHE_DIR | Browser cache root; defaults to ~/.cache/scrapeless. |
SCRAPELESS_BROWSER_BIN | Absolute path to a compatible executable; skips automatic installation. |
A custom config path does not change Browser’s default file. The wrapper passes its resolved key via an environment variable, but Browser’s default saved key can override it. Use the shared default file to keep authentication consistent.
Output and Request Bodies
HTTP commands support --json for the full response and -o, --output <file> to save output. By default, results go to stdout; with -o, they are saved to the file instead. Progress, task-ID notices during polling, and errors go to stderr. For Crawl and AI creation, --no-wait includes the creation response and its task ID in the result output.
Unlocker, Crawl, AI, and Scraping API creation commands accept --data '<JSON>' or --data @request.json. Precedence is explicit flags → body fields → CLI defaults; nested objects merge and arrays replace.
Google Search always returns the complete result JSON, even when --json is omitted. Use -o, --output <file> to save it. Task-ID notices and errors go to stderr.
Task Handling
Crawl and AI creation poll every 5 seconds for up to 300 seconds by default. Add --no-wait to return after creation. Query an existing task with its returned ID:
scrapeless crawl result TASK_ID --wait --timeout 300 --interval 5
scrapeless ai result TASK_ID --platform chatgpt --wait --timeout 300 --interval 5Use the ID for the corresponding service. A polling timeout does not cancel the server task. Check completed for Crawl or success for AI and inspect actual content; exit code 0 alone is insufficient.
--timeout and --interval use seconds; scraping options --wait-for and --scrape-timeout use milliseconds. Each HTTP request has a separate 60-second timeout. Task-creation POSTs are not retried automatically.
Google Search Tasks
Google Search returns results immediately on HTTP 200. If the API responds with HTTP 201 and a taskId, the CLI automatically polls for the result every 5 seconds for up to 300 seconds. Adjust these values with --interval and --timeout.
To resume checking an existing task without creating another request:
scrapeless scraping-api result TASK_ID \
--timeout 300 --interval 5 \
-o search.jsonThe result command polls automatically; no --wait flag is needed.
HTTP 4xx or 5xx responses and polling timeouts exit with code 1. On a polling timeout, stderr includes the task ID, last response, and a command to continue checking. The timeout does not cancel the server task. Task-creation POST requests are not retried automatically.
Browser Requirements
The wrapper needs npm, access to npm and GitHub Releases, and cache write permissions. Sessions run in the cloud; the wrapper runs locally without a separate Chromium install.
The pinned release supports macOS x64/arm64, Linux x64/arm64, and Windows x64; no Windows arm64 binary is bundled.
Troubleshooting
| Issue | What to check |
|---|---|
| Missing key or HTTP 401/403 | Check the saved key, account permissions, and balance; run scrapeless auth status. |
| Crawl or AI polling timeout | Keep the task ID and use the result ... --wait commands above; inspect status and actual content. |
| Browser installation fails | Check npm, network access, cache write permissions, and the platform requirements above. |
Unknown scraping-api command | Update with npm install -g scrapeless@latest, then run scrapeless scraping-api google-search --help. |
| Google Search polling timeout | Use the reported task ID with scrapeless scraping-api result TASK_ID. Increase --timeout if needed. |
| Google Search parameter validation fails | Provide a non-empty query, use only one of --location and --uule, and use a non-negative --start. Local search requires offsets in multiples of 20. |