IntegrationsLangChain

LangChain

What is LangChain

Connect Scrapeless to your Python LangChain application with langchain-scrapeless. Call tools directly to search or extract data, or register them with an agent so a model can select tools and use their results.

Start with the Google Search quickstart, then connect it to an agent or choose another product below.

Available tools

The integration exposes four product areas. Import the tool classes from langchain_scrapeless.

ProductWhen to use itTool classes
Scraping API — Google SearchFind search results and page URLsScrapelessDeepSerpGoogleSearchTool
CrawlExtract known URLs or discover pages from a websiteScrapelessCrawlerScrapeTool, ScrapelessCrawlerCrawlTool
Web UnlockerExtract a page with options such as JavaScript renderingScrapelessUniversalScrapingTool
AI ScraperCollect AI-platform answers and available sourcesScrapelessAIScraperTool, engine-specific tools such as ScrapelessChatGPTScraperTool, and ScrapelessAIScraperResultTool

Web Unlocker naming: The product is Web Unlocker. In langchain-scrapeless 0.1.4, its Python class is still named ScrapelessUniversalScrapingTool; use that exact name in imports and code.

Setup

You need Python 3.10 or later and a Scrapeless API key with access and credits for your chosen product. Direct tool calls require only this key. The optional agent example also requires a model-provider key.

This guide targets langchain-scrapeless 0.1.4, which installs Scrapeless Python SDK 1.3.0. The agent example uses LangChain 1.x.

Install the package

Create and activate a virtual environment.

macOS / Linux:

python3 -m venv .venv
source .venv/bin/activate

Windows PowerShell:

py -3 -m venv .venv
..venvScriptsActivate.ps1

Then install the integration:

python -m pip install "langchain-scrapeless==0.1.4"
python -m pip check

Configure your API key

Replace the placeholder and set the variable in the terminal where you will run Python.

macOS / Linux:

export SCRAPELESS_API_KEY="YOUR_SCRAPELESS_API_KEY"

Windows PowerShell:

$env:SCRAPELESS_API_KEY = "YOUR_SCRAPELESS_API_KEY"

The tools read SCRAPELESS_API_KEY automatically. Keep your key out of source files and shared logs.

Quickstart: search Google

Save this as search.py:

from langchain_scrapeless import ScrapelessDeepSerpGoogleSearchTool

tool = ScrapelessDeepSerpGoogleSearchTool(handle_tool_error=False)
result = tool.invoke({
    "q": "Scrapeless documentation",
    "hl": "en",
    "gl": "us",
})

items = result.get("organic_results") or []
if not items:
    print("No organic results returned. Inspect the response:", result)
for item in items[:3]:
    print(item.get("title"), item.get("link"))
    print(item.get("snippet", ""))

Run it in your activated environment:

python search.py

When organic results are available, the script prints up to three titles, URLs, and snippets. hl selects the search language and gl selects the country. An empty list alone does not establish that a request failed; inspect the full response.

Search snippets are summaries. Pass the returned URLs to Crawl when you need page content. The examples use handle_tool_error=False so LangChain tool exceptions propagate instead of becoming tool output.

Connect a LangChain agent

The agent uses a model to select tools and compose answers. Scrapeless executes the search. Model-provider usage is separate from Scrapeless usage.

Install and configure the model integration

In the same environment, install:

python -m pip install "langchain>=1,<2" "langchain-openai>=1,<2"
python -m pip check

Keep SCRAPELESS_API_KEY set and add your OpenAI credentials.

macOS / Linux:

export OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
export OPENAI_MODEL="YOUR_TOOL_CALLING_MODEL"

Windows PowerShell:

$env:OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"
$env:OPENAI_MODEL = "YOUR_TOOL_CALLING_MODEL"

Replace the model placeholder with a model ID available to your account that supports tool calling. OPENAI_MODEL is a variable used explicitly by this example.

Create and run the agent

Save as agent.py:

import os

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langchain_scrapeless import ScrapelessDeepSerpGoogleSearchTool

model = ChatOpenAI(model=os.environ["OPENAI_MODEL"])
search_tool = ScrapelessDeepSerpGoogleSearchTool(handle_tool_error=False)
agent = create_agent(model=model, tools=[search_tool])

result = agent.invoke({
    "messages": [{
        "role": "user",
        "content": (
            "Search the web for Scrapeless documentation and return three "
            "relevant documentation links with a short description of each."
        ),
    }],
})

# Show the model's tool calls, tool responses, and final answer.
for message in result["messages"]:
    message.pretty_print()

Run python agent.py. In the printed messages, look for a call to scrapeless_deepserp_google_search, its tool response, and a final answer based on those results. Inspect the response for usable data; a model answer alone does not confirm that Scrapeless was called.

Registering a tool makes it available but does not force a call on every request. To add another capability, create the corresponding tool instance and include it in tools=[...].

More product examples

These examples are independent. Save each block as a .py file and run python filename.py in the configured environment.

Crawl

Use ScrapelessCrawlerScrapeTool for known URLs. Supply one URL or a batch in urls:

from langchain_scrapeless import ScrapelessCrawlerScrapeTool

tool = ScrapelessCrawlerScrapeTool(handle_tool_error=False)
result = tool.invoke({
    "urls": ["https://example.com"],
    "formats": ["markdown"],
    "only_main_content": True,
    "poll_timeout": 180,
})

if result.get("status") != "completed":
    raise RuntimeError(f"Scrape did not complete: {result.get('status')}")

pages = result.get("data") or []
print("Returned pages:", len(pages))
for page in pages:
    metadata = page.get("metadata") or {}
    print(metadata.get("sourceURL"), metadata.get("statusCode"))
    print(page.get("markdown", ""))

The call waits for the batch task. Check each page’s status and content as well as the task status. poll_timeout is measured in seconds; the separate page timeout parameter is measured in milliseconds.

To discover linked pages instead, use ScrapelessCrawlerCrawlTool with {"url": "https://example.com", "limit": 5, "formats": ["markdown"]}. It also waits for completion and returns a dictionary with status and data. limit is a maximum, so fewer pages may be returned. This tool does not expose poll_timeout in version 0.1.4.

Web Unlocker

Request Markdown with JavaScript rendering enabled:

from pathlib import Path

from langchain_scrapeless import ScrapelessUniversalScrapingTool

tool = ScrapelessUniversalScrapingTool(handle_tool_error=False)
content = tool.invoke({
    "url": "https://example.com",
    "js_render": True,
    "response_type": "markdown",
})

if not isinstance(content, str) or not content.strip():
    raise RuntimeError("Expected non-empty Markdown text; inspect the response")

Path("page.md").write_text(content, encoding="utf-8")
print("Saved page.md")

Open page.md to check the extracted body. Text formats include markdown, html, and plaintext. Use returned text directly; it does not have Crawl’s status and data fields. Keep the input URL with the content as its source.

AI Scraper

Use an engine-specific tool or select an engine through ScrapelessAIScraperTool. Supported engine values are chatgpt, perplexity, copilot, gemini, aimode, overview, grok, and alexa.

This example uses the ChatGPT engine within AI Scraper:

from langchain_scrapeless import ScrapelessChatGPTScraperTool

tool = ScrapelessChatGPTScraperTool(handle_tool_error=False)
task = tool.invoke({
    "prompt": "Explain HTTP caching and include sources where available.",
    "country": "US",
    "web_search": True,
    "shopping": False,
    "output_format": "json",
})

if task.get("status") != "success":
    raise RuntimeError(f"Task did not complete: {task.get('task_id')}")

answer = task["task_result"]
print(answer.get("result_text", ""))
print("Links:", answer.get("links") or [])
print("Content references:", answer.get("content_references") or [])

The ChatGPT tool returns text by default; output_format="json" returns a Python dictionary containing task status and result fields. The unified tool and other engine-specific tools default to a full task dictionary. Inputs and answer fields vary by engine; for example, Copilot requires mode. Sources may be empty, and Google AI Overview may complete without an overview.

This call requires only SCRAPELESS_API_KEY. AI Scraper collects platform answers; the ChatOpenAI model in the agent example has a separate role and key.

Task handling

Most examples above wait for results. For longer workflows:

  • AI Scraper: set wait=False and output_format="json" in the engine input, save task_id, and retrieve it with ScrapelessAIScraperResultTool. Pass the same engine. Lookup supports wait, poll_timeout, and poll_interval; the latter two use seconds.

  • Batch scrape: if polling times out, keep the task ID in the error. Use client.scraping_crawl.scrape.check_batch_scrape_status(task_id) through the Python SDK to query the existing batch.

  • Website crawl: use the SDK’s async_crawl_url() to submit and check_crawl_status() to retrieve the saved task. The status lookup returns an object with attributes such as status and data.

These are remote-task workflows, separate from Python async/await. A polling timeout does not cancel the remote task. Query the existing ID before submitting again. See the Python SDK guide and package examples for detailed usage.

Troubleshooting

ProblemWhat to check
Import or dependency errorUse Python 3.10+ and the same virtual environment for installation and execution. Run python -m pip check and python -m pip show langchain-scrapeless scrapeless langchain-core.
Missing key or authentication failureSet the key in the process running Python. Check validity, product access, and credits. The agent’s model-provider key is separate.
Parameter validation errorCheck the selected tool and engine schema. For the unified AI Scraper tool, put engine-specific fields inside input. SDK, REST, and tool parameter names can differ.
TimeoutPreserve the task ID and follow the lookup guidance above. Re-running submission may create another task.
Empty or unexpected outputInspect the full response, task state, and requested content fields. For Web Unlocker text-decoding errors, include the SDK version and sanitized traceback when reporting the issue.
Agent does not searchVerify the direct quickstart first. Check tool registration and model tool-calling support, then inspect the execution messages with an explicit search request.