LangChain
What is LangChain
Connect Scrapeless to your Python LangChain application with langchain-scrapeless. Call tools directly to search or extract data, or register them with an agent so a model can select tools and use their results.
Start with the Google Search quickstart, then connect it to an agent or choose another product below.
Available tools
The integration exposes four product areas. Import the tool classes from langchain_scrapeless.
| Product | When to use it | Tool classes |
|---|---|---|
| Scraping API — Google Search | Find search results and page URLs | ScrapelessDeepSerpGoogleSearchTool |
| Crawl | Extract known URLs or discover pages from a website | ScrapelessCrawlerScrapeTool, ScrapelessCrawlerCrawlTool |
| Web Unlocker | Extract a page with options such as JavaScript rendering | ScrapelessUniversalScrapingTool |
| AI Scraper | Collect AI-platform answers and available sources | ScrapelessAIScraperTool, engine-specific tools such as ScrapelessChatGPTScraperTool, and ScrapelessAIScraperResultTool |
Web Unlocker naming: The product is Web Unlocker. In langchain-scrapeless 0.1.4, its Python class is still named ScrapelessUniversalScrapingTool; use that exact name in imports and code.
Setup
You need Python 3.10 or later and a Scrapeless API key with access and credits for your chosen product. Direct tool calls require only this key. The optional agent example also requires a model-provider key.
This guide targets langchain-scrapeless 0.1.4, which installs Scrapeless Python SDK 1.3.0. The agent example uses LangChain 1.x.
Install the package
Create and activate a virtual environment.
macOS / Linux:
python3 -m venv .venv
source .venv/bin/activateWindows PowerShell:
py -3 -m venv .venv
..venvScriptsActivate.ps1Then install the integration:
python -m pip install "langchain-scrapeless==0.1.4"
python -m pip checkConfigure your API key
Replace the placeholder and set the variable in the terminal where you will run Python.
macOS / Linux:
export SCRAPELESS_API_KEY="YOUR_SCRAPELESS_API_KEY"Windows PowerShell:
$env:SCRAPELESS_API_KEY = "YOUR_SCRAPELESS_API_KEY"The tools read SCRAPELESS_API_KEY automatically. Keep your key out of source files and shared logs.
Quickstart: search Google
Save this as search.py:
from langchain_scrapeless import ScrapelessDeepSerpGoogleSearchTool
tool = ScrapelessDeepSerpGoogleSearchTool(handle_tool_error=False)
result = tool.invoke({
"q": "Scrapeless documentation",
"hl": "en",
"gl": "us",
})
items = result.get("organic_results") or []
if not items:
print("No organic results returned. Inspect the response:", result)
for item in items[:3]:
print(item.get("title"), item.get("link"))
print(item.get("snippet", ""))Run it in your activated environment:
python search.pyWhen organic results are available, the script prints up to three titles, URLs, and snippets. hl selects the search language and gl selects the country. An empty list alone does not establish that a request failed; inspect the full response.
Search snippets are summaries. Pass the returned URLs to Crawl when you need page content. The examples use handle_tool_error=False so LangChain tool exceptions propagate instead of becoming tool output.
Connect a LangChain agent
The agent uses a model to select tools and compose answers. Scrapeless executes the search. Model-provider usage is separate from Scrapeless usage.
Install and configure the model integration
In the same environment, install:
python -m pip install "langchain>=1,<2" "langchain-openai>=1,<2"
python -m pip checkKeep SCRAPELESS_API_KEY set and add your OpenAI credentials.
macOS / Linux:
export OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
export OPENAI_MODEL="YOUR_TOOL_CALLING_MODEL"Windows PowerShell:
$env:OPENAI_API_KEY = "YOUR_OPENAI_API_KEY"
$env:OPENAI_MODEL = "YOUR_TOOL_CALLING_MODEL"Replace the model placeholder with a model ID available to your account that supports tool calling. OPENAI_MODEL is a variable used explicitly by this example.
Create and run the agent
Save as agent.py:
import os
from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
from langchain_scrapeless import ScrapelessDeepSerpGoogleSearchTool
model = ChatOpenAI(model=os.environ["OPENAI_MODEL"])
search_tool = ScrapelessDeepSerpGoogleSearchTool(handle_tool_error=False)
agent = create_agent(model=model, tools=[search_tool])
result = agent.invoke({
"messages": [{
"role": "user",
"content": (
"Search the web for Scrapeless documentation and return three "
"relevant documentation links with a short description of each."
),
}],
})
# Show the model's tool calls, tool responses, and final answer.
for message in result["messages"]:
message.pretty_print()Run python agent.py. In the printed messages, look for a call to scrapeless_deepserp_google_search, its tool response, and a final answer based on those results. Inspect the response for usable data; a model answer alone does not confirm that Scrapeless was called.
Registering a tool makes it available but does not force a call on every request. To add another capability, create the corresponding tool instance and include it in tools=[...].
More product examples
These examples are independent. Save each block as a .py file and run python filename.py in the configured environment.
Crawl
Use ScrapelessCrawlerScrapeTool for known URLs. Supply one URL or a batch in urls:
from langchain_scrapeless import ScrapelessCrawlerScrapeTool
tool = ScrapelessCrawlerScrapeTool(handle_tool_error=False)
result = tool.invoke({
"urls": ["https://example.com"],
"formats": ["markdown"],
"only_main_content": True,
"poll_timeout": 180,
})
if result.get("status") != "completed":
raise RuntimeError(f"Scrape did not complete: {result.get('status')}")
pages = result.get("data") or []
print("Returned pages:", len(pages))
for page in pages:
metadata = page.get("metadata") or {}
print(metadata.get("sourceURL"), metadata.get("statusCode"))
print(page.get("markdown", ""))The call waits for the batch task. Check each page’s status and content as well as the task status. poll_timeout is measured in seconds; the separate page timeout parameter is measured in milliseconds.
To discover linked pages instead, use ScrapelessCrawlerCrawlTool with {"url": "https://example.com", "limit": 5, "formats": ["markdown"]}. It also waits for completion and returns a dictionary with status and data. limit is a maximum, so fewer pages may be returned. This tool does not expose poll_timeout in version 0.1.4.
Web Unlocker
Request Markdown with JavaScript rendering enabled:
from pathlib import Path
from langchain_scrapeless import ScrapelessUniversalScrapingTool
tool = ScrapelessUniversalScrapingTool(handle_tool_error=False)
content = tool.invoke({
"url": "https://example.com",
"js_render": True,
"response_type": "markdown",
})
if not isinstance(content, str) or not content.strip():
raise RuntimeError("Expected non-empty Markdown text; inspect the response")
Path("page.md").write_text(content, encoding="utf-8")
print("Saved page.md")Open page.md to check the extracted body. Text formats include markdown, html, and plaintext. Use returned text directly; it does not have Crawl’s status and data fields. Keep the input URL with the content as its source.
AI Scraper
Use an engine-specific tool or select an engine through ScrapelessAIScraperTool. Supported engine values are chatgpt, perplexity, copilot, gemini, aimode, overview, grok, and alexa.
This example uses the ChatGPT engine within AI Scraper:
from langchain_scrapeless import ScrapelessChatGPTScraperTool
tool = ScrapelessChatGPTScraperTool(handle_tool_error=False)
task = tool.invoke({
"prompt": "Explain HTTP caching and include sources where available.",
"country": "US",
"web_search": True,
"shopping": False,
"output_format": "json",
})
if task.get("status") != "success":
raise RuntimeError(f"Task did not complete: {task.get('task_id')}")
answer = task["task_result"]
print(answer.get("result_text", ""))
print("Links:", answer.get("links") or [])
print("Content references:", answer.get("content_references") or [])The ChatGPT tool returns text by default; output_format="json" returns a Python dictionary containing task status and result fields. The unified tool and other engine-specific tools default to a full task dictionary. Inputs and answer fields vary by engine; for example, Copilot requires mode. Sources may be empty, and Google AI Overview may complete without an overview.
This call requires only SCRAPELESS_API_KEY. AI Scraper collects platform answers; the ChatOpenAI model in the agent example has a separate role and key.
Task handling
Most examples above wait for results. For longer workflows:
-
AI Scraper: set
wait=Falseandoutput_format="json"in the engine input, savetask_id, and retrieve it withScrapelessAIScraperResultTool. Pass the sameengine. Lookup supportswait,poll_timeout, andpoll_interval; the latter two use seconds. -
Batch scrape: if polling times out, keep the task ID in the error. Use
client.scraping_crawl.scrape.check_batch_scrape_status(task_id)through the Python SDK to query the existing batch. -
Website crawl: use the SDK’s
async_crawl_url()to submit andcheck_crawl_status()to retrieve the saved task. The status lookup returns an object with attributes such asstatusanddata.
These are remote-task workflows, separate from Python async/await. A polling timeout does not cancel the remote task. Query the existing ID before submitting again. See the Python SDK guide and package examples for detailed usage.
Troubleshooting
| Problem | What to check |
|---|---|
| Import or dependency error | Use Python 3.10+ and the same virtual environment for installation and execution. Run python -m pip check and python -m pip show langchain-scrapeless scrapeless langchain-core. |
| Missing key or authentication failure | Set the key in the process running Python. Check validity, product access, and credits. The agent’s model-provider key is separate. |
| Parameter validation error | Check the selected tool and engine schema. For the unified AI Scraper tool, put engine-specific fields inside input. SDK, REST, and tool parameter names can differ. |
| Timeout | Preserve the task ID and follow the lookup guidance above. Re-running submission may create another task. |
| Empty or unexpected output | Inspect the full response, task state, and requested content fields. For Web Unlocker text-decoding errors, include the SDK version and sanitized traceback when reporting the issue. |
| Agent does not search | Verify the direct quickstart first. Check tool registration and model tool-calling support, then inspect the execution messages with an explicit search request. |