Extract data from any website with our powerful, intelligent crawler. Custom selectors, real-time analysis, and comprehensive data extraction.
Everything you need to extract data from websites efficiently and reliably
Target specific elements with precise CSS selectors. Extract exactly the data you need from any webpage structure.
Optimized crawling engine with intelligent caching and parallel processing for maximum speed and efficiency.
Handle dynamic content, JavaScript rendering, and anti-bot measures with advanced bypass techniques.
Comprehensive metrics including load times, response codes, content analysis, and performance insights.
Export data in JSON, CSV, XML formats. Download or integrate directly with your applications via API.
Custom headers, user agents, timeouts, redirects, and proxy support for maximum flexibility.
Get started in just a few simple steps
Paste the website URL you want to crawl in the input field
Choose extraction options and add custom CSS selectors if needed
Click the crawl button and watch the magic happen in real-time
Download your extracted data in multiple formats or use our API
Test our powerful crawler with your own URLs
Extract data from any website with advanced options
Love our API? Star the project on GitHub, contribute to its development, report issues, or explore the source code. Join our growing community of developers!
Integrate our crawler API into your applications
$api_url = 'https://allwebcrawler.pro/api/crawler/crawl';
$data = [
'url' => 'https://example.com',
'selectors' => [
'title' => 'h1',
'description' => '.description'
],
'extractHeadings' => true,
'extractLinks' => true
];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $api_url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($data));
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json'
]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
$result = json_decode($response, true);
curl_close($ch);
const crawlData = async (url, selectors = {}) => {
const response = await fetch('/api/crawler/crawl', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: url,
selectors: selectors,
extractHeadings: true,
extractLinks: true,
extractImages: true,
timeout: 30000
})
});
const data = await response.json();
return data;
};
// Usage
crawlData('https://example.com', {
title: 'h1',
price: '.price'
}).then(result => {
console.log(result);
});
import requests
import json
def crawl_website(url, selectors=None):
api_url = "https://allwebcrawler.pro/api/crawler/crawl"
payload = {
"url": url,
"selectors": selectors or {},
"extractHeadings": True,
"extractLinks": True,
"extractImages": True,
"timeout": 30000
}
headers = {
"Content-Type": "application/json"
}
response = requests.post(
api_url,
data=json.dumps(payload),
headers=headers
)
return response.json()
# Usage
result = crawl_website(
"https://example.com",
{"title": "h1", "description": ".description"}
)
print(json.dumps(result, indent=2))
curl -X POST https://allwebcrawler.pro/api/crawler/crawl \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"selectors": {
"title": "h1",
"description": ".description",
"price": ".price"
},
"extractHeadings": true,
"extractLinks": true,
"extractImages": true,
"timeout": 30000
}'
Everything you need to know about our crawler