ScrapeGraphAI: Python web scraping library

ScrapeGraphAI is a web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents.

Verified

Core facts

Read by a script from the official API, not rewritten by a model.

License
MIT
Languages
Python
Maintainer
ScrapeGraphAI
Verified

What it is

ScrapeGraphAI is a web scraping Python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, Markdown, etc.). ScrapeGraphAI is licensed under the MIT License. ScrapeGraphAI is meant to be used for data exploration and research purposes only.

Main features

  1. The library has multiple standard scraping pipelines that can be used to extract information from a website (or local file).

  2. SmartScraperGraph extracts information from a single page given a user prompt and a source URL, and it is a single-page scraper that only needs a user prompt and an input source.

  3. Other pipelines can be used to extract information from multiple pages, generate Python scripts, or even generate audio files.

  4. SpeechGraph is a single-page scraper that extracts information from a website and generates an audio file, and ScriptCreatorGraph is a single-page scraper that extracts information from a website and generates a Python script.

  5. SearchGraph is a multi-page scraper that extracts information from the top n search results of a search engine; SmartScraperMultiGraph extracts information from multiple pages given a single prompt and a list of sources; ScriptCreatorMultiGraph generates a Python script for extracting information from multiple pages and sources.

  6. Each of these graph pipelines has a multi version, which allows calls of the LLM in parallel.

Getting started

  1. It is recommended to install the ScrapeGraphAI library in a virtual environment to avoid conflicts with other libraries.

  2. Different LLMs can be used through APIs such as OpenAI, Groq, Azure, Gemini, MiniMax and more, or local models using Ollama.

  3. To use local models, have Ollama installed and download the models using the ollama pull command.

From the README · bash
pip install scrapegraphai

# IMPORTANT (for fetching websites content)
playwright install
From the README · python
from scrapegraphai.graphs import SmartScraperGraph

# Define the configuration for the scraping pipeline
graph_config = {
    "llm": {
        "model": "ollama/llama3.2",
        "model_tokens": 8192,
        "format": "json",
    },
    "verbose": True,
    "headless": False,
}

# Create the SmartScraperGraph instance
smart_scraper_graph = SmartScraperGraph(
    prompt="Extract useful information from the webpage, including a description of what the company does, founders and social media links",
    source="https://scrapegraphai.com/",
    config=graph_config
)

# Run the pipeline
result = smart_scraper_graph.run()

import json
print(json.dumps(result, indent=4))

Who it is for

In the comparison of the open-source library and the managed API, the open-source library is a Python library run by the user, while the managed API is a hosted cloud service called via SDK. The open-source library runs on the user's own infrastructure (self-hosted), while the managed API runs on ScrapeGraphAI cloud. For the LLM, the open-source library uses one the user brings (OpenAI, Groq, Gemini, Azure, or local via Ollama), while the managed API's LLM is managed.

For browser and JS rendering, the open-source library is configured by the user (Playwright). Proxies and anti-bot handling are the user's responsibility in the open-source library and are included in the managed API. The capabilities of the open-source library are graph pipelines, while those of the managed API are Scrape, Extract, Search, Crawl, Monitor and History.

The cost model of the open-source library is LLM tokens plus the user's own infrastructure, while that of the managed API is pay-as-you-go credits. Scaling and maintenance are the user's responsibility in the open-source library and fully managed in the managed API. The open-source library is MIT licensed; the managed API's SDK is MIT and the API service is paid.

The project says to choose the open-source library for full control, on-prem/self-hosted data, local LLMs (Ollama), or fine-grained cost tuning, when the user is happy to manage browsers, proxies and scaling themselves. The project collects anonymous usage metrics to enhance package quality and user experience.

Sources

This page is written from the official sources below. Open them to check the original text.

Verified ·

This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.