classes

ScrapeResult

Bases: BaseModel

success property

success: bool

Indicates whether the scraping operation was successful.

Returns:

Name	Type	Description
bool	bool	True if scraping succeeded; otherwise False.

error property

error: str

Provides the error message if the scraping operation failed. May also contain debug information for successful scrapes.

Returns:

Name	Type	Description
str	str	The error message when scraping fails, or None if successful.

succeed staticmethod

succeed(link: str, content: str, metadata: dict = None) -> ScrapeResult

Creates a ScrapeResult instance for a successful scraping operation.

Parameters:

Name	Type	Description	Default
link	str	The URL that was scraped.	required
content	str	The content fetched from the link.	required
metadata	dict	Additional data scraped from the link.	None

Returns:

Name	Type	Description
ScrapeResult	ScrapeResult	An instance with scrape_success set to True.

fail staticmethod

fail(link: str, error: str) -> ScrapeResult

Creates a ScrapeResult instance for a failed scraping operation.

Parameters:

Name	Type	Description	Default
link	str	The URL attempted.	required
error	str	The error message describing the failure.	required

Returns:

Name	Type	Description
ScrapeResult	ScrapeResult	An instance with scrape_success set to False.

AnalysisResult

Bases: BaseModel

success property

success: bool

Indicates whether the analysis operation was successful.

Returns:

Name	Type	Description
bool	bool	True if analysis succeeded; otherwise False.

error property

error: str

Provides the error message if the analysis operation failed.

Returns:

Name	Type	Description
str	str	Provides the error message when analysis fails. May also contain debug information for successful analyses.

succeed staticmethod

succeed(output: dict) -> AnalysisResult

Creates an AnalysisResult instance for a successful analysis operation.

Parameters:

Name	Type	Description	Default
output	dict	The extracted analysis data.	required

Returns:

Name	Type	Description
AnalysisResult	AnalysisResult	An instance with analysis_success set to True.

fail staticmethod

fail(error: str) -> AnalysisResult

Creates an AnalysisResult instance for a failed analysis operation.

Parameters:

Name	Type	Description	Default
error	str	The error message detailing the failure.	required

Returns:

Name	Type	Description
AnalysisResult	AnalysisResult	An instance with analysis_success set to False.

IScraper

Bases: ABC

scrape abstractmethod

scrape(link: str) -> ScrapeResult

Fetches content from the specified URL.

Parameters:

Name	Type	Description	Default
link	str	The URL to scrape.	required

Returns:

Name	Type	Description
ScrapeResult	ScrapeResult	The result of the scraping operation.

scrape_multiple

scrape_multiple(links: Iterable[str]) -> Generator[Tuple[str, ScrapeResult], None, None]

Get content from multiple urls.

get_expected_link_format

get_expected_link_format() -> str | Pattern

Returns the expected regex format for links compatible with the scraper. Returning None indicates compatibility with any link format.

Returns:

Name	Type	Description
str	str | Pattern	The expected link format as a regex string or compiled pattern.

IAnalyzer

Bases: ABC

analyze abstractmethod

analyze(content: str) -> AnalysisResult

Analyzes the provided content to extract structured information.

Parameters:

Name	Type	Description	Default
content	str	The text content to analyze.	required

Returns:

Name	Type	Description
AnalysisResult	AnalysisResult	The result containing analysis output or error details.

analyze_multiple

analyze_multiple(contents: Dict[str, str]) -> Generator[Tuple[str, AnalysisResult], None, None]

Analyze multiple contents.

ILinkCollector

Bases: ABC, Iterable[str]

collect_links abstractmethod

collect_links() -> Iterable[str]

Fetches a list of links to scrape.

Returns:

Type	Description
Iterable[str]	Iterable[str]: An iterable of URLs to scrape.