Skip to main content

crawler_services

Function crawler_services 

Source
pub(crate) fn crawler_services<S>(
    address_book_service: Buffer<BoxService<AddressBookRequest, AddressBookResponse, BoxError>, AddressBookRequest>,
    peer_service: S,
) -> (RateLimitOnYield<Buffer<BoxService<AddressBookRequest, AddressBookResponse, BoxError>, AddressBookRequest>, AddressBookResponse>, RateLimitBySkipping<CrawlFanout<S>>)
where S: Service<Request, Response = Response, Error = BoxError> + Clone + Send + 'static, S::Future: Send + 'static,
Expand description

Builds the candidate selection services used by the crawler: a NextPeerService and a CrawlService.

Uses address_book_service to choose candidates and store crawled addresses, and peer_service to crawl the network for more addresses.