Crawler Python API¶
Getting started with Crawler is easy.
The main class you need to care about is Crawler
crawler.main¶
-
class
crawler.main.Crawler(url, delay, ignore)¶ Main Crawler object.
Example:
c = Crawler('http://example.com') c.crawl()
Parameters: - delay – Number of seconds to wait between searches
- ignore – Paths to ignore
-
crawl()¶ Crawl the URL set up in the crawler.
This is the main entry point, and will block while it runs.
-
get(url)¶ Get a specific URL, log its response, and return its content.
Parameters: url – The fully qualified URL to retrieve
-
crawler.main.run_main()¶ A small wrapper that is used for running as a CLI Script.
crawler.utils¶
-
utils.should_ignore(ignore_list, url)¶ Returns True if the URL should be ignored
Parameters: - ignore_list – The list of regexs to ignore.
- url – The fully qualified URL to compare against.
>>> should_ignore(['blog/$'], 'http://ericholscher.com/blog/')
True
# This test should fail
>>> should_ignore(['home'], 'http://ericholscher.com/blog/')
True
-
utils.log(url, status)¶ Log information about a response to the console.
Parameters: - url – The URL that was retrieved.
- status – A status code for the Response.
>>> log('http://ericholscher.com/blog/', 200)
OK: 200 http://ericholscher.com/blog/
>>> log('http://ericholscher.com/blog/', 500)
ERR: 500 http://ericholscher.com/blog/
# This test should fail
>>> log('http://ericholscher.com/blog/', 500)
OK: 500 http://ericholscher.com/blog/