metulburr wrote:what are the good points, bad points, and just "What exactly is" of selenium, scrapy, and mechanize?
- Selenium allows you to control a web browser from Python. With Selenium, you actually fire up a browser, such as Firefox, and issue commands to it. This is comparatively slow and clunky. Selenium was designed for, and is primarily used in, testing.
- mechanize emulates certain browser behaviours without relying on an external program. Basically, it is itself a light weight browser. It will not give you the full range of functionality of a real browser, but it is faster and much simpler to use.
- Scrapy is a web scraping framework. It also implements some browser behaviors and in that respect it is similar to mechanize. But it also comes with a bunch of stuff to help you scrape data from web sites, such as crawlers, classes for defining data structures to be extracted, etc. It is bigger and more complex than mechanize and comes with a steeper learning curve; but once you learn it, you can implement scrapers much faster as the framework gives you a lot of the stuff you would have to implement yourself with mechanize. Scrapy is also faster for scraping a large number of web pages.
need something like Selenium or pyv8