Python for web automation

This is the place for queries that don't fit in any of the other categories.

Python for web automation

Postby Flasharmstro » Tue Jul 30, 2013 1:55 am

I'm trying to make a simple script in python that will scan a tweet for a link and then visit that link.
I'm having trouble determining which direction to go from here. From what I've researched it seems that I can Use Selenium or Mechanize? Which can be used for browser automation. Would using these be considered web scraping?


I can learn one of the twitter apis , the Requests library, and pyjamas(converts python code to javascript) so I can make a simple script and load it into google chrome's/firefox extensions.

Which would be the better option to take?
Posts: 1
Joined: Tue Jul 30, 2013 1:50 am

Re: Python for web automation

Postby micseydel » Tue Jul 30, 2013 4:19 am

Don't scrape anything there's an API for, unless you have very good reason to and don't need your script to work more than once. Keep in mind also that many services prohibit scraping. That said, I use mechanize and lxml for all my scraping and love it (still haven't hit up scrapy tutorial we have here that IIRC setrofim wrote).

For your Twitter thing, you'd use their API and then simple string manipulation though. As far as a browser extension goes... I don't know about that, although maybe someone else here will.
Due to the reasons discussed here we will be moving to on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.
User avatar
Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Return to General Coding Help

Who is online

Users browsing this forum: No registered users and 7 guests