Need help for a web scraping script / spider programming

This is the place for queries that don't fit in any of the other categories.

Need help for a web scraping script / spider programming

Postby meta0wtf » Wed Jun 26, 2013 4:33 am

Hello everyone,

I am looking to create my first big project in Python. The idea for this is a script that interacts with an XLS spreadsheet that contains Magic the Gathering cards, and URLs to mtgprice.com to pull prices off their site and write them into my Spreadsheet. I am using xlrd, xlwt, and scrapy to do this. The problem comes with setting up the spider to pull the prices off the pages as I have not gotten this far yet, I haven't started the process of writing into the XLS spreadsheet. If anyone has any ideas on how to set up this spider, or another, easier way to do this I would greatly appreciate it.
meta0wtf
 
Posts: 2
Joined: Wed Jun 26, 2013 4:27 am

Re: Need help for a web scraping script / spider programming

Postby micseydel » Wed Jun 26, 2013 4:45 am

I would suggest you read section 3.3.4 here. That said, I use mechanize and lxml to do all my scraping, or urllib and urllib2 instead of mechanize when logging in is not required.

Also, if this question didn't have the issue of the terms and conditions, you'd still have to provide your entire code attempt before we would really help you with this.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1390
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Need help for a web scraping script / spider programming

Postby meta0wtf » Wed Jun 26, 2013 4:50 am

Thanks alot, I have a few other websites I could use if I find one that it is allowed I will just switch to that one. Also, I can post my code once I get everything written down and in. No point uploading my code if I still have errors. Thank you for pointing that out to me though :D
meta0wtf
 
Posts: 2
Joined: Wed Jun 26, 2013 4:27 am


Return to General Coding Help

Who is online

Users browsing this forum: Mvpo666, W3C [Linkcheck] and 3 guests