Python scraping help

This is the place for queries that don't fit in any of the other categories.

Python scraping help

Postby M1234 » Wed Jul 17, 2013 11:19 am

Hi,

I'm trying to use a data scraper to scrape off certain details from Metacritic (located here: https://classic.scraperwiki.com/scraper ... scraper_2/). However when I run it, it returns a -1 for publisher, rating, genre, score and release. I am under the assumption that the div classes have changed on the Metacritic site, but I am having trouble seeing what the classes have changed to and from.

Any help would be much appreciated.

Thanks
M1234
 
Posts: 5
Joined: Wed Jul 17, 2013 11:14 am

Re: Python scraping help

Postby Crimson King » Wed Jul 17, 2013 4:11 pm

M1234 wrote:Hi,

I'm trying to use a data scraper to scrape off certain details from Metacritic (located here: https://classic.scraperwiki.com/scraper ... scraper_2/). However when I run it, it returns a -1 for publisher, rating, genre, score and release. I am under the assumption that the div classes have changed on the Metacritic site, but I am having trouble seeing what the classes have changed to and from.

Any help would be much appreciated.

Thanks


If you're using Firefox you could install the Firebug extension to check the divs you mention.
User avatar
Crimson King
 
Posts: 89
Joined: Fri Mar 08, 2013 2:42 pm
Location: Buenos Aires, Argentina

Re: Python scraping help

Postby metulburr » Wed Jul 17, 2013 5:08 pm

There are quite a few different ones.
grab_html.png
grab_html.png (192.24 KiB) Viewed 665 times
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1455
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Python scraping help

Postby M1234 » Thu Jul 18, 2013 10:56 am

metulburr wrote:There are quite a few different ones.
grab_html.png


Sorry, can't really figure out what you're wanting to show?
M1234
 
Posts: 5
Joined: Wed Jul 17, 2013 11:14 am

Re: Python scraping help

Postby metulburr » Thu Jul 18, 2013 10:59 am

i was extending on crimson kings'
If you're using Firefox you could install the Firebug extension to check the divs you mention.

which is showing you the html of whatever your looking at.
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1455
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: Python scraping help

Postby M1234 » Fri Jul 19, 2013 10:26 am

metulburr wrote:i was extending on crimson kings'
If you're using Firefox you could install the Firebug extension to check the divs you mention.

which is showing you the html of whatever your looking at.


Chrome highlights the div classes when I roll over the data, it's just I'm having trouble understanding what to change in the original Python code.
M1234
 
Posts: 5
Joined: Wed Jul 17, 2013 11:14 am

Re: Python scraping help

Postby M1234 » Fri Jul 19, 2013 10:32 am

e.g. In the code it states

Code: Select all
product_score = product.xpath("div/div/div/div/div/div[@class='score_value']/div[@class='score_summary metascore_summary']/span[2]/text()")
        if len(product_score) != 1 or product_score[0] == 'tbd':
            data['score'] = -1
        else:
            data['score'] = int(product_score[0])


If we were to use this page as the example: http://www.metacritic.com/game/playstat ... last-of-us

I'm having trouble understanding what to replace in this part of the code in order for it to show the score.
M1234
 
Posts: 5
Joined: Wed Jul 17, 2013 11:14 am

Re: Python scraping help

Postby snippsat » Fri Jul 19, 2013 11:11 am

I'm having trouble understanding what to replace in this part of the code in order for it to show the score.

Code: Select all
print data['score']

Code save all info in a dictionary called data.
So if i do a quick demo.
Code: Select all
>>> data = {}
>>> product_score = ['95'] # The lxml parsed value from website
>>> data['score'] = int(product_score[0])
>>> print data
{'score': 95}
>>> print data['score']
95
User avatar
snippsat
 
Posts: 188
Joined: Thu Feb 21, 2013 12:04 am

Re: Python scraping help

Postby M1234 » Mon Aug 12, 2013 10:44 am

snippsat wrote:
I'm having trouble understanding what to replace in this part of the code in order for it to show the score.

Code: Select all
print data['score']

Code save all info in a dictionary called data.
So if i do a quick demo.
Code: Select all
>>> data = {}
>>> product_score = ['95'] # The lxml parsed value from website
>>> data['score'] = int(product_score[0])
>>> print data
{'score': 95}
>>> print data['score']
95


Tried running it but couldn't get it to work for some reason. Could you possibly post an edit of the full code so I could test it out? It would be much appreciated! Thank you.
M1234
 
Posts: 5
Joined: Wed Jul 17, 2013 11:14 am


Return to General Coding Help

Who is online

Users browsing this forum: No registered users and 5 guests