Simulating IE10 or under

Simulating IE10 or under

Postby tsutttra12 » Thu Jan 30, 2014 7:44 pm

Hello,
I've been doing web scraping for some time and recently learned on this very site how to properly pass authorization through a site by submitting a log-in form using Requests. It's my understanding that including headers is quite alike posting data for a log-in form. I have a website that i need to scrape data from and the only browser it'll work on is IE 10 or under(of all browsers). Using urllib to read a request returns an error as I expected. I figured i can post a header to trick it into thinking its IE10. Am i correct in thinking this? and if so how should the header keyword:value dictionary look?
Thanks in advanced for any help
tsutttra12
 
Posts: 8
Joined: Fri Jan 17, 2014 4:44 pm

Re: Simulating IE10 or under

Postby micseydel » Thu Jan 30, 2014 9:55 pm

I tend to use the third party mechanize module when I need to login, but whether you use it something else what you're looking to change is your user agent. Whatever module you decide to use, a Google search for it and "user agent" should turn up what you're looking for.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1439
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Simulating IE10 or under

Postby metulburr » Thu Jan 30, 2014 9:58 pm

how should the header keyword:value dictionary look?





Code: Select all
headers = {'User-agent': 'Mozilla/5.0',}
req = urllib2.Request(url, None, headers)
resp = urllib2.urlopen(req)


or

Code: Select all
req = urllib2.Request(url)
req.add_header('User-agent', 'Mozilla/5.0')
resp = urllib2.urlopen(req)
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1506
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY


Return to Networking

Who is online

Users browsing this forum: No registered users and 0 guests