Simulating IE10 or under

Simulating IE10 or under

Postby tsutttra12 » Thu Jan 30, 2014 7:44 pm

Hello,
I've been doing web scraping for some time and recently learned on this very site how to properly pass authorization through a site by submitting a log-in form using Requests. It's my understanding that including headers is quite alike posting data for a log-in form. I have a website that i need to scrape data from and the only browser it'll work on is IE 10 or under(of all browsers). Using urllib to read a request returns an error as I expected. I figured i can post a header to trick it into thinking its IE10. Am i correct in thinking this? and if so how should the header keyword:value dictionary look?
Thanks in advanced for any help
tsutttra12
 
Posts: 8
Joined: Fri Jan 17, 2014 4:44 pm

Re: Simulating IE10 or under

Postby micseydel » Thu Jan 30, 2014 9:55 pm

I tend to use the third party mechanize module when I need to login, but whether you use it something else what you're looking to change is your user agent. Whatever module you decide to use, a Google search for it and "user agent" should turn up what you're looking for.
Join the #python-forum IRC channel on irc.freenode.net!
User avatar
micseydel
 
Posts: 1113
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Simulating IE10 or under

Postby metulburr » Thu Jan 30, 2014 9:58 pm

how should the header keyword:value dictionary look?





Code: Select all
headers = {'User-agent': 'Mozilla/5.0',}
req = urllib2.Request(url, None, headers)
resp = urllib2.urlopen(req)


or

Code: Select all
req = urllib2.Request(url)
req.add_header('User-agent', 'Mozilla/5.0')
resp = urllib2.urlopen(req)
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1299
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY


Return to Networking

Who is online

Users browsing this forum: No registered users and 3 guests