Simulating IE10 or under

Simulating IE10 or under

Postby tsutttra12 » Thu Jan 30, 2014 7:44 pm

Hello,
I've been doing web scraping for some time and recently learned on this very site how to properly pass authorization through a site by submitting a log-in form using Requests. It's my understanding that including headers is quite alike posting data for a log-in form. I have a website that i need to scrape data from and the only browser it'll work on is IE 10 or under(of all browsers). Using urllib to read a request returns an error as I expected. I figured i can post a header to trick it into thinking its IE10. Am i correct in thinking this? and if so how should the header keyword:value dictionary look?
Thanks in advanced for any help
tsutttra12
 
Posts: 8
Joined: Fri Jan 17, 2014 4:44 pm

Re: Simulating IE10 or under

Postby micseydel » Thu Jan 30, 2014 9:55 pm

I tend to use the third party mechanize module when I need to login, but whether you use it something else what you're looking to change is your user agent. Whatever module you decide to use, a Google search for it and "user agent" should turn up what you're looking for.
Due to the reasons discussed here we will be moving to python-forum.io on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.
User avatar
micseydel
 
Posts: 2988
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Simulating IE10 or under

Postby metulburr » Thu Jan 30, 2014 9:58 pm

how should the header keyword:value dictionary look?





Code: Select all
headers = {'User-agent': 'Mozilla/5.0',}
req = urllib2.Request(url, None, headers)
resp = urllib2.urlopen(req)


or

Code: Select all
req = urllib2.Request(url)
req.add_header('User-agent', 'Mozilla/5.0')
resp = urllib2.urlopen(req)
we will be moving to python-forum.io on October 1 2016
more details here
User avatar
metulburr
 
Posts: 2226
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY


Return to Networking

Who is online

Users browsing this forum: No registered users and 2 guests