Simulating IE10 or under

Simulating IE10 or under

Postby tsutttra12 » Thu Jan 30, 2014 7:44 pm

I've been doing web scraping for some time and recently learned on this very site how to properly pass authorization through a site by submitting a log-in form using Requests. It's my understanding that including headers is quite alike posting data for a log-in form. I have a website that i need to scrape data from and the only browser it'll work on is IE 10 or under(of all browsers). Using urllib to read a request returns an error as I expected. I figured i can post a header to trick it into thinking its IE10. Am i correct in thinking this? and if so how should the header keyword:value dictionary look?
Thanks in advanced for any help
Posts: 8
Joined: Fri Jan 17, 2014 4:44 pm

Re: Simulating IE10 or under

Postby micseydel » Thu Jan 30, 2014 9:55 pm

I tend to use the third party mechanize module when I need to login, but whether you use it something else what you're looking to change is your user agent. Whatever module you decide to use, a Google search for it and "user agent" should turn up what you're looking for.
Due to the reasons discussed here we will be moving to on October 1, 2016.

This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.
User avatar
Posts: 3000
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Simulating IE10 or under

Postby metulburr » Thu Jan 30, 2014 9:58 pm

how should the header keyword:value dictionary look?

Code: Select all
headers = {'User-agent': 'Mozilla/5.0',}
req = urllib2.Request(url, None, headers)
resp = urllib2.urlopen(req)


Code: Select all
req = urllib2.Request(url)
req.add_header('User-agent', 'Mozilla/5.0')
resp = urllib2.urlopen(req)
we will be moving to on October 1 2016
more details here
User avatar
Posts: 2242
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Return to Networking

Who is online

Users browsing this forum: No registered users and 0 guests