Help regarding urllib.urlopen

Help regarding urllib.urlopen

Postby vbmade2000 » Sat Aug 24, 2013 11:05 am

Hello All,

I have following code that simply fetches html data from a robots.txt of a given url using urllib.urlopen function.

Here is my code.

Code: Select all
siteurl = siteurl.rstrip("/")
roboturl = siteurl + r'/robots.txt'   
robotdata = urllib.urlopen(roboturl).read()
print robotdata


When I run above code. It throws following exception.
Code: Select all
This is main function
Main URL : http://www.bestrecipes.com.au
$$$$$$$$$$:www.bestrecipes.com.au
###########-->www.bestrecipes.com.au/robots.txt
Traceback (most recent call last):
  File "dataintegrator.py", line 104, in <module>
    main()
  File "dataintegrator.py", line 81, in main
    print "Sitemap Url : " + getSiteMapUrl(i)
  File "D:\Malhar Data\Projects\Data Parsing\My Code\Final Part\libs\datareader.
py", line 50, in getSiteMapUrl
    robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given ur
l
  File "C:\Python26\lib\urllib.py", line 87, in urlopen
    return opener.open(url)
  File "C:\Python26\lib\urllib.py", line 203, in open
    return getattr(self, name)(url)
  File "C:\Python26\lib\urllib.py", line 461, in open_file
    return self.open_local_file(url)
  File "C:\Python26\lib\urllib.py", line 475, in open_local_file
    raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c
om.au\\robots.txt'



Here the path "www.bestrecipes.com.au/robots.txt" exists actually but it replaces "/" with "\\" before robots.txt.
I am Python newbie so not able to understand this problem. Please help me regarding this.


Thank you

Malhar Vora
Last edited by Yoriz on Sat Aug 24, 2013 1:10 pm, edited 1 time in total.
Reason: Added code tags & locked
vbmade2000
 
Posts: 1
Joined: Sat Aug 24, 2013 10:56 am

Re: Help regarding urllib.urlopen

Postby Yoriz » Sat Aug 24, 2013 1:07 pm

Hi, welcome to the forum
Please read the new users read this link in my signature.
New Users, Read This
Join the #python-forum IRC channel on irc.freenode.net!
Spam topic disapproval technician
Windows7, Python 2.7.4., WxPython 2.9.5.0., some Python 3.3
User avatar
Yoriz
 
Posts: 833
Joined: Fri Feb 08, 2013 1:35 am
Location: UK


Return to Networking

Who is online

Users browsing this forum: No registered users and 1 guest