Help regarding urllib.urlopen

Help regarding urllib.urlopen

Postby vbmade2000 » Sat Aug 24, 2013 11:05 am

Hello All,

I have following code that simply fetches html data from a robots.txt of a given url using urllib.urlopen function.

Here is my code.

Code: Select all
siteurl = siteurl.rstrip("/")
roboturl = siteurl + r'/robots.txt'   
robotdata = urllib.urlopen(roboturl).read()
print robotdata

When I run above code. It throws following exception.
Code: Select all
This is main function
Main URL :
Traceback (most recent call last):
  File "", line 104, in <module>
  File "", line 81, in main
    print "Sitemap Url : " + getSiteMapUrl(i)
  File "D:\Malhar Data\Projects\Data Parsing\My Code\Final Part\libs\datareader.
py", line 50, in getSiteMapUrl
    robotdata = urllib.urlopen(roboturl).read() # Reading robots.txt of given ur
  File "C:\Python26\lib\", line 87, in urlopen
  File "C:\Python26\lib\", line 203, in open
    return getattr(self, name)(url)
  File "C:\Python26\lib\", line 461, in open_file
    return self.open_local_file(url)
  File "C:\Python26\lib\", line 475, in open_local_file
    raise IOError(e.errno, e.strerror, e.filename)
IOError: [Errno 2] The system cannot find the path specified: 'www.bestrecipes.c\\robots.txt'

Here the path "" exists actually but it replaces "/" with "\\" before robots.txt.
I am Python newbie so not able to understand this problem. Please help me regarding this.

Thank you

Malhar Vora
Last edited by Yoriz on Sat Aug 24, 2013 1:10 pm, edited 1 time in total.
Reason: Added code tags & locked
Posts: 1
Joined: Sat Aug 24, 2013 10:56 am

Re: Help regarding urllib.urlopen

Postby Yoriz » Sat Aug 24, 2013 1:07 pm

Hi, welcome to the forum
Please read the new users read this link in my signature.
Due to the reasons discussed here we will be moving to on October 1 2016
This forum will be locked down and no one will be able to post/edit/create threads, etc. here from thereafter. Please create an account at the new site to continue discussion.
User avatar
Posts: 1672
Joined: Fri Feb 08, 2013 1:35 am
Location: UK

Return to Networking

Who is online

Users browsing this forum: No registered users and 2 guests