Can't login with urllib/urllib2

Can't login with urllib/urllib2

Postby davecotefilm » Sat Nov 30, 2013 9:55 am

I'm trying to log into a site with my credentials, using cookies I assume, though I'm new to using proper web id's and posting. I'm simply trying to log in to http://www.zulutrade.com
I am building a script that will feed and analyze my own trade info, but need to be able to login and have the cookie so the site will remember my python script for the duration it's running. Similar to a browser experience, but I want to do it with urllib/urllib2.

Here's my code:
Code: Select all
import cookielib, urllib2, urllib
class Acc:
    jar = cookielib.CookieJar()
    cookie = urllib2.HTTPCookieProcessor(jar)       
    opener = urllib2.build_opener(cookie)

    headers = {
        "User-Agent" : "Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.14) Gecko/20080609 Firefox/2.0.0.14",
        "Accept" : "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,text/png,*/*;q=0.5",
        "Accept-Language" : "en-us,en;q=0.5",
        "Accept-Charset" : "ISO-8859-1",
        "Content-type": "application/x-www-form-urlencoded",
        "Host": "classic.zulutrade.com"
    }

    def login(self):
        try:
            params = urllib.urlencode({'ctl00_z_TextBoxUsername':'myUsername','ctl00_z_TextBoxPassword':'myPassword','ctl00_z_ButtonLogin':'Login'})
            req = urllib2.Request('http://classic.zulutrade.com/?m=m&refsrc=classic.zulutrade.com%2F', params, self.headers)
            res = self.opener.open(req)
            html = res.read()

            file=open('outputData.txt','wb')
            file.write(html)
            file.close()

            #print res.getheader('location').split('/')[3]

        except urllib2.HTTPError, e:
            print e.msg
        except urllib2.URLError, e:
            print e.reason[1]
        return False

    def fetch(self,url):
        req = urllib2.Request(url,None,self.headers)
        res = self.opener.open(req)
        return res.read()

bla = Acc()
bla.login()


As you can see with 'outputData.txt' that is created, it simply prints out the html of the login page again. Yes I have emitted my username and password for obvious reasons, but if the login was done correctly, the outputData.txt would show html for 'incorrect username/password'...
What am I doing wrong here?? I'm trying to capture the cookie as well, which I think I'm doing right.
Please help:(
David
Last edited by Mekire on Sat Nov 30, 2013 10:20 am, edited 1 time in total.
Reason: First post lock.
davecotefilm
 
Posts: 4
Joined: Sat Nov 30, 2013 9:40 am

Re: Can't login with urllib/urllib2

Postby micseydel » Sat Nov 30, 2013 10:31 am

If you're able to, have you tried using mechanize? It handles all the cookie stuff for you.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1497
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: Can't login with urllib/urllib2

Postby davecotefilm » Sat Nov 30, 2013 11:11 am

Hi, I need to have my python script to do other things and run automatically. I don't think mechanize can do that.
davecotefilm
 
Posts: 4
Joined: Sat Nov 30, 2013 9:40 am

Re: Can't login with urllib/urllib2

Postby micseydel » Sat Nov 30, 2013 8:46 pm

Mechanize is just a wrapper for the modules you're trying to use, how does one have anything to do with making it automated or not?
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1497
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Can't login with Python Mechanize

Postby davecotefilm » Sun Dec 01, 2013 8:27 am

Hi, I was advised to use Mechanize Module to login to a website via python script, though I originally wanted to use urllib/urllib2. Now, I've read the docs for Mechanize and it just isn't working. I have cookie support added to my script, and it seems Mechanize just won't do the job. Can someone take a look at my code and tell me if I'm missing something? The OUTPUT my script creates (MechanizeDat.html) should show the page once I'm logged in, which should include my username and account etc, ... though it currently just shows the login screen all over again. For obvious reasons, I'm emitting my password/username from the script, but you will get the idea (you should get a 'wrong username/passowrd combination' but you will see you just get the log in screen):

Code: Select all
import mechanize
import cookielib
import urllib
import logging
import sys

def main():

    br = mechanize.Browser()
    cj = cookielib.LWPCookieJar()
    br.set_cookiejar(cj)

    br.set_handle_equiv(True)
    br.set_handle_gzip(True)
    br.set_handle_redirect(True)
    br.set_handle_referer(True)
    br.set_handle_robots(False)

    br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

    r= br.open('http://www.zulutrade.com/login')

    # Select the second (index one) form
    br.select_form(nr=0)

    #for control in br.form.controls:
    #  print control
     
    # User credentials
    br.form['ctl00$main$tbUsername'] = 'MYUSERNAME'
    br.form['ctl00$main$tbPassword'] = 'MYPASSWORD'

    #br.form['3'] = 'Sign In'

    # Login
    br.submit()

    # Open up comment page
    posting = 'https://www.zulutrade.com/user#dashboard'
    #rval = 'PoopSandwiches'
    # you can get the rval in other ways, but this will work for testing

    r = br.open(posting)

    # You need the 'uh' value from the first form
    #br.select_form(nr=0)
    #uh = br.form['uh']

    #br.select_form(nr=7)
    #thing_id = br.form['thing_id']
    #id = '#' + br.form.attrs['id']
    # The id that gets posted is the form id with a '#' prepended.

    #data = {'uh':uh, 'thing_id':thing_id, 'id':id, 'renderstyle':'html', 'r':rval, 'text':"Your text here!"}
    #new_data_dict = dict((k, urllib.quote(v).replace('%20', '+')) for k, v in data.iteritems())

    # not sure if the replace needs to happen, I did it anyway
    #new_data = 'thing_id=%(thing_id)s&text=%(text)s&id=%(id)s&r=%(r)s&uh=%(uh)s&renderstyle=%(renderstyle)s' %(new_data_dict)

    # not sure which of these headers are really needed, but it works with all
    # of them, so why not just include them.
    req = mechanize.Request('https://www.zulutrade.com/user#dashboard', None) #None was new_data
    req.add_header('Referer', posting)
    req.add_header('Accept', ' application/json, text/javascript, */*')
    req.add_header('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')
    req.add_header('X-Requested-With', 'XMLHttpRequest')
    cj.add_cookie_header(req)
    res = mechanize.urlopen(req)
    html=res.read()
    file=open('MechanizeDat.html','wb')
    file.write(html)
    file.close()

main()


Please help, as this is getting very frustrating (working on it a week now..),
Thank you so much and sorry I'm not smart:(
David
Last edited by micseydel on Sun Dec 01, 2013 9:51 pm, edited 1 time in total.
Reason: Merged from separate thread.
davecotefilm
 
Posts: 4
Joined: Sat Nov 30, 2013 9:40 am

Re: Can't login with urllib/urllib2

Postby davecotefilm » Sun Dec 01, 2013 2:03 pm

I have researched Mechanize, sorry I thought it was a python-run browser, I'm a noob. Anyways, I've tried it out and it's not working, in the same way urllib/urllib2 wasn't working.. can you take a look at my code? I've emitted the password/username but if it works the output file should show a page that states 'wrong username/password' as opposed to just bringing up the login page again. I've tried all id and name tags I could find in the source of the zulutrade page, it's just not working.
Code: Select all
import mechanize
import cookielib
import urllib
import logging
import sys

def main():

    br = mechanize.Browser()
    cj = cookielib.LWPCookieJar()
    br.set_cookiejar(cj)

    br.set_handle_equiv(True)
    br.set_handle_gzip(True)
    br.set_handle_redirect(True)
    br.set_handle_referer(True)
    br.set_handle_robots(False)

    br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

    r= br.open('http://www.zulutrade.com/login')

    # Select the second (index one) form
    br.select_form(nr=0)

    #for control in br.form.controls:
    #  print control
     
    # User credentials
    br.form['ctl00$main$tbUsername'] = 'MYUSERNAME'
    br.form['ctl00$main$tbPassword'] = 'MYPASSWORD'

    #br.form['3'] = 'Sign In'

    # Login
    br.submit()

    # Open up comment page
    posting = 'https://www.zulutrade.com/user#dashboard'
    #rval = 'PoopSandwiches'
    # you can get the rval in other ways, but this will work for testing

    r = br.open(posting)

    # You need the 'uh' value from the first form
    #br.select_form(nr=0)
    #uh = br.form['uh']

    #br.select_form(nr=7)
    #thing_id = br.form['thing_id']
    #id = '#' + br.form.attrs['id']
    # The id that gets posted is the form id with a '#' prepended.

    #data = {'uh':uh, 'thing_id':thing_id, 'id':id, 'renderstyle':'html', 'r':rval, 'text':"Your text here!"}
    #new_data_dict = dict((k, urllib.quote(v).replace('%20', '+')) for k, v in data.iteritems())

    # not sure if the replace needs to happen, I did it anyway
    #new_data = 'thing_id=%(thing_id)s&text=%(text)s&id=%(id)s&r=%(r)s&uh=%(uh)s&renderstyle=%(renderstyle)s' %(new_data_dict)

    # not sure which of these headers are really needed, but it works with all
    # of them, so why not just include them.
    req = mechanize.Request('https://www.zulutrade.com/user#dashboard', None) #None was new_data
    req.add_header('Referer', posting)
    req.add_header('Accept', ' application/json, text/javascript, */*')
    req.add_header('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8')
    req.add_header('X-Requested-With', 'XMLHttpRequest')
    cj.add_cookie_header(req)
    res = mechanize.urlopen(req)
    html=res.read()
    file=open('MechanizeDat.html','wb')
    file.write(html)
    file.close()

main()


Please help, I assure you I did my research, learned to implement the code, I'm not looking for a handout really I just can't figure it out:(
David
davecotefilm
 
Posts: 4
Joined: Sat Nov 30, 2013 9:40 am


Return to Networking

Who is online

Users browsing this forum: No registered users and 2 guests