installing python, eclipse/pydev and beautifulsoup

A forum for general discussion of the Python programming language.

installing python, eclipse/pydev and beautifulsoup

Postby igeek » Sat Mar 09, 2013 8:52 pm

hello,

I'm having a really have time getting the beautifulsoup module to work with python 2.7.3 in eclipse/pydev

I'm new to python so I'm not sure if this could be an installation error on my part..
maybe I dont have eclipse and pydev setup correctly?? I can upload a screenshot of my eclipse setup if that would help. thx!!

any advise would really be appreciated I've been spinning my wheels on this for a day now!!
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby metulburr » Sat Mar 09, 2013 10:51 pm

beautifulsoup has no reliance on an IDE (eclipse)

did you donwload it from here?
What OS are you on?

1) download
2) extract
3) go into directory extracted
4) execute code
Code: Select all
python setup.py install

(depending on OS code may be different)
5) if installation successful, test it in python interpreter
Code: Select all
>>>from bs4 import BeautifulSoup
>>>

if you get >>>, then it works
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1469
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Sun Mar 10, 2013 10:40 pm

Thanks for the help!

I'm using a Mac with Snow leopard OS 10.6.7 and yes that's where I downloaded BeautifulSoup from.

I got BeautifulSoup working with Python version 2.7.3.. thanks for the tip to use the python
shell to verify the installation that really helped!

I was trying to use easy install before but it wouldn't work for some reason.
Code: Select all
easy_install


I'm still getting an error with python 3.3 though.. any thoughts?
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby micseydel » Sun Mar 10, 2013 10:51 pm

"getting an error" is never enough information. If it's a Python traceback, post the full traceback verbatim in code tags, and otherwise post whatever the full error is.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1354
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Mon Mar 11, 2013 5:08 am

my apologies here it is.. I tried both bs4 and BeautifulSoup modules

Code: Select all
>>> from BeautifulSoup import BeautifulSoup
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    from BeautifulSoup import BeautifulSoup
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/BeautifulSoup.py", line 473
    raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__.__name__, attr)
                        ^
SyntaxError: invalid syntax
>>> from bs4 import BeautifulSoup
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    from bs4 import BeautifulSoup
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/bs4/__init__.py", line 359
    print soup.prettify()
             ^
SyntaxError: invalid syntax
>>>
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby micseydel » Mon Mar 11, 2013 5:14 am

It looks like you're using an old version of BeautifulSoup. Python 3 didn't gain support in BS until bs4.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1354
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Mon Mar 11, 2013 7:04 am

micseydel wrote:It looks like you're using an old version of BeautifulSoup. Python 3 didn't gain support in BS until bs4.


ok.. I thought the code
Code: Select all
from bs4 import BeautifulSoup

was importing the newest version of BS?

I downloaded and installed this one.. beautifulsoup4-4.1.3.tar.gz
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby micseydel » Mon Mar 11, 2013 7:38 am

Either you're using an old version, or the new code you got has a problem with it. If you're sure you have what you're supposed to, you should contact the BeautifulSoup people, let them know exactly what version you're using and give them the full traceback.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1354
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Mon Mar 11, 2013 7:59 am

micseydel wrote:Either you're using an old version, or the new code you got has a problem with it. If you're sure you have what you're supposed to, you should contact the BeautifulSoup people, let them know exactly what version you're using and give them the full traceback.


I'll try to install it again.. this sounds like a good option too!

Have you done any screen scraping with BeautifulSoup? I just stumbled on Scrapy.. I'm not sure if one is better then the other?
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby setrofim » Mon Mar 11, 2013 8:19 am

igeek wrote:I just stumbled on Scrapy.. I'm not sure if one is better then the other?

BeautifulSoup is for parsing html/xml. That's it. Scrapy is a more complete framework. E.g. it gives you a web crawler, allows you to automatically download images, etc. It's a lot more heavy-duty than BeautifulSoup and comes with more batteries. Because of that, it has a steeper learning curve (though not by that much) and may be more difficult to install (especially on Windows, where you'll need to get OpenSSL separately). The two actually serve quite different purposes (and IIRC, you could actually get Scrapy to use BeautifulSoup as its HTML parser).

Which you use depends on what you want to do. If you want to scrape some text from a well-known set of web pages, use BeautifulSoup. If you want to scrape entire websites (potentially with media content), use Scrapy.
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby micseydel » Mon Mar 11, 2013 8:22 am

I use lxml, Mechanize and Python 2 for my scraping. I tried Scrapy but couldn't figure it out. I've used BS before, but changed at some point for reasons I don't remember.
Join the #python-forum IRC channel on irc.freenode.net!

Please do not PM members regarding questions which are meant to be discussed publicly. The point of the forum is so that others can benefit from it. We don't want to help you over PMs or emails.
User avatar
micseydel
 
Posts: 1354
Joined: Tue Feb 12, 2013 2:18 am
Location: Mountain View, CA

Re: installing python, eclipse/pydev and beautifulsoup

Postby setrofim » Mon Mar 11, 2013 9:15 am

@OP:
Just this morning, someone on HN posted a link to a short web scraping in Python primer that you might find interesting. Also, check out HN comments for a good discussion of the relative merits of the various options available in Python (BeautifulSoup, lxml, mechanize, scrapy, pyquery, etc).
setrofim
 
Posts: 288
Joined: Mon Mar 04, 2013 7:52 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby metulburr » Mon Mar 11, 2013 10:30 am

Have you done any screen scraping with BeautifulSoup? I just stumbled on Scrapy.. I'm not sure if one is better then the other?

I have always used BeautifulSoup. I never learned any others as BeautifulSoup did everything i was interested in.
New Users, Read This
OS Ubuntu 14.04, Arch Linux, Gentoo, Windows 7/8
https://github.com/metulburr
steam
User avatar
metulburr
 
Posts: 1469
Joined: Thu Feb 07, 2013 4:47 pm
Location: Elmira, NY

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Mon Mar 11, 2013 6:21 pm

setrofim wrote:
igeek wrote:I just stumbled on Scrapy.. I'm not sure if one is better then the other?

BeautifulSoup is for parsing html/xml. That's it. Scrapy is a more complete framework. E.g. it gives you a web crawler, allows you to automatically download images, etc. It's a lot more heavy-duty than BeautifulSoup and comes with more batteries. Because of that, it has a steeper learning curve (though not by that much) and may be more difficult to install (especially on Windows, where you'll need to get OpenSSL separately). The two actually serve quite different purposes (and IIRC, you could actually get Scrapy to use BeautifulSoup as its HTML parser).

Which you use depends on what you want to do. If you want to scrape some text from a well-known set of web pages, use BeautifulSoup. If you want to scrape entire websites (potentially with media content), use Scrapy.


I need a program to scrape a huge list of parts (inventory id, price, quantity, description.. ect) on several different e-commerce web sites on different servers.

I need it to update databases on a main web site.. this needs to happen every day and I need to know how much data was imported.
The main web site is running using LAMP right now. I don't think I would need to download any images since I could just link to them via a URL.
I need to be able to parse through HTML and it needs to be fairly easy to maintain since we will need to tweak the code once in a while.

Thanks!!
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm

Re: installing python, eclipse/pydev and beautifulsoup

Postby igeek » Mon Mar 11, 2013 6:24 pm

setrofim wrote:@OP:
Just this morning, someone on HN posted a link to a short web scraping in Python primer that you might find interesting. Also, check out HN comments for a good discussion of the relative merits of the various options available in Python (BeautifulSoup, lxml, mechanize, scrapy, pyquery, etc).


I'm very interested. thank you!
igeek
 
Posts: 7
Joined: Sat Mar 09, 2013 8:44 pm


Return to General Discussions

Who is online

Users browsing this forum: No registered users and 2 guests

cron