installing python, eclipse/pydev and beautifulsoup

[igeek]

hello,

I'm having a really have time getting the beautifulsoup module to work with python 2.7.3 in eclipse/pydev

I'm new to python so I'm not sure if this could be an installation error on my part..
maybe I dont have eclipse and pydev setup correctly?? I can upload a screenshot of my eclipse setup if that would help. thx!!

any advise would really be appreciated I've been spinning my wheels on this for a day now!!

[metulburr]

beautifulsoup has no reliance on an IDE (eclipse)

did you donwload it from here?
What OS are you on?

1) download
2) extract
3) go into directory extracted
4) execute code

Code: Select all

python setup.py install

(depending on OS code may be different)
5) if installation successful, test it in python interpreter

Code: Select all

>>>from bs4 import BeautifulSoup
>>>

if you get >>>, then it works

[igeek]

Thanks for the help!

I'm using a Mac with Snow leopard OS 10.6.7 and yes that's where I downloaded BeautifulSoup from.

I got BeautifulSoup working with Python version 2.7.3.. thanks for the tip to use the python
shell to verify the installation that really helped!

I was trying to use easy install before but it wouldn't work for some reason.

Code: Select all

easy_install

I'm still getting an error with python 3.3 though.. any thoughts?

[micseydel]

"getting an error" is never enough information. If it's a Python traceback, post the full traceback verbatim in code tags, and otherwise post whatever the full error is.

[igeek]

my apologies here it is.. I tried both bs4 and BeautifulSoup modules

Code: Select all

>>> from BeautifulSoup import BeautifulSoup
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    from BeautifulSoup import BeautifulSoup
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/BeautifulSoup.py", line 473
    raise AttributeError, "'%s' object has no attribute '%s'" % (self.__class__.__name__, attr)
                        ^
SyntaxError: invalid syntax
>>> from bs4 import BeautifulSoup
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    from bs4 import BeautifulSoup
  File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/bs4/__init__.py", line 359
    print soup.prettify()
             ^
SyntaxError: invalid syntax
>>>

[micseydel]

It looks like you're using an old version of BeautifulSoup. Python 3 didn't gain support in BS until bs4.

[igeek]

It looks like you're using an old version of BeautifulSoup. Python 3 didn't gain support in BS until bs4.

ok.. I thought the code

Code: Select all

from bs4 import BeautifulSoup

was importing the newest version of BS?

I downloaded and installed this one.. beautifulsoup4-4.1.3.tar.gz

[micseydel]

Either you're using an old version, or the new code you got has a problem with it. If you're sure you have what you're supposed to, you should contact the BeautifulSoup people, let them know exactly what version you're using and give them the full traceback.

[igeek]

Either you're using an old version, or the new code you got has a problem with it. If you're sure you have what you're supposed to, you should contact the BeautifulSoup people, let them know exactly what version you're using and give them the full traceback.

I'll try to install it again.. this sounds like a good option too!

Have you done any screen scraping with BeautifulSoup? I just stumbled on Scrapy.. I'm not sure if one is better then the other?

[setrofim]

I just stumbled on Scrapy.. I'm not sure if one is better then the other?

BeautifulSoup is for parsing html/xml. That's it. Scrapy is a more complete framework. E.g. it gives you a web crawler, allows you to automatically download images, etc. It's a lot more heavy-duty than BeautifulSoup and comes with more batteries. Because of that, it has a steeper learning curve (though not by that much) and may be more difficult to install (especially on Windows, where you'll need to get OpenSSL separately). The two actually serve quite different purposes (and IIRC, you could actually get Scrapy to use BeautifulSoup as its HTML parser).

Which you use depends on what you want to do. If you want to scrape some text from a well-known set of web pages, use BeautifulSoup. If you want to scrape entire websites (potentially with media content), use Scrapy.

[micseydel]

I use lxml, Mechanize and Python 2 for my scraping. I tried Scrapy but couldn't figure it out. I've used BS before, but changed at some point for reasons I don't remember.

[setrofim]

@OP:
Just this morning, someone on HN posted a link to a short web scraping in Python primer that you might find interesting. Also, check out HN comments for a good discussion of the relative merits of the various options available in Python (BeautifulSoup, lxml, mechanize, scrapy, pyquery, etc).

[metulburr]

Have you done any screen scraping with BeautifulSoup? I just stumbled on Scrapy.. I'm not sure if one is better then the other?

I have always used BeautifulSoup. I never learned any others as BeautifulSoup did everything i was interested in.

[igeek]

I just stumbled on Scrapy.. I'm not sure if one is better then the other?

BeautifulSoup is for parsing html/xml. That's it. Scrapy is a more complete framework. E.g. it gives you a web crawler, allows you to automatically download images, etc. It's a lot more heavy-duty than BeautifulSoup and comes with more batteries. Because of that, it has a steeper learning curve (though not by that much) and may be more difficult to install (especially on Windows, where you'll need to get OpenSSL separately). The two actually serve quite different purposes (and IIRC, you could actually get Scrapy to use BeautifulSoup as its HTML parser).

Which you use depends on what you want to do. If you want to scrape some text from a well-known set of web pages, use BeautifulSoup. If you want to scrape entire websites (potentially with media content), use Scrapy.

I need a program to scrape a huge list of parts (inventory id, price, quantity, description.. ect) on several different e-commerce web sites on different servers.

I need it to update databases on a main web site.. this needs to happen every day and I need to know how much data was imported.
The main web site is running using LAMP right now. I don't think I would need to download any images since I could just link to them via a URL.
I need to be able to parse through HTML and it needs to be fairly easy to maintain since we will need to tweak the code once in a while.

Thanks!!

[igeek]

@OP:
Just this morning, someone on HN posted a link to a short web scraping in Python primer that you might find interesting. Also, check out HN comments for a good discussion of the relative merits of the various options available in Python (BeautifulSoup, lxml, mechanize, scrapy, pyquery, etc).

I'm very interested. thank you!