Considering python for a major calculation, help needed!

A forum for general discussion of the Python programming language.

Considering python for a major calculation, help needed!

Postby mr.sxxg » Tue Jan 07, 2014 11:55 pm

Hello there!

I'm a physicist with a very hard problem at hand and I could really use an advice. My problem - think an ugly mix of classical and quantum physics with a lot of monte carlo looping and stiff, nonlinear ODE solving - by far the most complex task I was to do and had it planned to finish it by summer. My programming reach is very limited, indeed, I usually write monolithic C codes (MinGW, Netbeans, Windows), smuggling C++ features every now and then, spawning ugly console and common controls dialog applications that spit out hard to read .dat files. If you think of all the bad things real programmers advise us against, all the bad habits I can shot my leg with, all the crappy ways to handle pointer or allocation bunched - I call that my programming skill. And I'm boring you with this so you get to grasp my shock and awe with this python thing.

Shocked, I tell you! Looking at python examples and documentation, PTSD kicked in. I felt like my old-school-catholic OCD grandmother witnessing a loose Mardi Gras party. Classes touching with no }; in between them, variables that can be this or that, no compiler overlord to serve ... I've seen things, man! I CLI'd man scanf just to slap myself back to reality. :)

So I just need a help with a strategic decision. Do I continue with my old ways or should I join this travesty?! On C/C++ up side, there is a familiarity with the language and work environment, some experience with numerical problems and threading. On the down side, I really lose a *lot* of time figuring programming details rather than figuring physics and that annoys me. I will get program done faster in C than in py because I still have to learn the damn thing, and it's clear advantage for C. Also I have to have some graphics for the problem at hand and I'm just loving what I see with wxGlade and mathlibplot. What I had in mind for graphing is some lib to throw in or maybe just some BitBlting and handling WM_PAINT to draw variables and parameters and how they relate to each other. Now, this is no small thing in several ways - when you have thing written and debugged, there is a lot of test runs to explore configuration space and having vector field drawn with MayaVi2 and not just have a history graph of an average of a single vector component is a great comparative advantage for python. But my concern is - is python really easier to write than C/C++? Anyone, who has done 5000+ LOC projects in both of them - I'd really appreciate input from an experienced developer.

As for debugging phase, I'm loving the fact that python is interpreted, but it seems to me that modern IDEs cancel this comparative advantage for python. But if bug *production* is much lower for py than c, this could be up side for python. But really can't get a good insight googling - how would you people compare debugging on C/C++ against python?

Science: it is my impression that py community has done unbelievable effort so you can do good science with it. And I'm loving scipy and gsl interfaces and all, but it's all in vain if ...execution times. I read of it a lot, but still not sure. What of threading in py? When I write C, I can make damn sure to load all my CPU's cores mouthful. Can a skilled programmer do the same in py? What about tight loops? Montecarlo is all about loops and having Interpreter and loops share the sentence? Yikes! How reliable is this pypy thing? I've seen runtimes of py with pypy au pair with c coding. Like, 10-20% behind, but I can live with that, especially if I can make it up in development time and preserved nervs. On the other hand, does pypy have compatibility issues known? Specially against scipy and gsl interfacing.

Calculations are to be done overnight at work, eight cored Intel's beasts with 24GB of RAM and brutal performance bus - a beauty. Actually, I got half of dozen of those, but I don't thing I can manage across network parallel computing.

So I know this is just a pile of random thoughts and no central question asked, maybe bordering with forum rules. But I'd really appreciate if someone would reflect, especially if my skills are subset of yours. If total time, from zero to final numerical results, and efficiency of the time spent - meaning focus on physics rather that language details - is to be used as criteria, do I stand the chance to get my money back with py?

Grateful,
S.
mr.sxxg
 
Posts: 3
Joined: Tue Jan 07, 2014 9:57 pm

Re: Considering python for a major calculation, help needed!

Postby casevh » Wed Jan 08, 2014 5:58 am

Hi,

Interesting questions! I'll state upfront that I won't have an answer for you.

Instead of looking at the problem as a choice between Python or C, I would ask "Can I use Python and C together?" I think it is easier to write and debug Python code (this is a Python forum...) but C code is faster (I actually write more C than Python...). But all the C code that I write is called from Python.

Take the parts of your project that are relatively high-level and not CPU bound and write them in Python (GUI display, data management are candidates). Prototype your algorithms in Python. Use as many of the scientific libraries as possible (see below). If your algorithm doesn't mesh well with the libraries, then code them in C - either as standalone applications reading data files or as extensions called from Python. Separating the high-level logic from the core computation may make it easier to use multiple computers instead of just a single computer.

The Python ecosystem has many tools for the scientific programmer: numpy, scipy, Sage, sympy, cython, numba, matplotlib, ipython, and more. If your problem can use these tools, they may offer enough performance.

Regarding threading... Python's design makes the use of multiple cores via threading challenging. Python extensions written in C can use multiple cores (directly, or by releasing the internal synchronization lock (aka GIL) that Python uses). I would suggest looking at multiple processes instead. Using distinct processes will scale directly to using multiple computers. But if your algorithms are tightly coupled, you may lose too much time to I/O overhead.

One last bit of advice: there is an old adage "A Fortran programmer can write Fortran code in any language." Try not to write a C program in Python, and vice versa. Each language has advantages and programming styles that are best suited for each language.

Not sure if this helps or not...

casevh

Note: I have not used Cython or numba but I found an interesting article comparing them:

http://nbviewer.ipython.org/url/jakevdp ... thon.ipynb
casevh
 
Posts: 70
Joined: Sat Feb 09, 2013 7:35 am

Re: Considering python for a major calculation, help needed!

Postby Kebap » Wed Jan 08, 2014 10:14 am

mr.sxxg wrote:If you think of all the bad things real programmers advise us against, all the bad habits I can shot my leg with, all the crappy ways to handle pointer or allocation bunched - I call that my programming skill.

...

I really lose a *lot* of time figuring programming details rather than figuring physics and that annoys me.


These seem to be valid arguments for you to try and prototype your next code with python instead. One of the main advantages is the much increased usability and lowered time to get from ideas to working code. So much, some people say the reduced running time of said code can be ignored compared to those advantages. But you can of course still port a working prototype to C in the end, if you feel the need.

I can recommend the google python classes by Nick Parleante for a first intro. They are intended for people who already have some programing knowledge, but are also very low-key. They also cover how to test your ideas in the python interactive console directly, instead of cycling through massive help files. I find this extremely helpful and miss the feature in other languages.
Learn: How To Ask Questions The Smart Way
Join the #python-forum IRC channel on irc.freenode.net and chat with uns directly!
Kebap
 
Posts: 390
Joined: Thu Apr 04, 2013 1:17 pm
Location: Germany, Europe

Re: Considering python for a major calculation, help needed!

Postby mr.sxxg » Thu Jan 09, 2014 6:00 pm

Kebap wrote:
mr.sxxg wrote:If you think of all the bad things real programmers advise us against, all the bad habits I can shot my leg with, all the crappy ways to handle pointer or allocation bunched - I call that my programming skill.

...

I really lose a *lot* of time figuring programming details rather than figuring physics and that annoys me.


These seem to be valid arguments for you to try and prototype your next code with python instead. One of the main advantages is the much increased usability and lowered time to get from ideas to working code. So much, some people say the reduced running time of said code can be ignored compared to those advantages. But you can of course still port a working prototype to C in the end, if you feel the need.

I can recommend the google python classes by Nick Parleante for a first intro. They are intended for people who already have some programing knowledge, but are also very low-key. They also cover how to test your ideas in the python interactive console directly, instead of cycling through massive help files. I find this extremely helpful and miss the feature in other languages.


Thanks for the yt tip,Ill be sure to check it out! I think I will give py a go!
mr.sxxg
 
Posts: 3
Joined: Tue Jan 07, 2014 9:57 pm

Re: Considering python for a major calculation, help needed!

Postby mr.sxxg » Thu Jan 09, 2014 6:07 pm

casevh wrote:Hi,

Interesting questions! I'll state upfront that I won't have an answer for you.

Instead of looking at the problem as a choice between Python or C, I would ask "Can I use Python and C together?" I think it is easier to write and debug Python code (this is a Python forum...) but C code is faster (I actually write more C than Python...). But all the C code that I write is called from Python.

Take the parts of your project that are relatively high-level and not CPU bound and write them in Python (GUI display, data management are candidates). Prototype your algorithms in Python. Use as many of the scientific libraries as possible (see below). If your algorithm doesn't mesh well with the libraries, then code them in C - either as standalone applications reading data files or as extensions called from Python. Separating the high-level logic from the core computation may make it easier to use multiple computers instead of just a single computer.

The Python ecosystem has many tools for the scientific programmer: numpy, scipy, Sage, sympy, cython, numba, matplotlib, ipython, and more. If your problem can use these tools, they may offer enough performance.

Regarding threading... Python's design makes the use of multiple cores via threading challenging. Python extensions written in C can use multiple cores (directly, or by releasing the internal synchronization lock (aka GIL) that Python uses). I would suggest looking at multiple processes instead. Using distinct processes will scale directly to using multiple computers. But if your algorithms are tightly coupled, you may lose too much time to I/O overhead.

One last bit of advice: there is an old adage "A Fortran programmer can write Fortran code in any language." Try not to write a C program in Python, and vice versa. Each language has advantages and programming styles that are best suited for each language.

Not sure if this helps or not...

casevh

Note: I have not used Cython or numba but I found an interesting article comparing them:

http://nbviewer.ipython.org/url/jakevdp ... thon.ipynb


This actually helps a lot! Not just with the link,where I learned pypy has issues with numpy (which would be critical in my original project), but for broader view on c and py. I think Ill really do what you suggested - especially because much can be done in uncoupling some things and getthis thing going on all my workstations,where I can pick up to 5x the speed,which would cover much of the learning time. Thanks people! And see you around! ;)
mr.sxxg
 
Posts: 3
Joined: Tue Jan 07, 2014 9:57 pm


Return to General Discussions

Who is online

Users browsing this forum: snippsat, W3C [Linkcheck] and 4 guests