to py or not to py ?

to py or not to py ?

Post by Serge Orlo » Thu, 29 Jun 2006 05:47:21



Do it yourself using struct module.


Take a look at Mercurial < http://www.yqcomputer.com/ ; sources.
It's a high performance python application. Or watch Bryan
O'Sullivan's Mercurial presentation
< http://www.yqcomputer.com/ ; he
talks briefly how they made it work fast.

But writing high performance application in python requires
self-discipline and attention to details, looking at the way you spell
I think it will be a challenge ;)
 
 
 

to py or not to py ?

Post by Carl J. Va » Thu, 29 Jun 2006 05:51:07


One more comment would be that running parallel applications on python
can be problematic because of the way python does threading (and i'm not
presently aware of other threading libraries). Parallelizing in python
doesn't do much good unless you spend a lot of time blocking on IO
because of the GIL.

--

Carl J. Van Arsdall
XXXX@XXXXX.COM
Build and Release
MontaVista Software

 
 
 

to py or not to py ?

Post by Steve Hold » Thu, 29 Jun 2006 16:08:49


[...]

This doesn't distinguish Python: are there any languages that you can
write high-performance applications in *without* self-discipline and
attention to details?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.yqcomputer.com/
Love me, love my blog http://www.yqcomputer.com/
Recent Ramblings http://www.yqcomputer.com/
 
 
 

to py or not to py ?

Post by Robert Ker » Thu, 29 Jun 2006 16:47:21


Use protocol 2. Also, since it looks like you will be dealing with homogeneous
arrays of numerical data, you should think about using numpy to store that data.
With protocol 2, cPickle only adds 129 bytes per array no matter the size.

http://www.yqcomputer.com/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
 
 

to py or not to py ?

Post by Robert Ker » Thu, 29 Jun 2006 17:11:54


I do essentially the same thing (only with pickles of constructive solid
geometry definitions).


... because?


An old, but still relevant overview is here:

http://www.yqcomputer.com/

A good event-driven framework for Python is Twisted. It's what I use for the
program I mention above.

http://www.yqcomputer.com/

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
 
 
 

to py or not to py ?

Post by Carl J. Va » Fri, 30 Jun 2006 02:21:12


Because of the GIL only one thread can actually run at a time. So if
you are going for speed (say you have an SMP box) and your two requests
require significant computation, you'd want each processor to take on a
little bit of that work to get things done faster. Now say you want
these two to be computed simultaneously and you are using python, you
won't get that, one thread will process and then the other will process
as the GIL is passed around between the python threads. Now if your
bottleneck is IO then you are in good shape for python threads (Your
machine only has one network port and you don't have a lot of
computation overhead, so your single processor can serve things up
faster than your network card can take it).

So its not problems that arise, its more like inefficiency. Anyhow, it
all depends on what you are trying to do and the needs of your
application.

-carl

--

Carl J. Van Arsdall
XXXX@XXXXX.COM
Build and Release
MontaVista Software
 
 
 

to py or not to py ?

Post by Tom Plunke » Fri, 30 Jun 2006 07:06:40


I've recently been wondering about this, since in the work I do, a lot
of time is spent doing disk I/O. So if I want the UI to remain
responsive, I could spawn an IO thread to handle requests, and do a
pretty simple "just whack new requests onto the queue" without locks
since I'm guaranteed to not have the IO thread read at the same time
as the requestor thread?

...what exactly constitutes an atomic operation in Python, anyway?

e.g.

class IoThread:
# ...

# called from the other thread...
def RequestFile(self, name):
self.fileQueue.append(name)

# called during the IO thread
def GetNextFile(self);
next = self.fileQueue[0]
self.fileQueue.pop(0)
return next

?
-tom!
 
 
 

to py or not to py ?

Post by Steve Hold » Fri, 30 Jun 2006 21:46:24


Standard (and excellent) advice to those new to Python: use threading,
not thread, and have threads communicate using Queue.Queue(s).

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.yqcomputer.com/
Love me, love my blog http://www.yqcomputer.com/
Recent Ramblings http://www.yqcomputer.com/
 
 
 

to py or not to py ?

Post by Carl J. Va » Sat, 01 Jul 2006 03:58:40


Well, although only one thread can run at a time due to the GIL you
can't accurately predict when the GIL is going to be released and
therefore you don't know when another thread is going to pick up and
start going (GIL is released on every so many byte instructs - correct
me if i'm wrong, certain operations that have a bit to do with IO, and
modules you wrote yourself where you manually release the GIL using
macros provided in the C API). If you have your own data structure that
is shared among threads you can use the threading modules
synchronization constructs to get the job done, using locks, conditions,
and events. Queue.Queue is also a good way to go about communicating
with threads.





--

Carl J. Van Arsdall
XXXX@XXXXX.COM
Build and Release
MontaVista Software