[Python-3000] Practical uses of stackless technology

[Carl Friedrich Bolz]

Hi stackless-dev!

As some of you might know, the PyPy project is aiming at including many
of stackless' features into its Python interpreter. Most of them are now
implemented and we're now working on the report we are writing for the
EU about this part of our work.

Being technical hacker types, we can cope with describing the ins and
outs of how our code works, but are less sure on the motivations for
stackless-style technologies as used in real-world applications :)
(especially since this part of PyPy is not used for any applications
yet).

So this is where we come to you, the existing stackless community: why
are you interested in stackless? What benefits does it bring to you and
your business? (the EU always likes to hear that businesses are using
things :)

As well as the more general stackless ideas (tasklets, channels, etc) we
are interested in hearing about any use people are making of the tasklet
pickling features.

It would be wonderful if we could get some nice quotable, attributable
snippets, but general ideas would also be very welcome.

Cheers and thanks,

mwh & Carl Friedrich Bolz

[Carl Friedrich Bolz]

Post by Carl Friedrich Bolz
Hi stackless-dev!
As some of you might know, the PyPy project is aiming at including many
of stackless' features into its Python interpreter. Most of them are now
implemented and we're now working on the report we are writing for the
EU about this part of our work.
Being technical hacker types, we can cope with describing the ins and
outs of how our code works, but are less sure on the motivations for
stackless-style technologies as used in real-world applications :)
(especially since this part of PyPy is not used for any applications
yet).
So this is where we come to you, the existing stackless community: why
are you interested in stackless? What benefits does it bring to you and
your business? (the EU always likes to hear that businesses are using
things :)
As well as the more general stackless ideas (tasklets, channels, etc) we
are interested in hearing about any use people are making of the tasklet
pickling features.
It would be wonderful if we could get some nice quotable, attributable
snippets, but general ideas would also be very welcome.
Cheers and thanks,
mwh & Carl Friedrich Bolz

argh! completely wrong list of course. Very sorry for the noise.

[Talin]

Post by Carl Friedrich Bolz

Post by Carl Friedrich Bolz
Hi stackless-dev!
As some of you might know, the PyPy project is aiming at including many
of stackless' features into its Python interpreter. Most of them are now
implemented and we're now working on the report we are writing for the
EU about this part of our work.
Being technical hacker types, we can cope with describing the ins and
outs of how our code works, but are less sure on the motivations for
stackless-style technologies as used in real-world applications :)
(especially since this part of PyPy is not used for any applications
yet).
So this is where we come to you, the existing stackless community: why
are you interested in stackless? What benefits does it bring to you and
your business? (the EU always likes to hear that businesses are using
things :)
As well as the more general stackless ideas (tasklets, channels, etc) we
are interested in hearing about any use people are making of the tasklet
pickling features.
It would be wonderful if we could get some nice quotable, attributable
snippets, but general ideas would also be very welcome.
Cheers and thanks,
mwh & Carl Friedrich Bolz

argh! completely wrong list of course. Very sorry for the noise.

Maybe the wrong list - but maybe the right discussion. Specifically, how
much of the stackless technology should be adopted in Python3K?

I haven't used stackless myself, but I *have* written code that pushes
generators to their limits and beyond, and wanted something very much
*like* stackless's capabilities to be integrated into the mainstream.

My particular application is writing backtracking search engines,
Prolog-style unification, and so on. This requires writing generators
that communicate across many recursive stack levels.

One of these days I want to re-write my algebraic solver using a
stackless-type interpreter. Unfortunately I already have 3 versions of
Python installed on this machine (2.3 cause Mac OS X requires it, 2.4.4
because my primary apps haven't been ported to 2.5 yet, and 2.5 for
writing new code) and I really don't want to install yet another.

I understand, from reading the docs on stackless, that it jumps through
some fairly significant hoops to try and keep the Python C API intact.
Given the "cruft-removal" nature of Py3K, what could be done to minimize
the amount of hoop-jumping that stackless has to do?

-- Talin

[Giovanni Bajo]

Post by Talin
Maybe the wrong list - but maybe the right discussion. Specifically,
how much of the stackless technology should be adopted in Python3K?

Nowadays, I have moved to greenlet. It is a bare-bone implementation of
continuations, without channels, schedulers, ecc. It is fine for basic
continuation usages, and it is just a standard Python extension. If it were
standardized more, I'm sure it would grow a Python higher-level library
implementing missing constructs on it.

[Richard Tew]

Post by Talin

Post by Carl Friedrich Bolz
Being technical hacker types, we can cope with describing the ins and
outs of how our code works, but are less sure on the motivations for
stackless-style technologies as used in real-world applications :)
(especially since this part of PyPy is not used for any applications
yet).

Maybe the wrong list - but maybe the right discussion. Specifically, how
much of the stackless technology should be adopted in Python3K?

I currently maintain Stackless for what its worth and I also use it in
my day to day work at CCP.

The primary benefits of Stackless IMO are gained through the use of
the minimal but solid set of features which it provides, implemented on
top of its switching.

I am refering to the scheduler, the tasklets (aka microthreads) and
channels which it provides. These are all tools which I consider a
natural enhancement to Python because of the way they facilitate
writing simple and readable code.

What I am saying is that the biggest gain is to adopt it in its
entirely.

Post by Talin
I understand, from reading the docs on stackless, that it jumps through
some fairly significant hoops to try and keep the Python C API intact.
Given the "cruft-removal" nature of Py3K, what could be done to minimize
the amount of hoop-jumping that stackless has to do?

I think that any docs you may find on the web should be taken
with a grain of salt. Pretty much all of them are out of date. And
while there might be some aspects that are still relevant, I know
they only served to confuse me when I was initially trying to
understand Stackless so that I could port it to later releases of
mainline Python. Stackless has changes a lot since it had
continuations at its core (nowadays the complication they
gave is nowhere to be seen and it has had several rewrites
since then IIRC).

As I understand it, Stackless does not do its hoop-jumping
to keep the C API intact. Rather the smaller part of its
patches which could be qualified as hoop jumping serve
two main purposes.

a) Unwinding and reducing how much of the stack is used.
b) Switching to another tasklet without doing the "hard"
switching. (otherwise known as "soft" switching).

"Hard" switching is where the stack contents are saved, and
the hand-written assembly code is invoked to change the
stack pointer and save any registers. Reducing the how much
stack is used also of course reduces how much the "hard"
switching has to save.

The bulk of the Stackless code is in the Stackless
directory, and implements hard switching, the
scheduler, tasklets and channels. And pickling of
running code of course.

The smaller part of it is the changes to mainline code.

Hope this helps,
Richard.

[Andrew Koenig]

Post by Carl Friedrich Bolz
Being technical hacker types, we can cope with describing the ins and
outs of how our code works, but are less sure on the motivations for
stackless-style technologies as used in real-world applications :)

Continuations. Which in turn are useful in multiuser interactive
applications, among other places.