Whither now, Oh Scheme.

Whither now, Oh Scheme.

Post by R Racin » Tue, 28 Oct 2003 11:45:33


We are amidst a computer programming language renaissance. New languages
and lots of new users to play with them. Daily it seems.

12-18 months the #scheme IRC channel on freenode.net was essentially
abandoned. Often myself or another, on occasion a crowd of 3. As I type,
on a Sunday evening, there are 40 users.

Discussions on #scheme are generally threefold in nature; SRFI's, as
several SRFI authors are frequently on, a bit of homework or Scheme newbie
assistance, and as one might imagine a good deal of comparative
implementation discussion.

There seems to be a consensus Scheme(s) for every situation except one,
serious application development. And a number of Schemers are interested
in doing just that. A case can be made that a SIFSAD (Scheme Intended For
Serious Application Development) does not exist today, what is worse, it
is doubtful one will exist tomorrow.

But suppose there was a plan for SIFSAD, a roadmap for a Scheme Intended
For Serious Application Development, what would it look like. You would
have to start from somewhere, have a destination in mind, the path
becomes just a bit of machete work.

Assuming the best opportunity for Sifsad is an evolutionary one from the
core of an existing Scheme implementation, here are some hypothetical
Sifsad bios.

Scheme 48 / PreScheme compiler - PreScheme compiler is resurrected,
initially emitting C code. Later native emitters Itanium/AMD 64 bit
systems were added.

PLT/MzScheme - mzc compiler is enhanced with aggressive optimizations.
MzScheme becomes not only one of the functionally richest implementations
but the fastest as well.

Chicken/Bigloo/Gambit/Larceny/Scheme->C et al. Consensus is reached on one
code base, remaining authors, recognizing the will of the Scheme community
work to add the best features of each into the common code base. The
resulting Scheme->C compiler is widely regarded as the best HLL compiler
available.

Chez Scheme - Individual licenses are made available at reasonable cost.
Source is GPL'd for non-commercial use.

MITScheme - Port to new 64 bit systems is successfully achieved. Module
system, syntax-case support is added. With memory constraints lifted,
development of lightning fast, large memory footprint application are
possible in an incremental compilation environment.


Whither now, Scheme.
 
 
 

Whither now, Oh Scheme.

Post by feli » Tue, 28 Oct 2003 16:25:47

R Racine" < XXXX@XXXXX.COM > wrote in message news:< XXXX@XXXXX.COM >...

Could you elaborate on that? Why do you think (say) Bigloo or PLT
might not suitable for serious app development?


PreSchene might not be anyones favorite Scheme dialect.


Interesting alternative. But Mzc still has to provide clean interfacing
to the MzScheme runtime system, which is not really tuned for
maximum performance, but for other things (debuggability, ease of use,
robustness, etc.)


(BTW, Larceny is not a Scheme->C compiler)

So that would mean we reduce all Scheme->C compilation strategies down
to the least common divisor:

- drop Chicken's fast continuations
- drop Gambit's (forthcoming) very efficient multithreading system
- drop Bigloo's/Scheme->C's direct compilations style and make it a CPS compiler
(you want 1st class continuations and TCO, right?)

What you will get is a Scheme implementation that is either unusable,
incomplete or inefficient.


Hm. Can't say much about that...


What many people don't realize is that there CAN'T BE NO SINGLE ALL-POWERFUL
SCHEME implementation. Tradeoffs have to be made, unless you want to
produce a mediocre one. Chicken (for example) will never beat Bigloo, in
terms of raw performance, yet Bigloo's (or PLT's) continuations are
awfully inefficient. Damn, it's even impossible to pin down a single
perfect implementation strategy (Cheney-on-the-MTA? Direct style?
Trampoline style? Bytecode VM? Threaded VM?). What GC? Conservative?
Ref. counting? Stop-and-copy? Mark-and-sweep? Which is best? Or,
more importantly, which is best for *all* applications? None, I'd say.

Several Scheme implementations are more than adequate for serious development
and people use it for that. In fact, Schemes generally provide better
performance and often have better foreign function interfaces than
languages like Python, Ruby or Perl, which seem to be well accepted for serious
stuff. Scheme is more rigorously defined, is better suited to
compilation and provides incredibly powerful syntactic abstractions.

It *is* easy to get lost in the number of implementations, and many
of those are somewhat half-finished, partly because it's so easy
to whip up a simple Scheme, yet this has absolutely nothing to do
with Scheme not being ready for development of real-world code.


cheers,
felix

 
 
 

Whither now, Oh Scheme.

Post by feli » Tue, 28 Oct 2003 18:49:14


Or is petite larceny already available?
It seems it isn't, but I may be wrong.


cheers,
felix
 
 
 

Whither now, Oh Scheme.

Post by R Racin » Tue, 28 Oct 2003 22:28:37


On Sun, 26 Oct 2003 23:25:47 -0800, felix wrote:

In my previous post I mentioned a threefold path to Nirvana. Determine a
starting point, define an endpoint, get the mechete ready. To properly
select an implementation to evolve into Sifsad, it only makes sense to
select an implementation that is best to build of off. There is a
distinct chance that the "best" implementation to move forward with is not
even one of the top 2 or 3 implementations used today.

So a priori, agreed, no debate, compromises must and will occur. However,
I will debate whether it is possible to a) effectively determine which
tradeoffs to select IF the end goal is adequately defined, b) compromises
can not be ameliorated by modular code design c) such tradeoffs inevitably
result in mediocrity.

For example,
The end goal is defined.
- Speed of application. Very important. - Efficient use of large amounts
of memory. Very important.
- Full debugging. Continuation restarts ???
- Core fullblown MOP. Highly optimized dispatch. - Modules, standalone
compilation, interfaces/signatures (also parametric
interfaces/signatures) and runtime determinable implementations. [Imagine
the SRFI-44 debate on the definition of a collections library in the light
of a SIG/UNITs or Scheme48/Chez interfaces or SML sigs....
- Standalone, static exe capability.
- Real multithreading capable of utilizy multiple processors. - so on and
so forth...

The point is, define the goal and tradeoffs become a debate in the context
of what is necessary to achieve the goal.

Another point, Larceny [as you correctly pointed out is not just a
Scheme->C system, later tonight I intend to post on why proposing Larceny
makes sense] has 5 - 6 different GC systems. The Larceny core is very
well designed and supports plugable GC systems. What is the penalty for
this flexibility? I doubt the efficiency of the Twobit compiled code is
impacted. PLT also has 2 GC/VM systems. Such things can be abstracted in
the code base to support multiple solutions and pluggability with minimal
impact.

Bottom line, I believe it IS possible to allow for flexible pluggable
strategies to many of the issues you raised such as various VM strategies.

Couldn't you, being well versed on the Cheney-on-the-MTA approach either
show that this approach is decidedly superior then the MzScheme approach
or is a must have option in Sifsad and then assist in adding it to
MzScheme? (Assuming MzScheme makes sense as the base system.)

Must two or more! Scheme distributions exist, complete with different
runtimes and libraries, predicated on the single point of bifurcation as
to how continuation capture is occuring??!!

In the context of doing comparative analysis via two small experimental
systems yes. In the world of the application developer where the method
of continuation capture is invisible, it is decidedly not justification
for forking two blown Scheme systems. Just capture the damn things, make
it stable, make it fast and MAKE IT ONE Scheme System. Thank you very
much.


Regards,


Ray
 
 
 

Whither now, Oh Scheme.

Post by Scott G. M » Tue, 28 Oct 2003 23:21:52


I hope I'm wrong, but it seems you have a simplified view of Scheme
architectures. Continuation capture is probably *the* fundamental
feature that drives selecting the implementation strategy. One cannot
have a modular continuation capture implementation. Thats why systems
with slow call/cc are unlikely to get much better without rearchitecting
themselves at a low level.

If I'm using continuations heavily, I'm going to want to choose an
implementation with that property. If I'm not using them all, but I
demand high performance otherwise, then I'm likely to make a completely
different choice. Its these sort of trade offs which make sifsad a bad
idea. You should ask yourself what the real problem is that prevents
serious application development. I would argue that its the lack of
large, (standard?, maybe.) library. This means covering things such as
usable GUI toolkits, extensive database connectivity, mature threading,
networking, datastructures... the sort of things career programmers take
for granted from the platform libraries of C++ or Java.

The fallacy is believing this is only possible if we standardize on one
Scheme.

Scott
 
 
 

Whither now, Oh Scheme.

Post by Ray Dillin » Wed, 29 Oct 2003 02:30:59

Scott G. Miller" wrote:

Partly.... A scheme that compiles to a well-designed intermediate
form could have two back-ends; one that heap-allocates and garbage
collects call frames, and one that uses the hardware stack. These
back-ends would generate code that obeyed two different runtime
models, but there's also a "tail" end -- keyhole optimization of
machine code -- that could be shared between them. The runtime
symbol table and associated code could also be shared between the
two models.

So you'd wind up duplicating maybe half of a simple compiler to
accomodate the fundamentally different designs. And effort spent
on the crankiest and most bottomless, nonportable areas -- machine
code and cache optimization -- would be sharable. By the time
you'd done aggressive optimizations and ported to a half-dozen
different hardware/OS combinations, the duplicated effort might
be a tenth or less of the compiler.

From a compilation point of view, it's easy to scan scheme code and
see if you can find places where call/cc is ever used. You could
make a first-order choice of which backend to invoke just by
checking for it. But the right thing to do would be to profile
it at the intermediate-code level and make a hard assessment of
which model is a "win" for the given program.

Much of what would need to be done can only be done in scheme as
a result of whole-program optimization. And that means getting
program code away from the REPL, because as long as you have the
REPL in the system, you absolutely cannot prove that something
isn't going to be redefined or mutated. It also means very
serious support for optional declarations to eliminate unnecessary
typechecks and very serious support for memory and CPU profiling.

Finally, we really *really* need a linkable object file format
that we don't have to go through an FFI for. FFI's distort or
contort the meaning of scheme code; they introduce special cases,
cause wraparound or length errors in integers, truncate complex
numbers, create exceptions to garbage collection handling, and
wreak all kinds of misfits with the runtime model. We paper over
the problems reasonably well, but still they never quite work
right. When scheme programs link to scheme libraries they shouldn't
need to use braindead C calling conventions.



I think maybe there needs to be a 'SISFAD' standard, above and
beyond R5RS, that specifies a lot of things R5RS doesn't specify.
I'd like to see a bunch of people implement it, much as a bunch
of people have implemented R5RS.

A SISFAD standard would expressly forbid some of the things that
make some schemes unusable for serious application development,
like limits on the memory size (guile and MIT scheme have this
problem particularly badly) and failure to support the full
numeric tower. It would specify a format for libraries portable
across all implementations of SISFAD, define which R5RS and other
functions are found in what libraries, define a set of OS calls
accessible through libraries, and straighten out a few things
like binary I/O primitives for pipes, sockets and files.

It would specify the syntax of performance declarations, but the
only requirement of implementations should be that they must not
barf on the syntax -- actually using it for performance enhancement
is a plus, but not barfing on it is crucial.

Bear
 
 
 

Whither now, Oh Scheme.

Post by Anton van » Wed, 29 Oct 2003 03:03:16


This makes much more sense to me than "standardizing on one Scheme".

Of course, the first thing to be standardized has to be a better acronym
than SIFSAD!!

Anton
 
 
 

Whither now, Oh Scheme.

Post by Scott G. M » Wed, 29 Oct 2003 04:19:04


I've been speaking deliberately abstractly, but many of these topics
were covered at Matthias Radestock's ILC presentation, and will likely
come up again in some detail around the Scheme Workshop and LL3. See
you there!

Scott
 
 
 

Whither now, Oh Scheme.

Post by Bruce Step » Wed, 29 Oct 2003 04:56:23

"Anton van Straaten" < XXXX@XXXXX.COM > writes:




As far as I understand it, that's what SRFIs are about. The existing
ones don't seem to me to go nearly far enough, though.

Part of what makes (for example) Perl good is CPAN, and all the
conventions (and the resulting community) that make CPAN possible. So
I can download a tarball, unpack it, run Makefile.PL using my chosen
Perl interpreter, and then "make; make test; make install" will work
(with high probability).

That's all made easier because Perl has a single implementation (give
or take), of course. Even so, if there were a common FFI (even a
restricted one), and a few extra things (a common module and/or
package system, perhaps a common object system) something similar
could be built for Scheme.

I'm guessing it won't happen, though. I'm not sure quite what it is,
but something seems to prevent such cooperation.

And that seems to mean that there isn't a scheme community in the same
way that there's a Perl community---so I can be confident of getting
Perl's LDAP package and being able to use it, but Bigloo's equivalent
< http://www.yqcomputer.com/ ; doesn't even build with
the current bigloo, presumably because bigloo's community is simply
too small. (I found much the same with some RScheme libraries, and
doubtless the same is true of most scheme implementations.)
 
 
 

Whither now, Oh Scheme.

Post by campbel » Wed, 29 Oct 2003 05:22:46

Would you like a pony, too?
 
 
 

Whither now, Oh Scheme.

Post by feli » Wed, 29 Oct 2003 07:42:24

n Mon, 27 Oct 2003 13:28:37 GMT, R Racine < XXXX@XXXXX.COM > wrote:


Possible, *if* a Sifsad (geez, what an awful name! ;-) is possible
and practical, which I seriously doubt...


No disagreement here.


But you want speed to, right? Ok, so have several optimization settings.


Oh, how about speed? I assume a simple procedure call is more efficient
(whatever tricks your dynamic dispatch plays, it will not beat the
direct procedure call, naturally). Here you have your first tradeoff.
Why do you want OO baggage in the core, when you want speed at the same
time?


What kind of modules? How easy to use should they be? Should they
allow interactive use? Man, do you realize how much work has gone into
Scheme module systems, yet none really satisfies everybody!


Yes, this is not new. People on c.l.s (and elsewhere) debate about these
things
for decades, now. Have they reached only the slightest bit of consensus?
No, they haven't. Why, I ask you.


Absolutely. Yet, there are implementation strategies that are very tightly
coupled with there collectors. One example is Cheney-on-the-MTA, another
is "traditional" direct style compilers that target C, which mostly use
conservative
GC.


Possible, yes. But not always adequate. I claim that the ideal Scheme
implementation you have in mind will be completely unusable for others.


It doesn't (if I may say so). I wouldn't touch the MzScheme sources
unless physically forced to do so. That Cheney-on-the-MTA is superior
(to direct style, like Bigloo) is something that I'm firmly convinced
off. And? That doesn't matter to someone who isn't interested in anything
but raw speed of straight-line code. Tradeoffs, again.


If you look carefully, you'll find many more differences than only
continuation capture. And capture is only *one* issue with continuations.
How about safe-for-space complexity? Reification? Storage consumption?


Many people have tried to do so. Yet, the ideal Scheme system hasn't been
done yet.
If the unification of all Scheme implementation efforts is the really
important issue for you, then you effectively strive for mediocrity,
unless you happen to be a Scheme implementation wizard, vastly ahead
of all the others. Mind you, that would be nice!


cheers,
felix
 
 
 

Whither now, Oh Scheme.

Post by Bruce Step » Wed, 29 Oct 2003 08:23:55

felix < XXXX@XXXXX.COM > writes:

[...]


Probably true. In that sense, Perl, Python, etc., are mediocre---some
reasonable uses of the languages are inefficient.

On the other hand, if you've got a one-day sort of problem to solve
that requires access to LDAP, SSL, PostgreSQL, and gtk, then the
mediocre solutions win.

Heck, people have been writing reasonable size applications in Tcl for
years, largely because it had a very convenient binding to Tk. tkman
(a *really* nice manpage reader) was first written (about 10 years
ago, apparently) when Tcl was a strongly string-based interpreter; the
author even wrote a paper about the various hackery he used to make it
fast enough (the files had non-essential spaces removed and ghastly
things like that).

Even then, there were presumably choices that ought to have been
better (Tcl's far from a perfect language, and it was much worse in
1993); but Tcl had a convenient binding to Tk and an easy to use FFI,
and that was enough for it to be more usable for a large class of
applications.

For a big application, the work necessary to bind a few libraries is
dwarfed by the work necessary to attack the real problem. However,
that leaves lots of little applications where you're naturally going
to choose a language which has lots of convenient packages. Perhaps
more importantly, I suspect big applications often start off as small
ones---something like Perl makes it easier to start work on a problem.
 
 
 

Whither now, Oh Scheme.

Post by Bradd W. S » Wed, 29 Oct 2003 09:06:08


Heck yeah. More than a few times, I've started a big project by writing
a prototype in Perl. More precisely, I try to hack it up in Perl, and if
that doesn't work, I do a better implementation in a more appropriate
language. As a bonus, the initial hack-job implementation gives me
enough experience with the problem domain that I can do a better design
for the "real" version.
--
Bradd W. Szonye
http://www.yqcomputer.com/
My Usenet e-mail address is temporarily disabled.
Please visit my website to obtain an alternate address.
 
 
 

Whither now, Oh Scheme.

Post by R Racin » Wed, 29 Oct 2003 09:12:31


I do. I represent the pitch fork wielding, torch waving, unwashed masses
of fustrated Scheme application developers. And yes, maybe I am a mass of
one. (shades of a "silent" majority here)

I am not saying that Sifsad will have some trival property flag and will
then suddenly manifest 3 modes of continuation capture.

I'm just saying that after a decade or two, is it unreasonable to suggest
that there has been enough experimental versions, and multiple approaches
to the reach a "reasonable" conclusion (not a perfect conclusion) with
regard to implementing continuation capture if one were to design Sifsad.

As one of the unwashed, I don't care how its done, I am sure I wouldn't
understand the internals if I tried. I can't slam dunk a basketball
either. So be it.

SML/NJ is fast (not the fastest, but commercially fast) and supports
continuations. And no, I am not saying, do it just like SML/NJ.

Ray
 
 
 

Whither now, Oh Scheme.

Post by Jens Axel » Wed, 29 Oct 2003 09:27:42


What is missing in DrScheme?



The Grand Unified Scheme is nothing but a dream. You will always need
to make compromises in implementations. That's why you ought to be
thrilled about the wide range of Scheme implementations in existence.
In other languages (e.g. Python/Perl) you are pretty much stuck with
one implementation.


That's a bold statement in these parts of the wood.

See the last discussion on the Grand Unified Scheme:

< http://www.yqcomputer.com/ ;


Perhaps a better idea was to begin making an FFI-SRFI?


--
Jens Axel Saard