[e-lang] Idle musings on doing E over again

[Kevin Reid]

I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently?

This is the list I repeatedly got out of bed to write, over a couple hours. It has at least two items which contradict each other. It probably has some items which are really bad ideas. It does not attempt any sort of consistent level of importance of its items. But I thought it might be mildly interesting.

Am I thinking of actually doing this, you ask? Well, at the moment it seems rather impractical with my available free time and to-dos. Inventing a new language is not inherently a good thing, *unless* it turned out that, for example, all current E code could be semi-automatically rewritten into the new language, thus reusing all that historical effort. On the other hand, I suspect that E *could* be more useful to the world right now if it weren't hampered by certain design choices.

(I'd be sticking this on the wiki if it weren't down at the moment...) (Sending to friam on the off chance it's an interesting topic.)

------------------------------------------------------------------------

Guidance
--------

• Early, build an implementation targeting JavaScript/SES, because what kind of modern language doesn't run "on the web"? However, allow only membraned interop with JavaScript code, because the alternative is to discard sameness.

• Aim for facilitating less-heavyweight implementations; fast interpreters, straightforwardly powerful compilers, small runtimes. (I have no idea how to put this in concrete terms to evaluate a design.)

• Start writing code with decent error handling from the start -- Make sure that every common error at all levels of abstraction produces nice results at the REPL as well as nice caller code. Use the feedback from this to get our error-propagation and error-value semantics right.

Call semantics (apply)
----------------------

• Coherent capability-styled error-propagation semantics. (The current problem is that post-sealed-throw, the natural synchronous error handling is ejectors and the natural asynchronous error handling is returning a broken reference, but there is no automatic conversion to be had. Perhaps sealed-throw was my mistake.)

• Errors are a sort of return value, broken references, but they're a sort of return value you can't incidentally-discard. Think of them like Maybe; the default action on a call is to match Just _.
If you want to pass on brokenness to your caller, you can explicitly choose to 'box a call' which is like a try{}; this gives you success-containing-failure (i.e. a slot containing a broken reference?)
TODO: Figure out whether this ruins the property that you can pass around a promise that may or may not have resolved to a broken reference.
(Perhaps the answer is that these failures are *not* broken references; they're one bit different from broken references. Boxing a call gets you a broken reference instead. You can also 'yield a call' which is explicitly passing on the broken reference you might receive. Hooray, we've invented too many kinds of calls!)

• Going the other direction, trying to have the ejector semantics in eventual programming: have a special argument-placeholder which means "The smash facet of the resolver for the innermost enclosing eventual send". (The eventual send syntactic sugar can set up a shadowing binding.)

• Consider defining errors to cause state rollback as well as nonlocal exit.

• Problem with E inheritance design: it's conflated with matchers, so type mirandas (respondsTo/getAllegedType) have the funny behavior of invoking the matcher.

• Think harder about how/if we want to provide property-access patterns; can we cooperate with methods-are-properties languages like JavaScript and Python? The original property access scheme failed because its fallback to get/set did not compose with inheritance, so it got watered down to just a getFoo/setFoo wrapper, but that is capitalization-based name mangling which is evil.

Kernel/AST/Eval
---------------

• Explicitly permit implementations to intern verbs.

• Prototype actual implementation of the ‘quasi-parser calls may be constant folded’ early.

• Keyword arguments of some sort; choose map syntax which is amenable to this or -- ooh ooh! -- build in _record objects_ to the language, which are optimized for a static set of keys.

Surface Syntax/Nonkernel AST
----------------------------

• Aim for a syntax with the elegance of Smalltalk. Try to reject keywords, or permit them only in ways which namespace them (e.g. my old idea of <keywords> like <this>).

• (Perhaps) Instead of environments which spill siblingward in the AST, provide such scoping as part of a general "omit the right bracket/flatten nesting" surface syntax. (I suspect that this doesn't actually work as it is less expressive, but it's worth testing, I think.)

• One way to test this: start with a Lisp-style surface syntax, and include a generic flattener.

• Can't be sufficient: consider objects defined within the arguments of a method call (e.g. a lexical version of the Measviz initialization).

• Build a simpler syntax, with less C-lineage-isms. Today's world is more diverse; the people who might be interested in E are less likely to be scared off by syntax. (Those who want familiar syntax can use JS/SES.) We tried tweaking our syntax towards C/Java to improve adoption and it didn't notably help.

• Syntax such that comments readily map to specific comment nodes in AST. This means that source code tools can refactor or prettyprint without losing comments.

• Make it easier to apply auditors all over the place; auditors on methods.

• Doc-comments-are-quasis from the ground up.

Naming
------

• Allow non-strings to be nouns and verbs, thus enabling
- trivial gensyms for expanded code. (Have a way to mark names as not-outside-this-scope so that we can choose to stringify them).
- avoidance of name conflicts in public "on this object" protocols.
• Idea: nominal-typed-interface objects which automatically provide name symbols.
interface foo { method x() } def object implements foo { method foo.x() {...} }
- Private protocols, as in the JS private name proposal — if we can make "not proxiable" work for us.

• Review Joule's design which contained non-string verbs.
• If nouns and verbs can be selfish, then also revisit the restriction of LiteralExpr to specific objects. Let's do better than Common Lisp!

• Start with some sort of module system that has no global namespace, or at least uses URIs as its global namespace.

• Avoid introducing any "well-known environment" issues in the distributed object protocol; start testing with application-local objects from the start.

Standard Library
----------------

• Build stream types into the standard library from the start, and design the library-idioms so that streams are cheap.

• Include a parser generator (PEG?) so as to quickly build self-hosting language tools and tools for data languages (e.g. XML, JSON) and little languages (e.g. regular expressions).

• Build HTML/XML/JSON quasis from the start, because interop with the web is a good thing.

• Design a coherent standard library, with less of a mishmash of Java-inherited objects plus E objects. Make sure that all the expected facilities of a modern language's library are reasonably available. (What are good resources for that?)

• Don't include synchronous file access.

• Do include POSIX interfaces so we aren't crippled by lack of e.g.
• file permission setting
• sockets
• capable terminal I/O

• Avoid the file-URL vs file-local-pathname confusion in the current API.

• (And on that note,) Figure out whether the URI-literal scheme can be kept without the every-URI-"scheme"-is-a-lexical-variable property.

[Dan Bornstein]

I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently? […]
(Sending to friam on the off chance it's an interesting topic.)

I hesitate to write this, since I like to be more of an "underpromise
and overdeliver" kind of guy, but the resonance was just too strong.

I spent the last year or so working in JavaScript and Node (for a
startup; our product is <https://medium.com/>). Going into it a year
ago, I thought that this pair of technologies / products would be a
good basis for doing the next layer of programming language and
platform development. Today, not so much.

Having gotten product out the door, I took a vacation. One of my goals
for the vacation was to start in earnest on what I thought a good
next-language might be, in terms of my own career / life-work arc.
Where I ended up was that I had a few ideas for an ultimate goal — in
particular I'm currently interested in exploring where all of
promise-based computation, a nearly-immutable data model, and software
transactional memory (for the mutable parts) meet — but I was (and am)
utterly dissatisfied with the prospects of exploring this territory
using any of the existing higher-level tools and languages that I
researched. My dissatisfaction stemmed from issues along the lines of
too many dependencies, poor semantic fit for the problem, and
philosophical incompatibility.

This led me to take a week and a bit to prototype what I think will be
a good-enough system to use as a substrate for my own further work. A
bit meta, I know. As with Kevin, I don't know how much progress I'll
be able to make in the short term, particularly because the work isn't
well-aligned with my job responsibilities. That said, I'm currently
quite excited about the prospects and have been spending an awful lot
of my evening and weekend time on it.

One thing that I feel is important at this level is an integral
parsing facility that can be used to write understandable code, and so
the prototype includes syntax for PEG grammar specification. (In a
way, this is a decades-overdue acknowledgment that Snobol had the
right idea here.)

Another high-order bit for me is avoiding gratuitous dependencies.
Getting to a self-hosted system seems pretty sensible to me, but I
don't want to have a ton of prerequisites for getting to that point.
Where I landed was to use C as the implementation language for a
minimal implementation, with the aim being for the code to be correct
and readable, eschewing both completeness and efficiency as primary
goals. Along the lines of how Scheme48 and PyPy are built, this C
implementation is meant to host exactly one program, a compiler for a
complete language implementation.

Because of where I'm positioning this in the stack, I'm aiming for
transparent Posix support but *not* ubiquitous asynchronous I/O (even
though I agree that that's ultimately desirable).

Though Kevin's and my short- to medium-term goals are probably
different, I think he is sensing the same underlying need in the
world. And I'll disagree, and take the stance that inventing a new
language *is* inherently good, and I want to help make it easier to do
so. Wish me luck!

Cheers,

-dan

[Kevin Reid]

— but I was (and am) utterly dissatisfied with the prospects of exploring this territory using any of the existing higher-level tools and languages that I researched. My dissatisfaction stemmed from issues along the lines of too many dependencies, poor semantic fit for the problem, and philosophical incompatibility.

Can you say something about what tools you researched? Should I pursue this work, I will of course need to choose a platform to start prototyping from, and it would be nice to reuse some research even if the results are negative.

One thing that I feel is important at this level is an integral
parsing facility that can be used to write understandable code, and so
the prototype includes syntax for PEG grammar specification. (In a
way, this is a decades-overdue acknowledgment that Snobol had the
right idea here.)

Parsing should be in the standard library. I'm not sure that a particular parsing technique should be built into the *language*. In E, we have quasiliterals (which I would keep nearly as-is in this hypothetical E-rev-2), which can be used to embed arbitrary little-languages; this is how I would embed a parser (a grammar) in a program. (There's actually a yac

Another high-order bit for me is avoiding gratuitous dependencies.
Getting to a self-hosted system seems pretty sensible to me, but I
don't want to have a ton of prerequisites for getting to that point.

I don't care for working with self-hosting-only systems because I want to be able to take it apart and change everything and not have a bootstrapping problem.

Though Kevin's and my short- to medium-term goals are probably
different, I think he is sensing the same underlying need in the
world. And I'll disagree, and take the stance that inventing a new
language *is* inherently good, and I want to help make it easier to do
so. Wish me luck!

I wish you luck.

Wish me sufficient free time!

[Dan Bornstein]

Post by Kevin Reid
Can you say something about what tools you researched?

To be clear, this wasn't academic research, and I didn't take notes.
Just briefly, I looked at / considered all of these (in no particular
order, and almost certainly missing some): JavaScript / V8 / Node,
Clojure, Go, LLVM, ANTLR, lex/yacc, C++, peg/leg, Haxe/NekoVM, and
Erlang.

Post by Kevin Reid
Parsing should be in the standard library. I'm not sure that a particular parsing technique should be built into the *language*.

At least for my own current work, I'm interested in making parsing
more fundamental, but YMMV of course!

My take is that tools like Awk (and to a lesser extent JavaScript)
have amply demonstrated that baking particular parsing techniques into
the language itself can make for a compelling programming experience.
I'm interested in exploring this space, but with a parsing language
that's better than standard(ish) regex syntax.

Post by Kevin Reid
I don't care for working with self-hosting-only systems because I want to be able to take it apart and change everything and not have a bootstrapping problem.

I generally agree. To reiterate, I am interested in a self-hosted
system as a platform for further development. I am looking forward to
doing a lot of non-self-hosted work on top of that platform.

Post by Kevin Reid
Wish me sufficient free time!

I wish you sufficient free time!

-dan

[Mark S. Miller]

[+awarth]

Have you looked at ometa <http://tinlizzie.org/ometa/>?

Post by Dan Bornstein

Post by Kevin Reid
Can you say something about what tools you researched?

To be clear, this wasn't academic research, and I didn't take notes.
Just briefly, I looked at / considered all of these (in no particular
order, and almost certainly missing some): JavaScript / V8 / Node,
Clojure, Go, LLVM, ANTLR, lex/yacc, C++, peg/leg, Haxe/NekoVM, and
Erlang.

Post by Kevin Reid
Parsing should be in the standard library. I'm not sure that a

particular parsing technique should be built into the *language*.
At least for my own current work, I'm interested in making parsing
more fundamental, but YMMV of course!
My take is that tools like Awk (and to a lesser extent JavaScript)
have amply demonstrated that baking particular parsing techniques into
the language itself can make for a compelling programming experience.
I'm interested in exploring this space, but with a parsing language
that's better than standard(ish) regex syntax.

Post by Kevin Reid
I don't care for working with self-hosting-only systems because I want

to be able to take it apart and change everything and not have a
bootstrapping problem.
I generally agree. To reiterate, I am interested in a self-hosted
system as a platform for further development. I am looking forward to
doing a lot of non-self-hosted work on top of that platform.

Post by Kevin Reid
Wish me sufficient free time!

I wish you sufficient free time!
-dan
_______________________________________________
e-lang mailing list
http://www.eros-os.org/mailman/listinfo/e-lang

[Dan Bornstein]

Post by Mark S. Miller
Have you looked at ometa <http://tinlizzie.org/ometa/>?

I hadn't. Thanks!

-dan

[Kevin Reid]

Post by Mark S. Miller
[+awarth]
Have you looked at ometa <http://tinlizzie.org/ometa/>?

How coincidental! I just picked out OMeta/JS as an experimental platform, as I wanted to use PEG for the grammar, and it is close to the sort of parser capabilities I'd like to have built into the resulting system.

First impression is that the implementation is mildly crufty, but nothing I can't work around, or fix and submit patches for:

• introduces many global definitions and modifies prototypes; I'll
either namespaceize the code or stuff it in an iframe.

• code generator emits undeclared variables for grammar names and is
therefore incompatible with strict mode eval; for now a regex solved
the problem but I imagine it would also be a one-line fix to the code.

(Still not promising that I'm actually going to follow through on the whole reinvent-E thing, but I'm having "fun" trying to figure out a coherent syntax, anyway.)

[Ben Laurie]

Post by Dan Bornstein

Post by Kevin Reid
Can you say something about what tools you researched?

To be clear, this wasn't academic research, and I didn't take notes.
Just briefly, I looked at / considered all of these (in no particular
order, and almost certainly missing some): JavaScript / V8 / Node,
Clojure, Go, LLVM, ANTLR, lex/yacc, C++, peg/leg, Haxe/NekoVM, and
Erlang.

Not Dart? :-)

[Thomas Leonard]

Hi Kevin,

If you want feedback from users, these were the main problems for us:

1. VatTP not using standard TLS libraries -> security risk due to lack of
updates.

2. CapTP "ping-of-death" vulnerabilities due to pipelining: e.g.
server<-getVersion()<-pow(2147483647).

3. Lack of a framework for persisting and reviving database resources
(timeMachine not useful; had to build our own).

4. Lack of static type-checking (Java developers are used to the compiler
checking things for them). Lack of a proper record type is related.

5. Performance (especially start-up time). Static type checking would help a
lot here, I think.

People complained a bit about the syntax, but I don't think anything else
would have been better. If it had been pure Java, they'd have had to learn
the same concepts. Personally, I think it's very good.

Integration with Java was very easy and useful, though very hard to explain to managers.

________________________________________
From: e-lang-***@mail.eros-os.org [e-lang-***@mail.eros-os.org] on behalf of Kevin Reid [***@switchb.org]
Sent: 28 September 2012 15:50
To: Discussion of E and other capability languages; ***@googlegroups.com
Subject: [e-lang] Idle musings on doing E over again

I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently?

This is the list I repeatedly got out of bed to write, over a couple hours. It has at least two items which contradict each other. It probably has some items which are really bad ideas. It does not attempt any sort of consistent level of importance of its items. But I thought it might be mildly interesting.

Am I thinking of actually doing this, you ask? Well, at the moment it seems rather impractical with my available free time and to-dos. Inventing a new language is not inherently a good thing, *unless* it turned out that, for example, all current E code could be semi-automatically rewritten into the new language, thus reusing all that historical effort. On the other hand, I suspect that E *could* be more useful to the world right now if it weren't hampered by certain design choices.

(I'd be sticking this on the wiki if it weren't down at the moment...) (Sending to friam on the off chance it's an interesting topic.)

------------------------------------------------------------------------

Guidance
--------

• Early, build an implementation targeting JavaScript/SES, because what kind of modern language doesn't run "on the web"? However, allow only membraned interop with JavaScript code, because the alternative is to discard sameness.

• Aim for facilitating less-heavyweight implementations; fast interpreters, straightforwardly powerful compilers, small runtimes. (I have no idea how to put this in concrete terms to evaluate a design.)

• Start writing code with decent error handling from the start -- Make sure that every common error at all levels of abstraction produces nice results at the REPL as well as nice caller code. Use the feedback from this to get our error-propagation and error-value semantics right.

Call semantics (apply)
----------------------

• Coherent capability-styled error-propagation semantics. (The current problem is that post-sealed-throw, the natural synchronous error handling is ejectors and the natural asynchronous error handling is returning a broken reference, but there is no automatic conversion to be had. Perhaps sealed-throw was my mistake.)

• Errors are a sort of return value, broken references, but they're a sort of return value you can't incidentally-discard. Think of them like Maybe; the default action on a call is to match Just _.
If you want to pass on brokenness to your caller, you can explicitly choose to 'box a call' which is like a try{}; this gives you success-containing-failure (i.e. a slot containing a broken reference?)
TODO: Figure out whether this ruins the property that you can pass around a promise that may or may not have resolved to a broken reference.
(Perhaps the answer is that these failures are *not* broken references; they're one bit different from broken references. Boxing a call gets you a broken reference instead. You can also 'yield a call' which is explicitly passing on the broken reference you might receive. Hooray, we've invented too many kinds of calls!)

• Going the other direction, trying to have the ejector semantics in eventual programming: have a special argument-placeholder which means "The smash facet of the resolver for the innermost enclosing eventual send". (The eventual send syntactic sugar can set up a shadowing binding.)

• Consider defining errors to cause state rollback as well as nonlocal exit.

• Problem with E inheritance design: it's conflated with matchers, so type mirandas (respondsTo/getAllegedType) have the funny behavior of invoking the matcher.

• Think harder about how/if we want to provide property-access patterns; can we cooperate with methods-are-properties languages like JavaScript and Python? The original property access scheme failed because its fallback to get/set did not compose with inheritance, so it got watered down to just a getFoo/setFoo wrapper, but that is capitalization-based name mangling which is evil.

Kernel/AST/Eval
---------------

• Explicitly permit implementations to intern verbs.

• Prototype actual implementation of the ‘quasi-parser calls may be constant folded’ early.

• Keyword arguments of some sort; choose map syntax which is amenable to this or -- ooh ooh! -- build in _record objects_ to the language, which are optimized for a static set of keys.

Surface Syntax/Nonkernel AST
----------------------------

• Aim for a syntax with the elegance of Smalltalk. Try to reject keywords, or permit them only in ways which namespace them (e.g. my old idea of <keywords> like <this>).

• (Perhaps) Instead of environments which spill siblingward in the AST, provide such scoping as part of a general "omit the right bracket/flatten nesting" surface syntax. (I suspect that this doesn't actually work as it is less expressive, but it's worth testing, I think.)

• One way to test this: start with a Lisp-style surface syntax, and include a generic flattener.

• Can't be sufficient: consider objects defined within the arguments of a method call (e.g. a lexical version of the Measviz initialization).

• Build a simpler syntax, with less C-lineage-isms. Today's world is more diverse; the people who might be interested in E are less likely to be scared off by syntax. (Those who want familiar syntax can use JS/SES.) We tried tweaking our syntax towards C/Java to improve adoption and it didn't notably help.

• Syntax such that comments readily map to specific comment nodes in AST. This means that source code tools can refactor or prettyprint without losing comments.

• Make it easier to apply auditors all over the place; auditors on methods.

• Doc-comments-are-quasis from the ground up.

Naming
------

• Allow non-strings to be nouns and verbs, thus enabling
- trivial gensyms for expanded code. (Have a way to mark names as not-outside-this-scope so that we can choose to stringify them).
- avoidance of name conflicts in public "on this object" protocols.
• Idea: nominal-typed-interface objects which automatically provide name symbols.
interface foo { method x() } def object implements foo { method foo.x() {...} }
- Private protocols, as in the JS private name proposal — if we can make "not proxiable" work for us.

• Review Joule's design which contained non-string verbs.
• If nouns and verbs can be selfish, then also revisit the restriction of LiteralExpr to specific objects. Let's do better than Common Lisp!

• Start with some sort of module system that has no global namespace, or at least uses URIs as its global namespace.

• Avoid introducing any "well-known environment" issues in the distributed object protocol; start testing with application-local objects from the start.

Standard Library
----------------

• Build stream types into the standard library from the start, and design the library-idioms so that streams are cheap.

• Include a parser generator (PEG?) so as to quickly build self-hosting language tools and tools for data languages (e.g. XML, JSON) and little languages (e.g. regular expressions).

• Build HTML/XML/JSON quasis from the start, because interop with the web is a good thing.

• Design a coherent standard library, with less of a mishmash of Java-inherited objects plus E objects. Make sure that all the expected facilities of a modern language's library are reasonably available. (What are good resources for that?)

• Don't include synchronous file access.

• Do include POSIX interfaces so we aren't crippled by lack of e.g.
• file permission setting
• sockets
• capable terminal I/O

• Avoid the file-URL vs file-local-pathname confusion in the current API.

• (And on that note,) Figure out whether the URI-literal scheme can be kept without the every-URI-"scheme"-is-a-lexical-variable property.

--
Kevin Reid <http://switchb.org/kpreid/>

[Kevin Reid]

Post by Thomas Leonard
Hi Kevin,
1. VatTP not using standard TLS libraries -> security risk due to lack of
updates.

As far as I know, The Plan Always Has Been to move to standard SSL/TLS/whichever-it-is-HTTPS-uses plus key checking; it's just that nobody has gotten around to implementing this in all these years. Certainly a new from-scratch implementation would take advantage of the latest libraries.

Post by Thomas Leonard
2. CapTP "ping-of-death" vulnerabilities due to pipelining: e.g.
server<-getVersion()<-pow(2147483647).

E has always intended to support mobile code, which by definition may hang the target vat. However, I have considered the possibility of locking down unserialization by defining a subset of all messages, which is those which are processed in time linear in the input size. Such a subset, plus application-specific messages, might well solve this problem, but I have not yet tried to actually implement it.

(Tricky part of defining such a condition coherently is reference cycles in the input.)

Post by Thomas Leonard
3. Lack of a framework for persisting and reviving database resources
(timeMachine not useful; had to build our own).

This strikes me as the sort of thing which would necessarily be application-specific. Can you describe what you think would be generalizable?

Post by Thomas Leonard
4. Lack of static type-checking (Java developers are used to the compiler
checking things for them).

I would hope to have a compiler smart enough to do inlining and allocation-elimination optimizations. There'll probably be enough information to issue "this will necessarily crash at runtime" warnings -- but this is an implementation feature, not a language feature, and I'm currently largely thinking about language features.

What type of static type system do you have in mind?

Post by Thomas Leonard
Lack of a proper record type is related.

Ah, I am very much in favor of including a record type. However, I'd just like to check — what is a “record type” to you?

This are the features I am thinking of for the “record type”:
• like a Map, but exposes its keys as methods/accessors, not subscripts
• optimized for many records of the same shape (key set) with few entries
• can express a subtype which is “records with exactly these keys”
• can be pattern-matched
• automatically provides mutable and immutable variants, 'with' copy-and-mutator operations, and zippers

These are the use cases I want it to serve:
• algebraic data types
• named parameters
• multiple return values
• possibly syntax trees

Post by Thomas Leonard
5. Performance (especially start-up time). Static type checking would help a
lot here, I think.

Startup time is certainly something I intend to keep in mind. It is relevant to two use cases I am particularly interested in supporting better than current E does:
• code in web pages (the common form of distributed computing)
• command-line utilities and other programs following the launch, do something, exit pattern.

I believe most of the startup time in current E implementations is in loading code (improvable by precompilation, if you have a place to stash the compile output) and in creating the objects in a vat (improvable by keeping the core code smaller and improving execution speed).

Could you explain how you expect static type checking to help startup time?

Post by Thomas Leonard
People complained a bit about the syntax, but I don't think anything else
would have been better. If it had been pure Java, they'd have had to learn
the same concepts. Personally, I think it's very good.

I'll keep that in mind. For what it's worth, most of my wilder syntactic ideas have so far produced terrible results when I try to write down a program in them :-)

Post by Thomas Leonard
Integration with Java was very easy and useful, though very hard to explain to managers.

I do not see Java as the right target for the primary implementation, but certainly host-platform integration should be a priority, whatever the host is.

[Paul E Baclace]

The ideas represented by E would have a bigger impact if they were
embedded into a popular language, especially if that happens before
major adoption of the host language occurs. People probably know the
counterfactual story about how capabilities almost got into Java (MarkM
and Agorics associates were contracting to Sun Labs when Live Oak was
turning into Java). In the interest of time, Java was instead rushed
out with the unwieldy security ACL security.properties (it was a
milestone for security in a widely used language, but it left much to be
desired.)

Scala is really taking off now; it is transitioning from pioneer to
early adopter stage, as evidenced by use in multiple open source
projects (Spark, Scalding), frequent mentions on blogs, an online course
(coursera.org has a free undergrad level class taught by Odersky which I
am taking now), and 5 recent books (sample chapters available at
manning.com). Any primary security mechanism introduced for Scala at
this point has a good chance at becoming a de facto standard as the
adoption grows.

Trend:
http://www.google.com/trends/explore#q=scala%20sbt&cmpt=q

Scala is a shoe-in for server-side Java programmers who prefer static
typing (who wants to write unit tests specifically to fill-in for lack
of static typing?), and want something more expressive with less
boilerplate, but it also has nice interactive REPLoop, functional
composition features with early and late parameter eval (reminds me of
Scheme), first-class functions, type inference, operators, traits, a
very rich collections library, and some language features inspired by
Haskell.


Paul

Post by Kevin Reid

Post by Thomas Leonard
Hi Kevin,
1. VatTP not using standard TLS libraries -> security risk due to lack of
updates.

As far as I know, The Plan Always Has Been to move to standard SSL/TLS/whichever-it-is-HTTPS-uses plus key checking; it's just that nobody has gotten around to implementing this in all these years. Certainly a new from-scratch implementation would take advantage of the latest libraries.

Post by Thomas Leonard
2. CapTP "ping-of-death" vulnerabilities due to pipelining: e.g.
server<-getVersion()<-pow(2147483647).

E has always intended to support mobile code, which by definition may hang the target vat. However, I have considered the possibility of locking down unserialization by defining a subset of all messages, which is those which are processed in time linear in the input size. Such a subset, plus application-specific messages, might well solve this problem, but I have not yet tried to actually implement it.
(Tricky part of defining such a condition coherently is reference cycles in the input.)

Post by Thomas Leonard
3. Lack of a framework for persisting and reviving database resources
(timeMachine not useful; had to build our own).

This strikes me as the sort of thing which would necessarily be application-specific. Can you describe what you think would be generalizable?

Post by Thomas Leonard
4. Lack of static type-checking (Java developers are used to the compiler
checking things for them).

I would hope to have a compiler smart enough to do inlining and allocation-elimination optimizations. There'll probably be enough information to issue "this will necessarily crash at runtime" warnings -- but this is an implementation feature, not a language feature, and I'm currently largely thinking about language features.
What type of static type system do you have in mind?

Post by Thomas Leonard
Lack of a proper record type is related.

Ah, I am very much in favor of including a record type. However, I'd just like to check — what is a “record type” to you?
• like a Map, but exposes its keys as methods/accessors, not subscripts
• optimized for many records of the same shape (key set) with few entries
• can express a subtype which is “records with exactly these keys”
• can be pattern-matched
• automatically provides mutable and immutable variants, 'with' copy-and-mutator operations, and zippers
• algebraic data types
• named parameters
• multiple return values
• possibly syntax trees

Post by Thomas Leonard
5. Performance (especially start-up time). Static type checking would help a
lot here, I think.

• code in web pages (the common form of distributed computing)
• command-line utilities and other programs following the launch, do something, exit pattern.
I believe most of the startup time in current E implementations is in loading code (improvable by precompilation, if you have a place to stash the compile output) and in creating the objects in a vat (improvable by keeping the core code smaller and improving execution speed).
Could you explain how you expect static type checking to help startup time?

Post by Thomas Leonard
People complained a bit about the syntax, but I don't think anything else
would have been better. If it had been pure Java, they'd have had to learn
the same concepts. Personally, I think it's very good.

I'll keep that in mind. For what it's worth, most of my wilder syntactic ideas have so far produced terrible results when I try to write down a program in them :-)

Post by Thomas Leonard
Integration with Java was very easy and useful, though very hard to explain to managers.

I do not see Java as the right target for the primary implementation, but certainly host-platform integration should be a priority, whatever the host is.

[Kevin Reid]

The ideas represented by E would have a bigger impact if they were embedded into a popular language, especially if that happens before major adoption of the host language occurs.

The main idea represented by E, object-capability security, has been embedded into ECMAScript 5 (essentially JavaScript); with a relatively simple initialization library, known as SES (Secure ECMAScript), in a modern (sufficiently standards-conformant) browser, you can convert any web page's scripting environment into a capability system. Certainly it is not “before major adoption”, but Mark Miller feels that this is the most important direction currently for practical adoption.

I'm talking about / sketching out this new-E idea not because I think pushing caps-in-ES5 is not the right option, but because I think it is important to continue the research project that E has been, and because I like E itself as a language -- it just needs some tweaking.

(Also, working with SES is my day job, and this is for fun.)

[Paul E Baclace]

I'm aware of SES, but my current interest is in server-server secure
distributed processing design patterns where no browser is involved.
Secure browsing is interesting to me as a user, but I'm not focused on
UI as a developer except for integration of existing components.

If anyone *is* interested in extending Scala for capabilities, I'd like
to hear about it.

Paul

Post by Kevin Reid

The ideas represented by E would have a bigger impact if they were embedded into a popular language, especially if that happens before major adoption of the host language occurs.

The main idea represented by E, object-capability security, has been embedded into ECMAScript 5 (essentially JavaScript); with a relatively simple initialization library, known as SES (Secure ECMAScript), in a modern (sufficiently standards-conformant) browser, you can convert any web page's scripting environment into a capability system. Certainly it is not “before major adoption”, but Mark Miller feels that this is the most important direction currently for practical adoption.
I'm talking about / sketching out this new-E idea not because I think pushing caps-in-ES5 is not the right option, but because I think it is important to continue the research project that E has been, and because I like E itself as a language -- it just needs some tweaking.
(Also, working with SES is my day job, and this is for fun.)

[Mark S. Miller]

---------- Forwarded message ----------
From: martin odersky <***@epfl.ch>
Date: Tue, Oct 2, 2012 at 10:14 AM
Subject: Re: [e-lang] Idle musings on doing E over again

[...] What do you think about an ocap variant of Scala? What would need to
change? How compatible could it be with existing Scala programs?

Hi Mark,

The main problem that I see here is Java interop. As long as Java is not
capability based the whole thing would be leaky. And then of course there's
the installed Scala base, which we can't break.

We are currently thinking of creating a Scala to Javascript platform though
(I'll have a student at EPFL starting on that soon). The idea would be to
keep the language but adopt JS libraries instead of Java ones. That might
be another chance to do something about it. We'd certainly be very
interested to do the right thing from an ocap viewpoint there.

Happy to have this forwarded on the list.

Cheers

- Martin

[Raoul Duke]

Post by Mark S. Miller
We are currently thinking of creating a Scala to Javascript platform though
(I'll have a student at EPFL starting on that soon). The idea would be to
keep the language but adopt JS libraries instead of Java ones.

presumably SES libraries, not JS libraries?

[Lex Spoon]

Paul raised the question of an ocap version of Scala about a month and
a half ago. Here's a very late reply with some thoughts on that.

Here are two overall strategies to keep in mind:

1. Use a compiler plugin to restrict the language to what is allowed.
This is one of the two places where a compiler plugin shines (the
other one being to refine the type system).

2. Start with Joe-E, modified for the Scala syntax. That will already
be a very reasonable language, and you could work out from there,
adding one restricted feature at a time after you analyze it for
ambient authority.


Here are some thoughts on the language itself:

1. It looks valuable to restrict case classes be real data holders, at
least unless they have some kind of @Unsafe annotation on them (do
other ocap variants have such an escape hatch?). I recall there was
some exploration in E of various kinds of frozen and transparent
data-holder objects. Similar things can be done in Scala-E by
restricting case classes: no mutable vars, no non-sealed non-final
classes, no overriding of compiler-generated methods such as equals().

2. The general nesting in Scala is likely to be helpful. When I tried
to make an ocap version of Smalltalk, I foundered on the inability to
put classes inside methods.

3. Scala has good concurrency primitives in its "Akka" framework,
including message passing and promises. Definitely mine out as much of
Akka as will make sense, so as to make Scala-E fit better in the
existing Scala ecosystem.

4. The type system can help! Upcasting and downcasting can potentially
be used like seal and unseal. Upcast is seal, downcast is unseal. So
long as the subclass is local and/or private, it is possible to
control what code can do the downcast, and thus what code can unseal
the treasure box and access what's inside.


Lex Spoon

[Thomas Leonard]

[...]

Post by Kevin Reid

Post by Thomas Leonard
3. Lack of a framework for persisting and reviving database resources
(timeMachine not useful; had to build our own).

This strikes me as the sort of thing which would necessarily be application-specific. Can you describe what you think would be generalizable?

For example, if I'm writing a Python server (e.g. in Django), I get a
database layer that will load objects from the database as needed. In
Java I get the same kind of thing with e.g. Hibernate. In both cases,
I can easily write services which manage collections of objects too
large to fit in memory.

In E, I could provide a single server object and do the database
access myself (or using one of the Java libraries), but then I lose
the security benefits of e.g. passing users SturdyRefs.

I made a custom system which revives E objects lazily from a database,
but there should be something provided by the standard library (my
system doesn't evict objects from memory either, so it only helps
start-up time; it doesn't stop the server running out of memory).
Probably not a language issue, except that E provides timeMachine in
privilegedScope.

Post by Kevin Reid

Post by Thomas Leonard
4. Lack of static type-checking (Java developers are used to the compiler
checking things for them).

I would hope to have a compiler smart enough to do inlining and allocation-elimination optimizations. There'll probably be enough information to issue "this will necessarily crash at runtime" warnings -- but this is an implementation feature, not a language feature, and I'm currently largely thinking about language features.

E's problem here is that it defines many things as runtime errors,
rather than compile-time. Warnings are good, but errors are better, if
possible. Also, there's a difference between:

- "This will certainly fail", and
- "This could fail in some situations"

e.g.

def foo(a, b) { return a + b }

is OK for E, but a staticly checked language would require you to
prove that "a" would have an "add" method which could accept "b" in
all cases. Of course, these compile-time errors can also become
run-time when running over a network against untrusted remote code (or
link-time, with out-of-date compiled code).

Post by Kevin Reid
What type of static type system do you have in mind?

No idea. Haskell is nice. ATS is a bit over-the-top ;-) Java would be
OK if it handled nulls and generics better.

Post by Kevin Reid

Post by Thomas Leonard
Lack of a proper record type is related.

Ah, I am very much in favor of including a record type. However, I'd just like to check — what is a “record type” to you?

So I can say e.g.:

struct User {
name :String,
email :Email
}

def register(u :User) { ... }

If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Post by Kevin Reid
• like a Map, but exposes its keys as methods/accessors, not subscripts
• optimized for many records of the same shape (key set) with few entries
• can express a subtype which is “records with exactly these keys”
• can be pattern-matched
• automatically provides mutable and immutable variants, 'with' copy-and-mutator operations, and zippers

Sounds good.

Cheers,

[Kevin Reid]

Post by Thomas Leonard
In E, I could provide a single server object and do the database
access myself (or using one of the Java libraries), but then I lose
the security benefits of e.g. passing users SturdyRefs.

If I recall correctly, the identityMgr has a 'fault handler' which lets you handle swissnum lookups. That seems like it would be sufficient to allow use of SturdyRefs with lazy-loaded objects. If not, please explain how.

Post by Thomas Leonard
I made a custom system which revives E objects lazily from a database,
but there should be something provided by the standard library (my
system doesn't evict objects from memory either, so it only helps
start-up time; it doesn't stop the server running out of memory).
Probably not a language issue, except that E provides timeMachine in
privilegedScope.

Well, it is somewhat of a 'language issue' that there exist a standard serialization tool which reflects the whole-vat scope of synchronicity and failure, but there's no particular reason it has to be the only one (hence the whole tearOffRoots protocol).

Post by Thomas Leonard
E's problem here is that it defines many things as runtime errors,
rather than compile-time. Warnings are good, but errors are better, if
- "This will certainly fail", and
- "This could fail in some situations"
e.g.
def foo(a, b) { return a + b }
is OK for E, but a staticly checked language would require you to
prove that "a" would have an "add" method which could accept "b" in
all cases. Of course, these compile-time errors can also become
run-time when running over a network against untrusted remote code (or
link-time, with out-of-date compiled code).

Uncautiously declaring "This is an error, but only if we can prove it so" is a bad idea, because it means that whether your code is runnable depends on the cleverness of the implementation. I agree that early indications of failure are good to have; we'll see how strict they can be made.

Post by Thomas Leonard

Post by Kevin Reid
What type of static type system do you have in mind?

No idea. Haskell is nice. ATS is a bit over-the-top ;-) Java would be
OK if it handled nulls and generics better.

Haskell's variety of type system has a number of disadvantages; one is (I understand) that it cannot handle OO-style subtyping; a particular value must belong to exactly one concrete type. (I'm not sure exactly how type classes fit in to this.)

Post by Thomas Leonard
struct User {
name :String,
email :Email
}
def register(u :User) { ... }
If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Yes, this is what I have in mind. In particular, what I am currently thinking is: records have an optional "type" reference. When the User guard gets an untyped (or wrong-typed?) record, it checks the record against the declared structure and produces a new record with the matching type. Thus, if the type already matches we have an O(1) check after the conversion.

Thus we can have records with just as much type information as real objects, but not require the creator of a record to have a reference to the type object (or, for network cases, to have a reference to the *same* type object). This is particularly relevant because (given sufficiently nice syntax) I want to use records as the standard solution to keyword arguments.

On the grounds of keeping the language small, what do you think of a quasi, i.e.
def makeUser := struct`
name :$String
email :$Email
`
?


On the other hand, another potentially valuable facility would be some way to avoid rerunning guards on an object even if they are purely structural guards. I've thought about providing a generalized cache for this purpose; I don't know whether doing so will turn out to be feasible (i.e. provide more performance benefit than cost).

[Raoul Duke]

Haskell's variety of type system has a number of disadvantages; one is (I understand) that it cannot handle OO-style subtyping.

you are most likely already totally aware but i'll mention it anyway,
see Scala, C#, F#, Ocaml, Haxe, et. al. for concrete implementations
of inferring things in a combined FP+OO language. (people seem to be
pretty frustrated when they go use Scala after using more just-fpish
inference.)

[Kevin Reid]

Post by Raoul Duke

Haskell's variety of type system has a number of disadvantages; one is (I understand) that it cannot handle OO-style subtyping.

you are most likely already totally aware but i'll mention it anyway,
see Scala, C#, F#, Ocaml, Haxe, et. al. for concrete implementations
of inferring things in a combined FP+OO language. (people seem to be
pretty frustrated when they go use Scala after using more just-fpish
inference.)

Actually, I haven't used any of those languages. I'll just add that to my todo list right now, since this is very relevant.

[Thomas Leonard]

[...]

Post by Kevin Reid

struct User { name :String, email :Email }
def register(u :User) { ... }
If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Yes, this is what I have in mind. In particular, what I am currently
thinking is: records have an optional "type" reference. When the User
guard gets an untyped (or wrong-typed?) record, it checks the record
against the declared structure and produces a new record with the
matching type. Thus, if the type already matches we have an O(1) check
after the conversion.

Sounds reasonable.

What about extra fields? It would be useful to accept a record with at least
the two fields above, but still preserve any other fields. Also, optional
fields (i.e. fields with default values) would be useful for backwards
compatibility.

I guess :User would coerce the fields? So after:

def user :User := raw

user.email() might not equal raw['email']?

Post by Kevin Reid
Thus we can have records with just as much type information as real
objects, but not require the creator of a record to have a reference to
the type object (or, for network cases, to have a reference to the *same*
type object). This is particularly relevant because (given sufficiently
nice syntax) I want to use records as the standard solution to keyword
arguments.
On the grounds of keeping the language small, what do you think of a
quasi, i.e. def makeUser := struct` name :$String email :$Email ` ?

I think I'd rather have some specific syntax (though no extensions to Kernel-E).

Post by Kevin Reid
On the other hand, another potentially valuable facility would be some
way to avoid rerunning guards on an object even if they are purely
structural guards. I've thought about providing a generalized cache for
this purpose; I don't know whether doing so will turn out to be feasible
(i.e. provide more performance benefit than cost).

--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[Kevin Reid]

Post by Thomas Leonard
[...]

Post by Kevin Reid

struct User { name :String, email :Email }
def register(u :User) { ... }
If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Yes, this is what I have in mind. In particular, what I am currently
thinking is: records have an optional "type" reference. When the User
guard gets an untyped (or wrong-typed?) record, it checks the record
against the declared structure and produces a new record with the
matching type. Thus, if the type already matches we have an O(1) check
after the conversion.

Sounds reasonable.
What about extra fields? It would be useful to accept a record with at least the two fields above, but still preserve any other fields.

That very much does not fit into my scheme. Could you outline some use cases?

Post by Thomas Leonard
Also, optional fields (i.e. fields with default values) would be useful for backwards compatibility.

Defaulted-optional fields will certainly exist, as these are meant to provide useful semantics for keyword-arguments (as Maps are used in current E, but more expressively and thus better for rough analyses).

Post by Thomas Leonard
def user :User := raw
user.email() might not equal raw['email']?

Yes, but only if you have something that coerces in that field, of course. This is basically the same as you had written

def User := Tuple[String, Email]

Post by Thomas Leonard

Post by Kevin Reid
On the grounds of keeping the language small, what do you think of a
quasi, i.e. def makeUser := struct` name :$String email :$Email ` ?

I think I'd rather have some specific syntax (though no extensions to Kernel-E).

I'm afraid you'll be disappointed; my current thinking is directed towards a Smalltalk-style minimal language: keep everything out of the language that can be expressed in terms of calls and closures. Such depends, of course, on having a syntax which makes it pleasant to express those calls and closures, so I will certainly attempt to avoid outcomes which are a mess of punctuation to express simple things.

[Thomas Leonard]

Post by Kevin Reid

Post by Thomas Leonard
[...]

struct User { name :String, email :Email }

Post by Kevin Reid

Post by Thomas Leonard

Post by Kevin Reid

Post by Thomas Leonard
If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Yes, this is what I have in mind. In particular, what I am currently
thinking is: records have an optional "type" reference. When the User
guard gets an untyped (or wrong-typed?) record, it checks the record
against the declared structure and produces a new record with the
matching type. Thus, if the type already matches we have an O(1) check
after the conversion.

Sounds reasonable.
What about extra fields? It would be useful to accept a record with at least the two fields above, but still preserve any other fields.

That very much does not fit into my scheme. Could you outline some use cases?

It's generally useful to support backwards-compatible updates. e.g. a future
version of the service might declare:

struct User {
name :String,
email :Email,
phone :nullOk[Phone] := null,
}

If a user registers with extra details that an old version doesn't need then
it can just store them in the database anyway. No need to reject the message.

(in OO terms, you might say e.g. "interface NewUser extends User", and
services which need a User will also accept a NewUser)
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[David Mercer]

(Pardon the top-posting, I'm on a client that won't behave nicely)

Thomas' inheritance use case is pretty much exactly how users are stored in ldap databases. The schema definitions define a top level User that gets inherited multiple levels down until you hit the typically used inetOrgUser, which sometimes gets extended.

ldap records normally carry around all of the types up the schema definition tree that they inherit for faster searching by object type, but that need not be so. OO languages that have built in ldap libraries normally instanciate them as their most specific type and let the languages inheritance mechanism check if they are usable in a call that takes a type that they inherit from.

So I suppose my point is that there are LOTS of very large object databases where Thomas' use case is a very common occurrence. Many of these corporate ldap databases I've dealt with in a past life as an infrastructure consultant have user types that get extended in java with all kinds of company specific data tacked onto them.

-David Mercer


--
David Mercer - http://dmercer.tumblr.com
IM: AIM: MathHippy Yahoo/MSN: n0tmusic
Facebook/Twitter/Google+/Linkedin: radix42
FAX: [phone] - BlackBerry PIN: 2105FDB2


-----Original Message-----
From: Thomas Leonard <***@it-innovation.soton.ac.uk>
Sender: e-lang-***@mail.eros-os.org
Date: Fri, 12 Oct 2012 10:16:51
To: <e-***@mail.eros-os.org>
Reply-To: Discussion of E and other capability languages
<e-***@mail.eros-os.org>
Subject: Re: [e-lang] Idle musings on doing E over again

Post by Kevin Reid

Post by Thomas Leonard
[...]

struct User { name :String, email :Email }

Post by Kevin Reid

Post by Thomas Leonard

Post by Kevin Reid

Post by Thomas Leonard
If "u" is received over the network then the system should verify its
structure and reject it if it's wrong. But when passing between
functions within a vat, it shouldn't keep rechecking it.

Yes, this is what I have in mind. In particular, what I am currently
thinking is: records have an optional "type" reference. When the User
guard gets an untyped (or wrong-typed?) record, it checks the record
against the declared structure and produces a new record with the
matching type. Thus, if the type already matches we have an O(1) check
after the conversion.

Sounds reasonable.
What about extra fields? It would be useful to accept a record with at least the two fields above, but still preserve any other fields.

That very much does not fit into my scheme. Could you outline some use cases?

It's generally useful to support backwards-compatible updates. e.g. a future
version of the service might declare:

struct User {
name :String,
email :Email,
phone :nullOk[Phone] := null,
}

If a user registers with extra details that an old version doesn't need then
it can just store them in the database anyway. No need to reject the message.

(in OO terms, you might say e.g. "interface NewUser extends User", and
services which need a User will also accept a NewUser)
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[Raoul Duke]

Post by David Mercer
Thomas' inheritance use case

well he showed Interface, not Class, extension; interface extension
seems slightly less problematic than actual is-a Class inheritance
because i can still do has-a that meets the interface spec. (down with
java-style oo! ;-)

[Raoul Duke]

Post by David Mercer
Thomas' inheritance use case

well he showed Interface, not Class, extension; interface extension
seems slightly less problematic than actual is-a Class inheritance
because i can still do has-a that meets the interface spec. (down with
java-style oo! ;-)

[Kevin Reid]

Post by Thomas Leonard
struct User {
name :String,
email :Email,
phone :nullOk[Phone] := null,
}
If a user registers with extra details that an old version doesn't need then it can just store them in the database anyway. No need to reject the message.

What comes to mind now is that there could be a pattern for extensible records: add a single field for extra data,
unknowns :Map[String, Data],
such that the User (version 1) guard coerces (using placeholder syntax!)
struct [
name = "Alice",
email = <mailto:***@example.org>,
phone = <tel:[phone]>,
]
into the typed struct
User [
name = "Alice",
email = <mailto:***@example.org>,
unknowns = ["phone" => <tel:[phone]>],
]
. That is, all unknown fields are stuffed into a plain Map (hm, or perhaps an untyped record). This keeps the record semantics simple for applications that don't want extensibility, and makes it easy to process extra fields if wanted. What do you think?

Post by Thomas Leonard
(in OO terms, you might say e.g. "interface NewUser extends User", and
services which need a User will also accept a NewUser)

That's a slightly different case -- then there exists a data type which knows the fields. In order to have this in my scheme, we'd have to essentially have a User guard which is "non-final" in the Java sense -- and then that raises the question of how much the behavior is allowed to vary.

[David-Sarah Hopwood]

Post by Kevin Reid

It's generally useful to support backwards-compatible updates. e.g. a future version
struct User {
name :String,
email :Email,
phone :nullOk[Phone] := null,
}
If a user registers with extra details that an old version doesn't need then it can
just store them in the database anyway. No need to reject the message.

What comes to mind now is that there could be a pattern for extensible records: add a
single field for extra data,
unknowns :Map[String, Data],
such that the User (version 1) guard coerces (using placeholder syntax!)
struct [
name = "Alice",
phone = <tel:[phone]>,
]
into the typed struct
User [
name = "Alice",
unknowns = ["phone" => <tel:[phone]>],
]
. That is, all unknown fields are stuffed into a plain Map (hm, or perhaps an untyped
record). This keeps the record semantics simple for applications that don't want
extensibility, and makes it easy to process extra fields if wanted. What do you think?

I prefer full 'row polymorphism'. The difference between that and an 'unknown' field
is that you can extend the record with new fields that are treated in exactly the same
way as old fields, which I think is important particularly for evolution of library
interfaces.

The type system complication is worth it since the main source of the complication is
in supporting subtyping at all, not in supporting this particular kind of subtyping.
In particular, if you have row polymorphism for fields then you can use it for
methods, approximately [*] as in OCaml
(http://caml.inria.fr/pub/docs/u3-ocaml/ocaml-objects.html), and potentially other
things like effects, as in Koka (http://lambda-the-ultimate.org/node/4589).


[*] As shown on the caml.inria.fr page, object types in OCaml are actually distinct
from record types, although using much of the same type system machinery.
I think this is overcomplicated. OCaml also tries to do too much inference on
object types IMHO. The general idea, though, is sound.

[Thomas Leonard]

Post by Kevin Reid

Post by Thomas Leonard
struct User {
name :String,
email :Email,
phone :nullOk[Phone] := null,
}
If a user registers with extra details that an old version doesn't need then it can just store them in the database anyway. No need to reject the message.

What comes to mind now is that there could be a pattern for extensible records: add a single field for extra data,
unknowns :Map[String, Data],
such that the User (version 1) guard coerces (using placeholder syntax!)
struct [
name = "Alice",
phone = <tel:[phone]>,
]
into the typed struct
User [
name = "Alice",
unknowns = ["phone" => <tel:[phone]>],
]
. That is, all unknown fields are stuffed into a plain Map (hm, or perhaps an untyped record). This keeps the record semantics simple for applications that don't want extensibility, and makes it easy to process extra fields if wanted. What do you think?

Is the type of a record visible to E?

If not, records are basically just Maps, and the guards just verify the
structure. Not running the guards repeatedly is simply an optimisation.

However, that's a bit odd, because the guards will only be called when
passing record between vats, not within a vat, and this difference will be
visible. It's not necessarily a problem, but a bit un-E-like. Unless record
guards are more limited than regular guards (e.g. pure functions which
verify but do not coerce).

If the type of a record is visible to E (e.g. a User behaves differently to
a NewUser), then sending a record to a remote vat must also send its type
information. Again, that will be tricky with user-specified guards, since
they can't travel in general.

How are you planning to implement them?
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[Kevin Reid]

Post by Thomas Leonard
Is the type of a record visible to E?

Yes.

Post by Thomas Leonard
If the type of a record is visible to E (e.g. a User behaves differently to a NewUser), then sending a record to a remote vat must also send its type information. Again, that will be tricky with user-specified guards, since they can't travel in general.

This is a particular case of the general distributed programming problem. It should be possible that if you have two instances of the same 'application' in two vats, then a record delivered between them preserves its type; on the other hand, it should be possible to not have 'the same application' on the other end and get an untyped record.

Here is one possible strategy, described in terms of current E:

Let us assume that CapTP uses Data-E serialization, with one customization: on the receiving end, if any given *call* (i.e. object construction/lookup) fail, then instead of aborting the unserialization, a broken reference is substituted. (This is probably a good idea for robustness, anyway: you would like to be able to make use of the parts of the message you understand.)

Then, suppose that the Portrayal of a typed record is a call to

toTypedRecord(type :RecordGuard, values :UntypedRecord)

which has this behavior: if the _type_ argument is a broken reference, then it returns an untyped record. (It would otherwise be identical to "values :type".)

This gives us DWIM behavior: if the type survives the trip, then the record is typed, else it is not.

Post by Thomas Leonard
How are you planning to implement them?

However is efficient, of course.

I'm figuring to design them such that a record *can* be implemented in a packed format; e.g. a type pointer and value field pointers, where the type object contains the layout information including field names. This is an implementation detail, and non-machine-level implementations can't do it this way (explicitly).

[Karp, Alan H]

Post by Kevin Reid
What comes to mind now is that there could be a pattern for extensible
records: add a single field for extra data,
unknowns :Map[String, Data],
such that the User (version 1) guard coerces (using placeholder syntax!)
struct [
name = "Alice",
phone = <tel:[phone]>,
]
into the typed struct
User [
name = "Alice",
unknowns = ["phone" => <tel:[phone]>],
]
. That is, all unknown fields are stuffed into a plain Map (hm, or
perhaps an untyped record). This keeps the record semantics simple for
applications that don't want extensibility, and makes it easy to
process extra fields if wanted. What do you think?

That's exactly what Stiegler and I did for SCoopFS. The UI running in the browser uses the waterken back end for persistence. We used this approach to avoid Marc having to change the Java code every time I decided I needed to stash a new piece of information. The downside is that the map field is essentially untyped, so you lose the advantages (Well, I think they are advantages.) of compile-time type checking.

________________________
Alan Karp
Principal Scientist
Virus Safe Computing Initiative
Hewlett-Packard Laboratories
1501 Page Mill Road
Palo Alto, CA 94304
[phone], fax [phone]
http://www.hpl.hp.com/personal/Alan_Karp

[Bill Frantz]

Post by Kevin Reid

Post by Thomas Leonard
1. VatTP not using standard TLS libraries -> security risk due to lack of
updates.

As far as I know, The Plan Always Has Been to move to standard
SSL/TLS/whichever-it-is-HTTPS-uses plus key checking; it's just
that nobody has gotten around to implementing this in all these
years. Certainly a new from-scratch implementation would take
advantage of the latest libraries.

I spent a considerable amount of time on this issue 12 or so
years ago. Note that SSL and TLS are the same thing, TLS is a
more modern version of SSL. HTTPS will use either, but there are
significant security issues with SSL1 and SSL2, so modern
browsers only will support SSL3, which also has issues, but they
are less likely to show up in real-world situations.

In the good old days, TLS was totally in bed with PKI and
certificate authorities. There are some recent RFCs which allow
the kind of key authentication needed for E, i.e. the public
key's hash is the vatID of the target vat.

I found the whole library thing a total none-starter. What TLS
libraries will be supported over all the target platforms? What
libraries provide the necessary interfaces to check the public
key against the vatID? I'm probably really old fashioned, but
coding the crypto from scratch seems a lot easier than answering
the above questions and validating the security of a library.

(The problem with security is that is isn't what you can do,
which can be answered by testing, but what you can't do, which
can only be answered by proof, although tiger teams can give you
some hints.)

Post by Kevin Reid

Post by Thomas Leonard
5. Performance (especially start-up time). Static type checking would help a
lot here, I think.

Startup time is certainly something I intend to keep in mind.
It is relevant to two use cases I am particularly interested in
• code in web pages (the common form of distributed computing)
• command-line utilities and other programs following the launch, do something, exit pattern.
I believe most of the startup time in current E implementations
is in loading code (improvable by precompilation, if you have a
place to stash the compile output) and in creating the objects
in a vat (improvable by keeping the core code smaller and
improving execution speed).
Could you explain how you expect static type checking to help startup time?

Java failed as a language for users because of its slow startup.
Since E was built on the Java virtual machine, it had the same
problem. Since Java is statically typed, I don't see how having
static types will help with the startup time.

Cheers - Bill

-------------------------------------------------------------------------
Bill Frantz | Airline peanut bag: "Produced | Periwinkle
[phone] | in a facility that processes | 16345
Englewood Ave
www.pwpconsult.com | peanuts and other nuts." - Duh | Los Gatos,
CA 95032

[Thomas Leonard]

Post by Bill Frantz

Post by Kevin Reid

Post by Thomas Leonard
1. VatTP not using standard TLS libraries -> security risk due to lack of
updates.

As far as I know, The Plan Always Has Been to move to standard
SSL/TLS/whichever-it-is-HTTPS-uses plus key checking; it's just
that nobody has gotten around to implementing this in all these
years. Certainly a new from-scratch implementation would take
advantage of the latest libraries.

I spent a considerable amount of time on this issue 12 or so
years ago. Note that SSL and TLS are the same thing, TLS is a
more modern version of SSL. HTTPS will use either, but there are
significant security issues with SSL1 and SSL2, so modern
browsers only will support SSL3, which also has issues, but they
are less likely to show up in real-world situations.
In the good old days, TLS was totally in bed with PKI and
certificate authorities. There are some recent RFCs which allow
the kind of key authentication needed for E, i.e. the public
key's hash is the vatID of the target vat.

That's handy. But even without it, generating a self-signed
certificate to hold the public key works fine (it's just a bit
wasteful).

Post by Bill Frantz
I found the whole library thing a total none-starter. What TLS
libraries will be supported over all the target platforms? What
libraries provide the necessary interfaces to check the public
key against the vatID? I'm probably really old fashioned, but
coding the crypto from scratch seems a lot easier than answering
the above questions and validating the security of a library.

Technically, maybe. But for adoption it's hard to convince people that
a non-standard crypto implementation will be OK. If there's a bug in
OpenSSL then:

1) it will get fixed promptly
2) the fix will get distributed (e.g. via the distributions)
3) it's not anyone's fault for choosing OpenSSL, because it's widely used

I don't think any of these are true for E's crypto.

I've only looked at Python and Java, but both provide easy access to
the server certificate used in a TLS connection.

Post by Bill Frantz
(The problem with security is that is isn't what you can do,
which can be answered by testing, but what you can't do, which
can only be answered by proof, although tiger teams can give you
some hints.)

Post by Kevin Reid

Post by Thomas Leonard
5. Performance (especially start-up time). Static type checking would help a
lot here, I think.

Startup time is certainly something I intend to keep in mind.
It is relevant to two use cases I am particularly interested in
• code in web pages (the common form of distributed computing)
• command-line utilities and other programs following the launch, do something, exit pattern.
I believe most of the startup time in current E implementations
is in loading code (improvable by precompilation, if you have a
place to stash the compile output) and in creating the objects
in a vat (improvable by keeping the core code smaller and
improving execution speed).
Could you explain how you expect static type checking to help startup time?

I meant that static typing would help for performance in general, not
startup specifically. I profiled E's startup a while back, but there
was nothing obviously the problem, just many little things.

Post by Bill Frantz
Java failed as a language for users because of its slow startup.
Since E was built on the Java virtual machine, it had the same
problem. Since Java is statically typed, I don't see how having
static types will help with the startup time.

E's startup time is a lot worse than Java's (even though Java is
indeed far too slow):

$ time python hello.py
python hello.py 0.02s user 0.00s system 84% cpu 0.024 total

$ time java Hello
java Hello 0.06s user 0.02s system 98% cpu 0.082 total

$ time rune hello.e
rune hello.e 9.23s user 0.34s system 200% cpu 4.785 total

[Kevin Reid]

Post by Thomas Leonard
I profiled E's startup a while back, but there
was nothing obviously the problem, just many little things.
[...]
E's startup time is a lot worse than Java's (even though Java is
$ time python hello.py
python hello.py 0.02s user 0.00s system 84% cpu 0.024 total
$ time java Hello
java Hello 0.06s user 0.02s system 98% cpu 0.082 total
$ time rune hello.e
rune hello.e 9.23s user 0.34s system 200% cpu 4.785 total

My hypothesis, which I don't recall the origin of and so may or may not be based on actual analysis, is that most of this time is spent in

• evaluating static initializers and global tables in E's Java code (which includes things like SafeJ tables)

• parsing/expanding/evaluating one-off E code

These could be improved by figuring out how to 'compile' the E implementation better, i.e. precompute the constant stuff.

Here's an experiment that could be tried: run hello.e not by the usual 'rune' (which ends up invoking a rather elaborate E program which does special things for 'top-level' programs), but by a Java stub which does nothing but create a vat, then parse and evaluate the code.

[Brian Warner]

Post by Bill Frantz
I found the whole library thing a total none-starter. What TLS
libraries will be supported over all the target platforms? What
libraries provide the necessary interfaces to check the public key
against the vatID? I'm probably really old fashioned, but coding the
crypto from scratch seems a lot easier than answering the above
questions and validating the security of a library.

As I mentioned on the captalk list last month, I'd be inclined to use
djb's NACL library (http://nacl.cr.yp.to/) to get
non-connection-oriented pair-wise encrypted sessions:

http://www.eros-os.org/pipermail/cap-talk/2012-September/015386.html

It's a lot smaller than OpenSSL (easy to embed in your app), and isn't
trying to accomodate legacy compatibility (which removes a large attack
surface). It also enables store-and-forward relaying of messages,
instead of requiring real-time live connections, which is a better fit
for Waterken's "one message at a time" style (but could still be used
with E's "live references which can die" style).

On the other hand, it doesn't immediately provide perfect forward
secrecy (ephemeral session keys, which isn't the default in SSL, but is
theoretically achievable). And you might feel obligated to convince
somebody that Curve25519/Salsa20/Poly1305 is secure despite being less
well-known than RSA/AES/HMAC.


cheers,
-Brian

[Norman Hardy]

Post by Brian Warner
http://nacl.cr.yp.to/

[Norman Hardy]

Sorry for noise! Finger glitch.

Post by Brian Warner
http://nacl.cr.yp.to/

[Mark S. Miller]

I think the lack of perfect forward secrecy is a significant issue. Is
there a good way to provide it within this framework?

Note that E's current VatTP has always provided perfect forward secrecy
(thanks Bill and Tyler!) and there are std TLS cyphersuites that do as well.

Post by Brian Warner

Post by Bill Frantz
I found the whole library thing a total none-starter. What TLS
libraries will be supported over all the target platforms? What
libraries provide the necessary interfaces to check the public key
against the vatID? I'm probably really old fashioned, but coding the
crypto from scratch seems a lot easier than answering the above
questions and validating the security of a library.

As I mentioned on the captalk list last month, I'd be inclined to use
djb's NACL library (http://nacl.cr.yp.to/) to get
http://www.eros-os.org/pipermail/cap-talk/2012-September/015386.html
It's a lot smaller than OpenSSL (easy to embed in your app), and isn't
trying to accomodate legacy compatibility (which removes a large attack
surface). It also enables store-and-forward relaying of messages,
instead of requiring real-time live connections, which is a better fit
for Waterken's "one message at a time" style (but could still be used
with E's "live references which can die" style).
On the other hand, it doesn't immediately provide perfect forward
secrecy (ephemeral session keys, which isn't the default in SSL, but is
theoretically achievable). And you might feel obligated to convince
somebody that Curve25519/Salsa20/Poly1305 is secure despite being less
well-known than RSA/AES/HMAC.
cheers,
-Brian
_______________________________________________
e-lang mailing list
http://www.eros-os.org/mailman/listinfo/e-lang

[Brian Warner]

Post by Mark S. Miller
I think the lack of perfect forward secrecy is a significant issue. Is
there a good way to provide it within this framework?

It can be done, but takes more work, and moves things back to being more
connection-oriented.

http://curvecp.org/ is worth studying. It's an encrypted replacement for
TCP (including forward-secrecy, flow-control, support for client
IP-address mobility, even hiding the client's long-term key from anyone
but the server), based on Curve25519. It requires a couple of round
trips to set up, and provides the usual byte pipe, so it's basically an
improved version of TLS.

You could get forward-secrecy without the connectionness of CurveCP by
having each side choose a short-term Curve25519 DH key, exchange them
under protection of a long-term Ed25519 signing key, then encrypt the
data with the derived session key. Every once in a while, one side picks
a new Curve25519 key, and starts using it (they must sign the "I'm using
a new key now" message with the long-term key). The two sides don't even
need to coordinate their changes.

The easy-but-noisy approach would include both sides short-term keys in
every message, along with long-term key's signature. Then you'd only
have to remember a sequence number. If a hundred bytes of overhead on
each message is too much, you could only send the keys when they change,
and have each side remember the most recent key in addition to the
sequence number.

The amount of forward-secrecy you get all comes down to when you choose
to forget the session key (and switch to a new one). With a
connection-oriented scheme, the loss of the connection is an obvious
place to forget things and thus protect the future (but even then, you
might want to speed up new connections by using some session-resumption
feature, and that loses the forward-secrecy, so it's not always
obvious). More sophisticated schemes will negotiate some sort of
periodic key rotation, so the "sessions" will be longer than a single
TCP connection but still not forever.

Post by Mark S. Miller
Note that E's current VatTP has always provided perfect forward
secrecy (thanks Bill and Tyler!) and there are std TLS cyphersuites
that do as well.

Yup. The issue, as always, is how easy it'd be to get at the needed
functionality. Most TLS libraries are very CA-centric. If you were
designing a new protocol, I don't know if it'd be easier to A: include
the crypto in the code and tell implementors/porters to provide a
platform function to deliver the encrypted messages, or B: rely upon the
platform's TLS library and tell implementors to find a way to intercept
the cert-validation process correctly. If the crypto is small enough, I
suspect A is more likely to work out, but maybe I've just had too much
exposure to TLS libraries :).

cheers,
-Brian

[Bill Frantz]

The ideas represented by E would have a bigger impact if they were embedded into a popular

language, especially if that happens before major adoption of the host language occurs.
The main idea represented by E, object-capability security, has
been embedded into ECMAScript 5 (essentially JavaScript); with
a relatively simple initialization library, known as SES
(Secure ECMAScript), in a modern (sufficiently
standards-conformant) browser, you can convert any web page's
scripting environment into a capability system. Certainly it is
not “before major adoption”, but Mark Miller feels that
this is the most important direction currently for practical adoption.
I'm talking about / sketching out this new-E idea not because I
think pushing caps-in-ES5 is not the right option, but because
I think it is important to continue the research project that E
has been, and because I like E itself as a language -- it just
needs some tweaking.

I would add promises and vat concurrency control to the list of
main ideas of E. Of the minor ideas, my favorite is quasi-literals.

Javascript already has vat concurrency control, and when people
suggest putting in thread-like features, they receive
significant pushback.

Promises can be added by libraries, but it is nice to have
language-level syntax.

Quasi-literals are proposed for future Javascript standardization.

Cheers - Bill

-----------------------------------------------------------------------
Bill Frantz | Privacy is dead, get over | Periwinkle
[phone] | it. | 16345
Englewood Ave
www.pwpconsult.com | - Scott McNealy | Los Gatos,
CA 95032

[Ben Laurie]

Post by Kevin Reid
I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently?

I'm afraid I lost track of the conversation here, but the one thing
I'd say is: don't build on top of the JVM. On top of LLVM would be
pretty neat (you might then want some kind of E-on-E interpreter for
interactive, if you really care about interactive, not sure it matters
that much, TBH - maybe you can even compile/link small programs fast
enough to not need it).

Post by Kevin Reid
This is the list I repeatedly got out of bed to write, over a couple hours. It has at least two items which contradict each other. It probably has some items which are really bad ideas. It does not attempt any sort of consistent level of importance of its items. But I thought it might be mildly interesting.
Am I thinking of actually doing this, you ask? Well, at the moment it seems rather impractical with my available free time and to-dos. Inventing a new language is not inherently a good thing, *unless* it turned out that, for example, all current E code could be semi-automatically rewritten into the new language, thus reusing all that historical effort. On the other hand, I suspect that E *could* be more useful to the world right now if it weren't hampered by certain design choices.
(I'd be sticking this on the wiki if it weren't down at the moment...) (Sending to friam on the off chance it's an interesting topic.)
------------------------------------------------------------------------
Guidance
--------
• Early, build an implementation targeting JavaScript/SES, because what kind of modern language doesn't run "on the web"? However, allow only membraned interop with JavaScript code, because the alternative is to discard sameness.
• Aim for facilitating less-heavyweight implementations; fast interpreters, straightforwardly powerful compilers, small runtimes. (I have no idea how to put this in concrete terms to evaluate a design.)
• Start writing code with decent error handling from the start -- Make sure that every common error at all levels of abstraction produces nice results at the REPL as well as nice caller code. Use the feedback from this to get our error-propagation and error-value semantics right.
Call semantics (apply)
----------------------
• Coherent capability-styled error-propagation semantics. (The current problem is that post-sealed-throw, the natural synchronous error handling is ejectors and the natural asynchronous error handling is returning a broken reference, but there is no automatic conversion to be had. Perhaps sealed-throw was my mistake.)
• Errors are a sort of return value, broken references, but they're a sort of return value you can't incidentally-discard. Think of them like Maybe; the default action on a call is to match Just _.
If you want to pass on brokenness to your caller, you can explicitly choose to 'box a call' which is like a try{}; this gives you success-containing-failure (i.e. a slot containing a broken reference?)
TODO: Figure out whether this ruins the property that you can pass around a promise that may or may not have resolved to a broken reference.
(Perhaps the answer is that these failures are *not* broken references; they're one bit different from broken references. Boxing a call gets you a broken reference instead. You can also 'yield a call' which is explicitly passing on the broken reference you might receive. Hooray, we've invented too many kinds of calls!)
• Going the other direction, trying to have the ejector semantics in eventual programming: have a special argument-placeholder which means "The smash facet of the resolver for the innermost enclosing eventual send". (The eventual send syntactic sugar can set up a shadowing binding.)
• Consider defining errors to cause state rollback as well as nonlocal exit.
• Problem with E inheritance design: it's conflated with matchers, so type mirandas (respondsTo/getAllegedType) have the funny behavior of invoking the matcher.
• Think harder about how/if we want to provide property-access patterns; can we cooperate with methods-are-properties languages like JavaScript and Python? The original property access scheme failed because its fallback to get/set did not compose with inheritance, so it got watered down to just a getFoo/setFoo wrapper, but that is capitalization-based name mangling which is evil.
Kernel/AST/Eval
---------------
• Explicitly permit implementations to intern verbs.
• Prototype actual implementation of the ‘quasi-parser calls may be constant folded’ early.
• Keyword arguments of some sort; choose map syntax which is amenable to this or -- ooh ooh! -- build in _record objects_ to the language, which are optimized for a static set of keys.
Surface Syntax/Nonkernel AST
----------------------------
• Aim for a syntax with the elegance of Smalltalk. Try to reject keywords, or permit them only in ways which namespace them (e.g. my old idea of <keywords> like <this>).
• (Perhaps) Instead of environments which spill siblingward in the AST, provide such scoping as part of a general "omit the right bracket/flatten nesting" surface syntax. (I suspect that this doesn't actually work as it is less expressive, but it's worth testing, I think.)
• One way to test this: start with a Lisp-style surface syntax, and include a generic flattener.
• Can't be sufficient: consider objects defined within the arguments of a method call (e.g. a lexical version of the Measviz initialization).
• Build a simpler syntax, with less C-lineage-isms. Today's world is more diverse; the people who might be interested in E are less likely to be scared off by syntax. (Those who want familiar syntax can use JS/SES.) We tried tweaking our syntax towards C/Java to improve adoption and it didn't notably help.
• Syntax such that comments readily map to specific comment nodes in AST. This means that source code tools can refactor or prettyprint without losing comments.
• Make it easier to apply auditors all over the place; auditors on methods.
• Doc-comments-are-quasis from the ground up.
Naming
------
• Allow non-strings to be nouns and verbs, thus enabling
- trivial gensyms for expanded code. (Have a way to mark names as not-outside-this-scope so that we can choose to stringify them).
- avoidance of name conflicts in public "on this object" protocols.
• Idea: nominal-typed-interface objects which automatically provide name symbols.
interface foo { method x() } def object implements foo { method foo.x() {...} }
- Private protocols, as in the JS private name proposal — if we can make "not proxiable" work for us.
• Review Joule's design which contained non-string verbs.
• If nouns and verbs can be selfish, then also revisit the restriction of LiteralExpr to specific objects. Let's do better than Common Lisp!
• Start with some sort of module system that has no global namespace, or at least uses URIs as its global namespace.
• Avoid introducing any "well-known environment" issues in the distributed object protocol; start testing with application-local objects from the start.
Standard Library
----------------
• Build stream types into the standard library from the start, and design the library-idioms so that streams are cheap.
• Include a parser generator (PEG?) so as to quickly build self-hosting language tools and tools for data languages (e.g. XML, JSON) and little languages (e.g. regular expressions).
• Build HTML/XML/JSON quasis from the start, because interop with the web is a good thing.
• Design a coherent standard library, with less of a mishmash of Java-inherited objects plus E objects. Make sure that all the expected facilities of a modern language's library are reasonably available. (What are good resources for that?)
• Don't include synchronous file access.
• Do include POSIX interfaces so we aren't crippled by lack of e.g.
• file permission setting
• sockets
• capable terminal I/O
• Avoid the file-URL vs file-local-pathname confusion in the current API.
• (And on that note,) Figure out whether the URI-literal scheme can be kept without the every-URI-"scheme"-is-a-lexical-variable property.
--
Kevin Reid <http://switchb.org/kpreid/>
_______________________________________________
e-lang mailing list
http://www.eros-os.org/mailman/listinfo/e-lang

[Kevin Reid]

Post by Ben Laurie

Post by Kevin Reid
I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently?

I'm afraid I lost track of the conversation here, but the one thing
I'd say is: don't build on top of the JVM.

What do you see as the disadvantages of doing so?

As noted in my original message, I see the best option as building on top of JavaScript. This is, yes, yet another high-level VM. However, it is one which is extremely widely available; I see it as the best chance to make a difference.

I would like to know whether your criticism of the JVM applies here as well.

Post by Ben Laurie
On top of LLVM would be
pretty neat (you might then want some kind of E-on-E interpreter for
interactive, if you really care about interactive, not sure it matters
that much, TBH - maybe you can even compile/link small programs fast
enough to not need it).

Based on the example set by Steel Bank Common Lisp, I would hope that we can have a sufficiently fast compiler to compile interactive execution. (For that matter, E-on-CL did, though I never exercised the interactive functionality very much.)

In the case of a high-level target, the best path seems to me to be a bootstrap interpreter written in the target, and a compiler to the target written in E. An interpreter for current E written in E is nearly trivial.

In the case of something like LLVM, I'm not sure what I'd choose for a bootstrap. I don't care for fully self-hosting systems that require bootstrap binaries and constrain system changes to incremental ones.

[Ben Laurie]

Post by Kevin Reid

Post by Ben Laurie

Post by Kevin Reid
I was trying to get to sleep last night, and was struck by the thought of: If we were designing E from scratch (with the same goals), what would I actually do differently?

I'm afraid I lost track of the conversation here, but the one thing
I'd say is: don't build on top of the JVM.

What do you see as the disadvantages of doing so?
As noted in my original message, I see the best option as building on top of JavaScript. This is, yes, yet another high-level VM. However, it is one which is extremely widely available; I see it as the best chance to make a difference.
I would like to know whether your criticism of the JVM applies here as well.

Only somewhat :-)

They worst thing about the JVM is its lack of portability and openness.

What's bad about both options is that you inherit a pile of security
issues that are nothing to do with you, and your runtime speed is
limited by their runtime speed...

Post by Kevin Reid

Post by Ben Laurie
On top of LLVM would be
pretty neat (you might then want some kind of E-on-E interpreter for
interactive, if you really care about interactive, not sure it matters
that much, TBH - maybe you can even compile/link small programs fast
enough to not need it).

Based on the example set by Steel Bank Common Lisp, I would hope that we can have a sufficiently fast compiler to compile interactive execution. (For that matter, E-on-CL did, though I never exercised the interactive functionality very much.)
In the case of a high-level target, the best path seems to me to be a bootstrap interpreter written in the target, and a compiler to the target written in E. An interpreter for current E written in E is nearly trivial.
In the case of something like LLVM, I'm not sure what I'd choose for a bootstrap. I don't care for fully self-hosting systems that require bootstrap binaries and constrain system changes to incremental ones.

You lost me :-)

Post by Kevin Reid
--
Kevin Reid <http://switchb.org/kpreid/>
_______________________________________________
e-lang mailing list
http://www.eros-os.org/mailman/listinfo/e-lang

[Kevin Reid]

Post by Ben Laurie

Post by Kevin Reid
In the case of something like LLVM, I'm not sure what I'd choose for a bootstrap. I don't care for fully self-hosting systems that require bootstrap binaries and constrain system changes to incremental ones.

You lost me :-)

If the only implementation of a language is a compiler written in that language, then:

* The compiler must have already been compiled, so you have a necessary
artifact which is not source text, which raises version control
issues (and Ken Thompson issues).

* In order to modify the language L into L', you must modify the
compiler to compile L' while being written in L, and also to compile
L' while being written in L', then compile the latter with the former,
(For small changes, these two compilers might happen to be the same.)

In order to avoid these issues, there must be an implementation of the language written in a different language.

[Bill Frantz]

Post by Kevin Reid

Post by Ben Laurie
I'm afraid I lost track of the conversation here, but the one thing
I'd say is: don't build on top of the JVM.

What do you see as the disadvantages of doing so?
As noted in my original message, I see the best option as
building on top of JavaScript. This is, yes, yet another
high-level VM. However, it is one which is extremely widely
available; I see it as the best chance to make a difference.
I would like to know whether your criticism of the JVM applies here as well.

I see the most important disadvantage of the JVM is its slow
startup. (Remember, my experience is a decade old.) This
slowness doesn't matter much for servers, where the JVM is
started when the server is started, but it was a pain when I was
doing data analysis in E.

I don't know about the issues with node.js, but it would be one
likely engine for a Javascript implementation.

Cheers - Bill

-----------------------------------------------------------------------
Bill Frantz |The nice thing about standards| Periwinkle
[phone] |is there are so many to choose| 16345
Englewood Ave
www.pwpconsult.com |from. - Andrew Tanenbaum | Los Gatos,
CA 95032

[Kevin Reid]

Post by Bill Frantz
I don't know about the issues with node.js, but it would be one
likely engine for a Javascript implementation.

Node.js is not a JavaScript implementation: it is a framework for writing servers which is written in JavaScript, specifically used with the JavaScript implementation V8. (I understand it includes native, er, C code to implement such things as file-system access.)

To run an E-on-JS in a command-line / posixish context we would likely be either using Node.js or using *the same techniques as* Node without using it as an implementation layer; I'm not sure which.

Disclaimer: I haven't actually studied Node.js in detail.