[e-lang] for-must-match

[Thomas Leonard]

Continuing with our experimental syntax changes to improve the
reliability of E, here's a patch that adds a "for-must-match" feature.

http://gitorious.org/~tal-itinnov/repo-roscidus/it-innovation/commit/3ad4f5c3072447647c617fdb56738bf929cb8a1a

(see also: http://wiki.erights.org/wiki/Surprise_list)


Normally in E, this silently skips the third item:

for a:int in [1, 2, "hi"] {
println(a)
}


Now, it throws an exception:

pragma.enable("for-must-match")

for a:int in [1, 2, "hi"] {
println(a)
}

To make a loop that skips non-matching items as before, use the "match"
keyword. e.g.

pragma.enable("for-must-match")

for match a:int in [1, 2, "hi"] {
println(a)
}

In our code-base of > 20,000 lines of E, I only found two places making
use of the old behaviour (or at least, all the tests passed after I
updated those two ;-).

In contrast, there have been many occasions where the old behaviour
caused problems. Most commonly, someone is looping over some tuples:

for [foo, bar] in tuples { ... }

and someone changes the format to add an extra field. Since this doesn't
cause any errors, it's hard to detect that anything is wrong.
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[Thomas Leonard]

Here's a patch to add list comprehensions:

? def list := [1, 2, 3]
? [2 * x for x in list]
# value: [2, 4, 6]

http://gitorious.org/repo-roscidus/e-core/commit/9bfacc4f748b4ea6a7cf64dc40d7bc9dd1890d38

And for map comprehensions:

? def map := ["alice" => 50, "bob" => 60]
? [v => k for k => v in map]
# value: [50 => "alice", 60 => "bob"]

http://gitorious.org/repo-roscidus/e-core/commit/980c8c93d41a69f242acdb5b83c9faccf918f4c3

They don't currently support the 'match' keyword, so you can't use
them to filter (every item must be mapped to something).

[Kevin Reid]

Post by Thomas Leonard
? def list := [1, 2, 3]
? [2 * x for x in list]
# value: [2, 4, 6]
http://gitorious.org/repo-roscidus/e-core/commit/9bfacc4f748b4ea6a7cf64dc40d7bc9dd1890d38
? def map := ["alice" => 50, "bob" => 60]
? [v => k for k => v in map]
# value: [50 => "alice", 60 => "bob"]
http://gitorious.org/repo-roscidus/e-core/commit/980c8c93d41a69f242acdb5b83c9faccf918f4c3
They don't currently support the 'match' keyword, so you can't use
them to filter (every item must be mapped to something).

I reject this proposal.

1. I am uncomfortable with it because I suspect it of violating E's syntactic conventions about scoping and evaluation order, though I would have to consult MarkM to be more precise.

2. Furthermore, we already have the accumulator syntax which covers these use cases. I recognize that the accumulator syntax is ugly; its problem is that it is both too general and not general enough. This proposal is not general enough.

I would rather see additional functional operations and use of lambda-args than additional syntax for operations on lists and maps; especially if they are part of a system which solves the parallel iteration problem.


As to your previous for-must-match pragma: I doubt its appropriateness for the language. However, it is entirely appropriate to add it as a pragma as an *experiment* -- that's why we have pragmas. You may commit for-must-match, provided that the documentation and *EVERY* code addition which supports it is marked as experimental and part of for-must-match.

One of the reasons I don't like it is that explicit testing in 'for' (especially with the ? such-that pattern) is often useful. Have you considered instead adding syntax which makes a pattern "must match", i.e. converts failure into an exception (like Haskell's @ "irrefutable patterns")? The default would be the opposite of what you have, but the syntax would make it easy to express that case where it currently isn't.

[Thomas Leonard]

Post by Kevin Reid

Post by Thomas Leonard
? def list := [1, 2, 3]
? [2 * x for x in list]
# value: [2, 4, 6]
http://gitorious.org/repo-roscidus/e-core/commit/9bfacc4f748b4ea6a7cf64dc40d7bc9dd1890d38
? def map := ["alice" => 50, "bob" => 60]
? [v => k for k => v in map]
# value: [50 => "alice", 60 => "bob"]
http://gitorious.org/repo-roscidus/e-core/commit/980c8c93d41a69f242acdb5b83c9faccf918f4c3
They don't currently support the 'match' keyword, so you can't use
them to filter (every item must be mapped to something).

I reject this proposal.
1. I am uncomfortable with it because I suspect it of violating E's syntactic conventions about scoping and evaluation order, though I would have to consult MarkM to be more precise.

It probably does. However, there's also consistency with other
languages to consider. The Wikipedia page has examples both ways,
though most seem to favour putting the expression first:

Some expr first example:

[2 * x for x in items] # E (proposed)
[2 * x for x in items] # Python
[2 * x for (x in items)] # JavaScript
[2 * x | x <- items ] # Haskell
(list-of (* 2 x) (x in items)) # Scheme (non-standard)

Expr last examples:

for (x in items) { 2 * x } # JavaFX
for (x <- items) yield 2*x # Scala

The Scala syntax could work in E, although it would interfere with
break(x) and involve creating a new keyword.

The JavaFX syntax (without the "yield" keyword) wouldn't work because
ordinary for loops are already defined to return the break() value, or
null. Even if that feature were removed, it would still be impossible
to tell whether a for loop was being evaluated for its value or for
its side-effects. e.g. in

return when (list) -> { for x in list { foo(x) } }

there is no way to know whether the programmer intends to return a
promise for a list of foo() results, or merely let the caller know
when all the items have been processed.

Post by Kevin Reid
2. Furthermore, we already have the accumulator syntax which covers these use cases. I recognize that the accumulator syntax is ugly; its problem is that it is both too general and not general enough. This proposal is not general enough.

I don't see why we need very general syntax. Syntax is for things you
do so often that short-cuts are useful. Creating a list by processing
each item in another list is a well-understood and common task, so
having syntax for it makes sense (e.g. replacing map() with list
comprehensions). Other things should only get special syntax once they
turn out to be useful (syntax for experiments, like lambda-args, is
fine, but I'm talking about enabled-by-default features for people who
want to use the language, not to experiment with it).

Post by Kevin Reid
I would rather see additional functional operations and use of lambda-args than additional syntax for operations on lists and maps; especially if they are part of a system which solves the parallel iteration problem.
As to your previous for-must-match pragma: I doubt its appropriateness for the language. However, it is entirely appropriate to add it as a pragma as an *experiment* -- that's why we have pragmas. You may commit for-must-match, provided that the documentation and *EVERY* code addition which supports it is marked as experimental and part of for-must-match.
One of the reasons I don't like it is that explicit testing in 'for' (especially with the ? such-that pattern) is often useful.

$ cd esrc
$ grep 'for .*?.* in' **/*.emaker
[ no results ]

I also found no results in my code-base.
It's the default that is the problem. In the code-base I'm looking at
now, I count 238 for loops, of which 2 should silently skip
non-matching items. The rest should throw an exception if there is no
match. The default should be for the 99% case, not the 1% case (maybe
other people use it differently, but these are the numbers for our
code).

Also, the current default tends to cause mysterious failures when
something silently fails to match. By contrast, forgetting to add the
"match" keyword causes a loud and obvious failure and is easily fixed.

[Thomas Leonard]

On 2011-10-22 15:00, Kevin Reid wrote:
[...]

Post by Kevin Reid
As to your previous for-must-match pragma: I doubt its
appropriateness for the language. However, it is entirely appropriate
to add it as a pragma as an *experiment* -- that's why we have
pragmas. You may commit for-must-match, provided that the
documentation and *EVERY* code addition which supports it is marked
as experimental and part of for-must-match.

I've now committed a cut-down version of this patch. When for-must-match
is enabled, all for patterns must match or an exception is thrown (i.e.
there is no new "for match" syntax).

I just changed the two places in my code that relied on it to use an
explicit "if". They were in the same bit of code... I suspect I wrote
that just after learning about the syntax to try it out.

I'm not sure what further documentation is needed. Aren't all the syntax
pragmas experimental?

By the way: as an experiment (not committed) I changed the ENodeBuilder
code to require patterns to match always (regardless of whether
for-must-match was enabled) for all syntax versions. With the above two
fixes to my code, all tests pass and everything seems to be working
(including updoc, rune, and all of the standard library that gets used
by my code).
--
Dr Thomas Leonard
IT Innovation Centre
Gamma House, Enterprise Road,
Southampton SO16 7NS, UK


tel: [phone]

mailto:***@it-innovation.soton.ac.uk
http://www.it-innovation.soton.ac.uk/

[Thomas Leonard]

Post by Thomas Leonard
? def list := [1, 2, 3]
? [2 * x for x in list]
# value: [2, 4, 6]
http://gitorious.org/repo-roscidus/e-core/commit/9bfacc4f748b4ea6a7cf64dc40d7bc9dd1890d38
? def map := ["alice" => 50, "bob" => 60]
? [v => k for k => v in map]
# value: [50 => "alice", 60 => "bob"]
http://gitorious.org/repo-roscidus/e-core/commit/980c8c93d41a69f242acdb5b83c9faccf918f4c3
They don't currently support the 'match' keyword, so you can't use
them to filter (every item must be mapped to something).

I've now added filtering:

http://gitorious.org/repo-roscidus/e-core/commit/b407e45a23ec942683fac84fc6156809321b805d

However, I didn't use the match syntax for this because, thinking
about it further, I don't think using pattern bindings is a good way
to test for a condition.

Consider a list of pairs [String,boolean] where we want to process all
the true values. With pattern matching (current E syntax), we'd write:

for [name :String, ==true] in pairs { ... }

But this only covers two cases (match and no-match). There are really
three cases:

match [name :String, ==true] => process this item
match [_ :String, _:boolean] => skip this item
match _ => report error

For switch statements and object definitions, this isn't a problem
because when a pattern doesn't match it falls though to the next case
and will eventually be reported or handled. But in for loops errors
are silently ignored.

Therefore, I used a more traditional syntax for list comprehensions:

[name for [name :String, active :boolean] in pairs if active]

I think this is better at separating out the expected format of the
item (Tuple[String,boolean]) from the condition we want to check. You
can use a matchBind in the if to get the old behaviour if required.

[Kevin Reid]

Post by Thomas Leonard
However, I didn't use the match syntax for this because, thinking
about it further, I don't think using pattern bindings is a good way
to test for a condition.
Consider a list of pairs [String,boolean] where we want to process all
for [name :String, ==true] in pairs { ... }
But this only covers two cases (match and no-match). There are really
match [name :String, ==true] => process this item
match [_ :String, _:boolean] => skip this item
match _ => report error
For switch statements and object definitions, this isn't a problem
because when a pattern doesn't match it falls though to the next case
and will eventually be reported or handled. But in for loops errors
are silently ignored.

This -- that some things expressible as patterns are expectations/assertions and some things are match failures -- is a significant observation.

Perhaps we *should* have a general rule that any time there is *only one* pattern (i.e. not a switch), it should not have a failure behavior other than throwing (or the explicit ejector offered by def).


Another possibility is if we could have some kind of general syntax for all patterns which expressed "unexpected" (throw) vs. "expected" (match fail) mismatches. That is, being able to write (deliberately horrible syntax for example):

for <<EXPECTED>> [name :String, <<UNEXPECTED>> ==false] in pairs {...}