unknown
1970-01-01 00:00:00 UTC
In later versions of BitC, the issue was resolved by making many separators
optional in much the way that was done in Haskell and similar languages. A
stateful intermediate pass is injected between the parser and the lexer
that inserts missing tokens (in our case mainly based on indentation). In
the usual algorithm, most such separators end up turning into terminators
(i.e. the last one is no longer optional by the time the parser is seeing
the token stream). The intermediate pass is stateful mainly because it
needs to keep track of various forms of brackets, and because the main
grammar requires two-token lookahead to identify missing tokens in certain
cases and can end up pushing a bracket back onto the input token stream.
Once I had the two-token lookahead and the pushback mechanism working, the
resulting grammar turned out to be surprisingly clean.
People have strong views pro and con about significant white space. What I
would mainly note here is that automatic terminator insertion tends to
finesse the problem by (a) making the last separator *appear* optional by
virtue of auto-insertion while (b) nonetheless retaining a regular and
consistent grammar.
I grumbled a lot about the difficulty of doing whitespace-driven insertion.
To do it well does require that the underlying language honor certain
bracketing properties. But if the language grammar is designed with
auto-insertion in mind, the results can be *very* pleasant.
Jonathan
--e89a8fb2062e62c03f04cc8d1439
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
In most languages, support for "trailing separators" (that is: terminators) tends to be very <i>ad hoc</i>. For example, a trailing ',' eventually came to be permitted in C/C++ enumerations, but not in arguments.<div>
<br></div><div>
optional in much the way that was done in Haskell and similar languages. A
stateful intermediate pass is injected between the parser and the lexer
that inserts missing tokens (in our case mainly based on indentation). In
the usual algorithm, most such separators end up turning into terminators
(i.e. the last one is no longer optional by the time the parser is seeing
the token stream). The intermediate pass is stateful mainly because it
needs to keep track of various forms of brackets, and because the main
grammar requires two-token lookahead to identify missing tokens in certain
cases and can end up pushing a bracket back onto the input token stream.
Once I had the two-token lookahead and the pushback mechanism working, the
resulting grammar turned out to be surprisingly clean.
People have strong views pro and con about significant white space. What I
would mainly note here is that automatic terminator insertion tends to
finesse the problem by (a) making the last separator *appear* optional by
virtue of auto-insertion while (b) nonetheless retaining a regular and
consistent grammar.
I grumbled a lot about the difficulty of doing whitespace-driven insertion.
To do it well does require that the underlying language honor certain
bracketing properties. But if the language grammar is designed with
auto-insertion in mind, the results can be *very* pleasant.
Jonathan
--e89a8fb2062e62c03f04cc8d1439
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
In most languages, support for "trailing separators" (that is: terminators) tends to be very <i>ad hoc</i>. For example, a trailing ',' eventually came to be permitted in C/C++ enumerations, but not in arguments.<div>
<br></div><div>