The system has an invariant that if you ever return an error
sentinel (error environment, error type) then that sentinel is caused
by an error that was reported to the user. We have had too many bugs
over the last little while where that was not the case!
(An example is if we mis-interpret the tree by calling `nth_tree` with
the wrong index or something, and get `None`, and think "oh must be a
syntax error", but it was really just the wrong index. Then there's an
error sentinel with no error diagnostic and we don't discover the
mistake until much farther along.)
Now we enforce this by requiring that whoever constructs the error
sentinel *prove* that they can do so by providing a diagnostic. It's
less efficient but prevents the problem.
This actually uncovered a couple of latent bugs where we were
generating error sentinels instead of a more appropriate type! Whoops!
So it turns out that I can't hold `&str` in token because it makes it
impossible to encapsulate a source file in the larger context- self
referential structure problems again. Everything gets rebuilt so that
the source can be passed through. While we're at it, more things
become Rc<> because, man..... life it too short.
Semantics in particular has become a giant hub of the module state: we
can basically just hold an Rc<Semantics> and have everything we could
possibly want to know about a source file, computed lazily if
necessary.
- Check for more error conditions
- Move to more definitive error assertions
- Simpler error messages in some cases
- Test more conditions more thoroughly, revisit old tests
