Many hyperlinks are disabled.
Use anonymous login
to enable hyperlinks.
50 most recent check-ins
|
2026-08-11
| ||
| 01:27 | bignum -> double is correctly rounded now; mp_get_double was not Numerical tower staging step 6, prerequisite 3. A one-ulp error, so it hid well, but it is on the path of every (inexact bignum), every mixed bignum/float arithmetic operation, and every bignum/float comparison. mp_get_double accumulates `d = d*2^MP_DIGIT_BIT + (double)dp[i]' from the top limb down. That rounds at every limb, twice per limb in fact: (double)dp[i] alone throws away 7 bits of a 60-bit limb before the addition rounds again. It is not correctly rounded and does not claim to be. Measured rather than argued: 4147 bignums checked against a correctly-rounded reference, 69 of them (1.7%) came back one ulp out. Replaced with a single rounding. Take the top 54 bits -- 53 of mantissa plus a round bit -- note whether any bit at all survives below them (sticky), round half-to-even once, and ldexp. After rounding q is at most 2^53, so (double)q is exact and ldexp is the only other operation; an exponent past DBL_MAX yields inf, which is the answer a bignum too large to represent should give. The limbs are read through the view rather than divided in the scratch arena on purpose: num_to_double is called from the middle of arithmetic and the arena is not re-entrant. No arena slot is touched, so this is safe to call from anywhere. Re-measured: 26224 cases, 0 wrong. That set is 25000 random bignums from 65 to 1100 bits, every near-tie around the 53-bit boundary for 70 different exponents, and 1842 cases that overflow to +inf.0. numcheck gains a bignum whose mantissa is not a power of two, and that is the other half of the commit. The four bignums added in the compare-fix commit are all 2^k, and a power of two converts to a double exactly -- so the matrix could not see a rounding error in this function any more than it could see the compare bug. With the messy value in, control vs new reports 508 differences, all of them on + - * / with an inexact operand, every one a last-significant-digit move. Without it, 0. Note in passing, found while writing the test and not fixed here: the printer's %.16g for a double is not enough to round trip every double. This commit's test value writes as 2.882014193361458e+54 and reads back as a *different* double; it needs 17 significant digits. write is required to emit something that reads back as the same number, and the image dump depends on that being true, so this wants its own commit. (double bignum) is still a type-error, incidentally -- the cvalue constructors go through the ten-way num_init macro and none of them has a bignum arm. Giving the macro one would also define (int8 bignum), which is the narrowing-cast territory numerical-tower.md puts explicitly out of scope, so it is left alone and the tests here go through (* x 1.0) instead. test, test7 1006/1006, selfhost-test, selfhost-expander-test and r7rs-coverage-test at the same 20 errors are all unchanged. Leaf check-in: 15f360592a user: e tags: tower | |
| 01:20 | truncate had no bignum arm, and round went through a double it did not need Numerical tower staging step 6, prerequisite 2 of 2. Like the compare fix, these are wrong answers on trunk today and land on their own merits. (truncate (expt 2 100)) was a type-error, now the bignum (floor (expt 2 100)) likewise, and ceiling, and exact (round (expt 2 100)) was 1.267650600228229e+30, now exact (round 9007199254740993) was 9007199254740992 (round 4611686018427387903) was #int64(4611686018427387904) fl_truncate tested isfixnum and iscprim and then fell to type_error. A bignum is a cvalue, so it matched neither. An exact integer is its own truncation, so the arm is one line -- but the blast radius was not one line, because (scheme base) defines floor, ceiling, round and exact all on top of truncate. Four procedures were broken for bignums by one missing test. This is also on the critical path for the rational layer: tests/color.lsp:79 is (truncate (/ x 5)) with x a fixnum, unittest.lsp loads it, and (/ x 5) becomes a ratnum in step 6 proper. round was wrong a second way, and this one is not about bignums at all. Its definition added 0.5 and truncated, so *every* exact integer took a round trip through a double it had no need of. Below 2^53 that round trip is lossless and it went unnoticed; above it, it silently perturbs the answer. 9007199254740993 and 4611686018427387903 are an ordinary fixnum and an int64 cprim, not bignums, and round was handing back a different integer than it was given. A bignum, having no double that represents it at all, came back as an approximation. The fix is to say that an exact integer is its own rounding, which is both correct and cheaper than the float path it replaces. The inexact path is untouched, deliberately: (round 3.5), (round -4.3), (round 0.5) and (round 2.5) all answer exactly as before. Note in passing, not fixed here: eflisp's floor/ceiling/truncate/round return an *exact* integer for an inexact argument, where R7RS requires inexact in, inexact out -- (round 3.5) is 4, not 4.0. lib7/chibi/test.scm:225 has the (inexact? res) check of test-equal? commented out with a note that "tests which expect an inexact value can accept an equivalent exact value", which is what lets the conformance suite pass over it. That is a real gap and it will have to be faced when numerator/denominator arrive, since r7rs-tests 982-986 require the inexact forms; it is not this commit's business. test7 goes 992 -> 1006. Fourteen of those are new eflisp-added tests here and the expected count in tools/bench-record.sh and .claude/skill-r7rs.md moves with them, or every later benchmark row would read FAIL. test, selfhost-test, selfhost-expander-test, etest7 and r7rs-coverage-test at the same 20 errors are all unchanged. numcheck was not run: truncate is not one of its nine operators, so the diff is zero by construction. check-in: a20a744a51 user: e tags: tower | |
| 01:13 | fix compare bug with bignum check-in: 26fa60880c user: e tags: tower | |
|
2026-08-10
| ||
| 23:05 | updated benchmark data check-in: 857bfa9cdf user: e tags: tower | |
| 22:16 | add reader for bignums check-in: b08dea6eb3 user: e tags: tower | |
| 21:27 | add bignums using libtommath check-in: 024ce818f4 user: e tags: tower | |
| 20:18 | add the two harnesses the numeric-tower work was verified with Both were throwaway scripts that turned out to be the only things that caught real regressions, so they belong in the tree rather than in a scratch directory. tools/numcheck.lsp + numcheck.sh -- print every arithmetic and comparison result over the cross-product of numeric representations and magnitudes (about 470k lines), run it under two binaries, diff, and classify each difference as a canonicalisation, a newly-raising case, a newly-answering case or a value change. Every suite in the tree passed unchanged through an intermediate state of the pairwise num_*2 refactor that had turned nine correct answers into overflow errors, including (* -1 #int64(4611686018427387904) 2), which is exactly -2^63. This is what found them. Validated on landing: against bf8bf26ad1 it reproduces the 6133 differences and the exact class breakdown recorded in numerical-tower.md. tools/readsweep.lsp -- read every datum of every file named on the command line and report only failures. The check for a change that makes the reader stricter, which no suite performs. Reproduces the six failures recorded for step 1. The headers carry the traps, because each of them silently produced a confident wrong answer at least once: building a control binary loses to make's mtime granularity and to stale objects (only 'make clean' is trustworthy), 'fossil cat -r REV f > f' truncates f before reading it, 'grep " error"' has no leading space to match and hides compiler errors, and zsh does not word-split an unquoted expansion so a $(find ...) argument list arrives as one filename. check-in: d0ebbf3c33 user: e tags: tower | |
| 20:07 | numeric predicates: separate representation from value (R7RS semantics) Numerical tower staging step 3. test7 986/986 (was 982 -- four tests came back), coverage unchanged at the same 20 errors, both self-host fixed points hold. Spike 2 framed this as 'integer? cannot be fixed on its own'. The sharper statement is that one predicate was answering two different questions: exact? asks about the *representation*, integer? and rational? ask about the *value*, and 2.0 is an integer even though it is inexact. Both were the ctype test 'not a float numtype', which was right for exact?/inexact?/exact-integer? by coincidence and wrong for integer?/rational?. So this separates the questions rather than correcting integer? and patching the fallout. (integer? 2.0) was #f, now #t (integer? 1.7976931348623157e308) was #f, now #t (integer? #\a) was #t, now #f (rational? 9007199254740992.0) was #f, now #t (rational? -4.3) was #f, now #t (inexact? 'foo) was #t, now #f (exact? 2.0), (exact-integer? 32.0) #f before and after The last line is why this is one commit: those were already right and had to stay right. Two bugs beyond the ones Spike 2 named, both from the shared implementation -- integer? answered #t for a character, because wchar has an integer numtype, and inexact? as (not (exact? x)) answered #t for every non-number. - efsrc/builtins.c: exact? and inexact? are C primitives now, so neither meaning has to be simulated in Lisp. integer? on a double is isfinite(d) && d == trunc(d), replacing a d <= DBL_MAXINT round trip -- simpler and correct past 2^53, which is what lets (rational? 1.79e308) answer #t. integer-valued? is the same question as integer? now; kept for lib/aliases.scm's one caller. - (scheme base): exact?/inexact? come from the native shim beside integer? rather than being defined through it; exact-integer? is (and (exact? x) (integer? x)); rational? is 'a number that is exact, or inexact and finite'. - exact-integer-sqrt's guard tightened from integer? to exact-integer?: its body is all div0, and the corrected integer? admits 17.0. - the nine 'check-arg integer?' sites in (scheme list) needed no change. The worry was a widened integer? admitting a fractional index and diverging; it cannot, since (integer? 2.5) is still #f. What it now admits is 2.0, which counts down to 0.0 and terminates. - lib/aliases.scm updated for base mode. Note that file does not currently load at all, for an unrelated pre-existing reason (line 4's top-level-bound?), so those definitions were checked in isolation. check-in: 6a013fd978 user: e tags: tower | |
| 19:50 | split exact from inexact before the accumulate loops; pairwise num_*2 Numerical tower staging step 2, part 3 of 3. fl_add_any and fl_mul_any now decide exactness before building any accumulator, and there are pairwise num_add2 / num_sub2 / num_mul2 for the two-operand case, which is what the VM asks for almost always. All of them route each operand through one shared accum_*_operand, so the two-operand and n-ary paths cannot disagree. fibfp -15.5%, fft -10.7%, everything else inside +-1.4% (interleaved, CPU time, min of 4). Spike 1 attributed ~2% to the *checks* it had added to the shared loops; the real cost was the loops themselves -- every float operation walked the switch that maintains two 64-bit exact accumulators, then folded them into the double at the end. It is also a bug fix, four times over. Maintaining the exact accumulators alongside a float operand meant *their* overflow escaped from operations whose result is an ordinary double: (* 1.5 #int64(0x4000000000000000) #int64(0x4000000000000000)) was "*: integer overflow", now 3.1901e37 (+ 1.5 #uint64(max) #uint64(max)) was "+: integer overflow", now 3.6893e19 (- 0.0 #uint64(max)) was "-: integer overflow", now -1.8447e19 (* -1 #uint64(max)) was 1, now raises (* -1 #uint64(0xe000000000000000)) was #int32(0), now raises (+ #int64(-4294967295) 0) was #int64(-4294967295), now a fixnum (- -0.0 0) was 0.0, now -0.0 The (* -1 #uint64(...)) family came from reconcile_mul_exact doing SMUL(Saccum, (int64_t)Uaccum), a cast that is meaningless once Uaccum passes S64_MAX: it reinterpreted the accumulator as negative, multiplied to a positive, then returned that or truncated it to 32 bits. Negative products now combine magnitudes unsigned and raise above 2^63. The canonicalisation is required rather than cosmetic -- compiler.lsp emits load0/load1 from (eq? x 0) and (eq? x 1). Recorded in the write-up because it is a trap worth not repeating: dropping the fixnum special case so every operand is routed by tag looks like a simplification and breaks correct answers. Positive T_INT64 operands go to the *unsigned* accumulator, Uaccum can then reach 2^63, and (int64_t)Uaccum stops meaning anything -- (* -1 #int64(4611686018427387904) 2) is exactly -2^63 and started raising. Fixnums going to the signed accumulator keeps Uaccum small enough for that cast; it is load-bearing. Verified by diffing a 467588-line matrix of every operation over every numeric representation against the previous binary: all 6133 differences classified, no unexplained ones. Every suite was green before and after each intermediate state, including the one with nine regressions in it, so the suites would not have caught this. test 982/982, both self-host fixed points, coverage unchanged at 20 errors. check-in: f066353b1f user: e tags: tower | |
| 19:50 | Makefile: define HEADERS, so objects actually depend on headers The pattern rule has always read '%.o: %.c $(HEADERS)', but HEADERS was never defined anywhere, so it expanded to nothing and no object depended on any header. efsrc/flisp.o gets its real prerequisites from the explicit rule that lists cvalues.c, print.c, read.c and friends, which is why this went unnoticed -- but eflib/dtypes.h and eflib/utils.h were prerequisites of nothing at all. Editing dtypes.h, which defines numerictype_t and every tag macro, rebuilt nothing that includes it. Coarse on purpose: a header edit now rebuilds every object, which costs a few seconds and is preferable to a silently stale one. check-in: 0e75752d7e user: e tags: tower | |
| 19:18 | fl_neg: negating a uint64 above 2^63 reinterpreted instead of negating Found while scoping num_sub2, which wanted to reuse fl_neg. Its T_UINT64 arm was mk_int64(-(int64_t)v), which reinterprets the bit pattern rather than negating the value: (- #uint64(0xffffffffffffffff)) => #int64(1) (- 0 #uint64(0x8000000000000001)) => #int64(9223372036854775807) Both answers are wrong by about 2^64. Above 2^63 the negation is representable in nothing this runtime has, so it now raises, which is the promote-where-you-can and raise-otherwise rule the rest of arithmetic follows. -(2^63) is still mk_int64(S64_MIN), so unittest.lsp's existing INT_MIN asserts are unaffected. The T_UINT32 arm had the same shape in miniature: for exactly 2^31 it computed -(int32_t)2^31, which is undefined, and happened to land on the right answer. Now stated directly. check-in: bf8bf26ad1 user: e tags: tower | |
| 19:10 | numerictype_t: named ordering predicates, real not-a-number sentinel Numerical tower staging step 2, part 2 of 3. Mechanical and behaviour-identical: both images rebuild byte-for-byte the same apart from the embedded manifest string, and test/test7/selfhost/selfhost-expander/coverage are all unchanged. The point is that numerictype_t can now grow append-only. Scoping this found the write-up undercounts badly, and misses the blocker that actually matters -- corrections are recorded in wiki/DevNotes/numerical-tower.md. There are 14 '>= T_FLOAT' sites in 3 files, not 5 in 2 (eflib/operators.c was missed entirely), and 12 switches over a numtype in 2 files, not ~35 in 6. - is_inexact_numtype / wider_numtype in eflib/dtypes.h replace all 15 ordering comparisons. wider_numtype must keep the signedness-interleaved tag order, not plain width: fl_bitwise_op depends on unsigned counting as wider than signed of the same width. - N_NUMTYPES was T_DOUBLE+1 *and* the value fltype_t::numtype holds for a type that is not a number, so appending a tag would have made a real tag collide with the sentinel: every opaque type would have claimed to be it, and valid_numtype would have refused to allocate it as a cprim. Now T_NOT_A_NUMBER = 0xff, deliberately not an enumerator -- as one it would add a dead arm to all twelve switches and blunt the -Wswitch warning that is the whole safety net. - the two 'T_FLOAT - 1' initialisers are gone. fl_log used one so a later '< T_FLOAT' would read 'exact' with no second argument; it now carries an explicit allexact flag. fl_atan's was dead, with its commented-out consumer. - -Wswitch is now complete, and immediately earned it: changing fl_lognot's and fl_ash's 'int ta' to numerictype_t reported two switches it had been blind to. both_nan's 'default: return 0' would have answered 'not a NaN' for a new inexact tag, silently; it now lists all ten. fl_bitwise_op's three assert(0) arms, which the shipped -DNDEBUG compiled out into a silent 'return NIL', now raise. Also recorded: commit 2 is behaviour-identical yet moves nboyer +2.8%, fft +1.8% and fibfp -1.9% reproducibly across two runs. That is code layout, which benchmarking.md already prices at ~2.8%, and it means the ~2% Spike 1 attributed to the shared accumulate loops cannot be resolved by this suite at all. check-in: e1a91cf490 user: e tags: tower | |
| 18:52 | ab-bench.sh: anchor a bare binary name to the cwd Its own documented invocation -- tools/ab-bench.sh eflisp-r7rs.control eflisp-r7rs -- reported CRASH on every row. A bare name passes the -x test, because that resolves against the cwd, but executing a word with no slash in it goes through PATH instead, where it is not found; every run exited 127. bench sets PATH=$ROOT:$PATH for this reason, ab-bench.sh did not. Only the crash checking stopped it from timing the 127s and reporting a spectacular improvement, which is the second time that check has earned its keep. check-in: 9dc8299b41 user: e tags: tower | |
| 18:52 | fix three arithmetic bugs in the generic (non-fixnum) paths Numerical tower staging step 2, part 1 of 3 (wiki/DevNotes/numerical-tower.md). Landed on their own merits, ahead of the pairwise num_*2 refactor, the same way the * / div0 / ash fixes were: these are wrong answers and undefined behaviour today, independent of whether the tower ever happens. The write-up says +, - and neg 'already checked for fixnum overflow and promoted'. That is true of the VM fast paths only -- fl_add_any had no 64-bit check whatsoever, so (+ #uint64(max) 1) silently answered 0. It now follows the same rule the rest of the runtime does: promote where the result is representable, raise where nothing can hold it. - fl_add_any: every Saccum/Uaccum update goes through SADD/UADD, mirroring the SMUL/UMUL already used by fl_mul_any (the macro block moved up to cover both). Also negpart, which computed -Saccum in the signed domain -- undefined at S64_MIN -- and now negates in the unsigned domain like ash_left does. - fl_idiv2: S64_MIN / -1 is the one quotient that leaves int64 range, and dividing it is undefined rather than merely wrapping. OP_IDIV has guarded the fixnum case since the earlier fix; the generic path answered #int64(-9223372036854775808) and now answers 2^63. - fl_neg: the s == n fixpoint test detects negating fixnum-min, but zero is a fixpoint too, so (div0 0 -1) came back as a heap #ptrdiff(0) that is not eq? to 0. OP_NEG has always had the != 0 guard; fl_neg now does too. This one matters beyond tidiness: compiler.lsp emits load0/load1 from (eq? x 0) and (eq? x 1), so a fixnum-representable value must never exist as a cprim. test 982/982, selfhost fixed point, coverage unchanged at the same 20 errors. check-in: f3f89ac609 user: e tags: tower | |
| 18:04 | reader: R7RS identifier grammar, radix guard, non-decimal floats Numerical tower staging step 1, part 2 of 2 (wiki/DevNotes/numerical-tower.md); Spike 2 steps 1, 2 and 4. #e/#i (step 3) still to come. - the #b/#o/#d/#x guard admits '+' and '.', not only '-', so #d.1, #d+5, #x+1A and #b+101 read instead of reporting "expected argument list" - a decimal point or exponent means decimal radix. R7RS defines <decimal R> only for R = 10, so #b1.1 and #x1.8 are errors; they used to answer the *decimal* reading, silently (#b1.1 => 1.1, not 1.5). Bases 11-14 now reach strtoull in the right base. string->number shares the parser and agrees. - numlike_nonidentifier(), shared by reader and printer, encodes the digit-exclusion rules of <identifier>: 1/2, 1s10, -3/4, 1+2i and .4x are neither numbers nor identifiers, so reading one raises rather than interning a symbol that fails much later as an unbound variable. |1/2| still works. The printer needs the same test or such a symbol would not read back, which the image dump depends on. test 982/982 (#d.1 re-enabled), selfhost and selfhost-expander fixed points, coverage unchanged at the same 20 errors. Both images rebuild byte-identical apart from the manifest string. check-in: fe2326ffc5 user: e tags: tower | |
| 17:56 | rename 1+/1-/1arg-lambda? to add1/sub1/unary-lambda? Numerical tower staging step 1, part 1 of 2 (wiki/DevNotes/numerical-tower.md). R7RS forbids a digit as an identifier's first character, and the reader is about to enforce that. The images store symbols as text and are read back by the very reader being changed, so the rename and the image rebuild must land before the strict reader, not with it. 53 live sites; add1/sub1 were collision-free in the tree. check-in: 3cbb5b4477 user: e tags: tower | |
| 16:11 | start implementation of r7rs numeric tower check-in: 31ab4e0d6f user: e tags: tower | |
| 15:43 | Create new branch named "tower" check-in: c2635b6d80 user: e tags: tower | |
| 15:39 | complete numeric tower spike writeup Leaf check-in: a0f45f93ea user: e tags: trunk | |
| 14:54 | fix a few arithmetic bugs in prepararation for numerical tower check-in: 456931afc8 user: e tags: trunk | |
|
2026-08-09
| ||
| 22:35 | add recursion limit and more improvements to stack-backtrace check-in: 5db486620c user: e tags: trunk | |
| 20:34 | add ctrl-c handling check-in: 06cc223470 user: e tags: trunk | |
| 19:40 | update notes check-in: 4e210ffd2f user: e tags: trunk | |
| 18:13 | add new record printers to clean up stack backtraces; add env and lib inspectors check-in: 28078b3ce6 user: e tags: trunk | |
| 14:46 | optimize environments with a hash table; use uninterned symbols in expander check-in: 5b9165cfd5 user: e tags: trunk | |
|
2026-08-08
| ||
| 22:39 | more benchmarking, negative result this time, newish diassembler bug fix check-in: d25d4aac56 user: e tags: trunk | |
| 22:19 | Add opcodes for ge2, le2, gt2 check-in: 27bda5c1bc user: e tags: trunk | |
| 20:43 | CLAUDE.md and the skill files: catch up with the optimization phase. CLAUDE.md had one outright error: it still said the initial heap is 512 KB per semi-space, which stopped being true when phase 1 raised INITIAL_HHEAP_SIZE_BYTES to 16 MB. Fixed, with a pointer to the measurements behind the choice. Also added the performance commands and the overridable build knobs, and selfhost-test to the test list. skill-r7rs.md claimed 'make efscm.boot' takes about 3 s; it takes about 0.1 s, and did before this work too. Documented the build knobs -- LTO in particular, since it is worth 20% and is the single largest performance factor in the build -- and the overridable heap size, including the detail that it needs a plain integer because an expression with parentheses reaches the compiler unquoted through make. Added a Benchmarks section: the suite is vendored now and works, the quick subset is the thing to use while iterating, 7 of 57 benchmarks cannot run and that set must not grow, and a single run per benchmark has a 3.7% spread, which is larger than most changes worth making. skill-base.md gained a section on bytecode and the peephole optimizer, which is the natural home for it. It covers the thing most likely to confuse someone reading disassembly -- that emit fuses pairs as it goes, so (car (cdr (cdr x))) comes out as cdra0 cadr and (if (not (< a b)) ..) as a single brlt -- the warning that these rules are individually unmeasurable but collectively worth 2.3%, the two rules that must match three instructions because not+brf rewrites the list without going back through emit, the five places that have to agree when adding an opcode (including compute_maxstack, which has no default arm), the fact that opcode position is an image-size lever via print.c's +35 bias, and that renumbering needs tools/xcompile.lsp rather than bootstrap.sh. Every make target, build variable and disassembly example asserted in these files was run against this tree while writing them. Dropped a count of the peephole rules that was both wrong and going to drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: e4dbbe784f user: e tags: trunk | |
| 20:32 | benchmarking.md: revise for consistency now that the phase is finished. The document had been appended to across four phases and several early sections had gone stale or wrong. The noise-floor section still claimed the quick gmean reproduced to 0.51% and that 1% was the decision threshold. That was measured before it was discovered that the metric ran each benchmark once and actually had a 3.7% spread. Replaced with a Measuring section that states plainly which rows of the results table use which metric -- single-run and +/-4% up to the phase 2 block, minimum-of-3 and +/-0.33% from phase 3 on -- why the minimum rather than the mean is the right estimator when the noise is one-sided, and the two ways this phase produced confident wrong answers: a single run inventing a 63% regression that re-measured as neutral, and two-run comparisons inventing both a gain and a regression that were entirely noise. The results table now says which half is which above the header, and the column meanings describe what gmean is currently computed from. How to reproduce was missing everything added after it was written: the quick subset target, -r/REPS, -b quick and -b reuse, the -- that getopts needs before a summary starting with a dash, the eflisp-peep profiling workflow, and the distinction between the dynamic pair counts that fusion candidates come from and the static occurrence counts image size depends on. It also documented a bench-record invocation that no longer parses. Phase 2's closing paragraph recommended the enum reordering as future work; it now records that phase 4 did it and what it was worth. Dropped a trailing paragraph that had been pasted in from chat, which described the reordering as still outstanding and repeated the call/ret discussion already above it. No measurements changed; this is the record catching up with what was learned. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: 11262c7fe4 user: e tags: trunk | |
| 20:20 | Optimization phase 4: renumber the opcodes for image size. efscm.boot -3.6%. Phase 2 turned up the fact that print.c biases every bytecode byte by +35 before printing it into the boot image as a string, so an opcode numbered 0 through 91 costs one character in the image and 92 upward costs four as \Xnn, while slot 57 costs two because it prints as a backslash. Opcode position is therefore a lever on image size with no bearing on semantics, and the numbering had never been chosen with that in mind: 8295 bytes, 4.1% of the image, were going to opcodes sitting above 92, while fifteen cheap slots were held by opcodes that are never emitted at all -- every .l long form plus largc, lvargc, keyargs and cddr. The right metric here is static occurrence counts, not the dynamic counts phase 3 used: image size depends on how often an opcode *appears*, not on how often it runs. Collected by tallying encode-byte-code over a full mkboot2. Everything from OP_LOADT up is now sorted by that count, cheapest slots first, with slot 57 given to a low-count opcode. Indices 0 through 44 are frozen, because builtin_names, builtin_arg_counts and vm_apply_labels are all indexed by those numbers and isbuiltin is uintval(x) <= OP_ASET. efscm.boot 203402 -> 195998, flisp.boot 41752 -> 39772, and the quick subset improves from 3.525/3.546 to 3.505/3.495 -- slightly faster as well as smaller, which is what less bytecode to fetch should do. Predicted 8148 bytes saved against 7404 actual; the histogram comes from what mkboot2 compiles, which is not exactly what gets dumped. With this the image ends the whole optimization phase 3.3% smaller than it started, rather than the few percent larger that was budgeted for. Renumbering cannot go through bootstrap.sh, because mkboot1 loads compiler.lsp and then executes what it just compiled, which puts new-numbering bytecode on the old VM. tools/xcompile.lsp does it instead: it compiles the sources under the old table, swaps the new table in as data, and from that point only prints the bytecode it produces, never calls it. Two traps in there cost most of the time and are both silent, so they are documented in the tool and in benchmarking.md. The #fn constructor computes maxstack by walking bytecode with the host VM's opcode semantics, so the old binary mis-walks new-numbering bytecode and stores garbage, frequently negative -- and since the reader recovers the +35 bias from the top byte of the maxstack field, a negative maxstack makes that byte 0xff, the reader unbiases by 255, and the whole image is mangled, loads without error, and dies later somewhere unrelated. And the reordered compiler.lsp source has to be the one compiled into the image: giving the new table only to the cross-compiler makes it emit new numbers while the image it produces still carries the old Instructions, so everything compiled at runtime afterwards is in the old numbering. Verified: make test, test7 at 983/983, r7rs-coverage-test at 21, selfhost-test, selfhost-expander-test, checklib --exports at 20, and the phase 3 fusion correctness checks. Both self-hosting fixed points reproduce byte-identical images, bootstrap.sh works normally again now that the numbering is self-consistent, and rebuilding from scratch reproduces efscm.boot at 195998. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: 4b2f05d53d user: e tags: trunk | |
| 19:30 | Optimization phase 3: ten fused opcodes from a measured execution profile, 2% more. Chosen from dynamic opcode-pair counts collected with make eflisp-peep over seven benchmarks, not from intuition. The execution profile bears little resemblance to the compile-time one: the top fusable pairs are loada0+call at 919M, loada1a0+lt2 at 895M, lt2+brt at 815M, loada+ret at 681M, loadc0+car at 478M, pair?+brt at 265M and numeq2+brf at 156M. Added, all appended to the end of the enum so no existing opcode is renumbered and the numbering baked into both boot images stays valid: brlt and brnlt with their .l forms for lt2 followed by a branch, brnumne for numeq2+brf, brp for pair? followed by a branch, and carc0 and carc1 for loadc0/loadc1 followed by car -- the closure-variable counterpart of cara0/cara1, which phase 2 identified as the most valuable of the existing rules. Two of these need a three-instruction pattern rather than a pair. The not+brf to brt rule rewrites the instruction list directly instead of going back through emit, so a lt2+brt rule never sees the brt that rule produces. (if (not (< a b)) ...) has to be matched as lt2+not+brf, and pair? the same way. That is why the 815M pair looked unfusable at first. Measured with configurations alternated to cancel machine drift: no new rules 3.624 and 3.612, the three branch fusions alone 3.569 and 3.580, all ten 3.525 and 3.546 -- 2.3% on the subset. On tak alone, best of four over three rounds, 7.517/7.515/7.500 without against 6.981/6.962/6.949 with, a 7.1% gain, which is what a benchmark dominated by one fused pattern should show. Full suite 9.124 to 8.944. efscm.boot grows 692 bytes, 0.34%, against a budget of a few percent. tools/bench-record.sh gains -r/REPS and now takes the minimum of REPS runs per benchmark before the geometric mean. This matters: the old single-run subset metric had a 3.7% spread, and measured against it the branch fusions first appeared to be worth +1.6% and the carc0/carc1/brp batch appeared to cost 1.1%. Both were noise. The fastest run is the one least contaminated by scheduler noise; averaging runs averages the noise back in. The same configuration now reproduces to 0.33% and all three configurations separate cleanly. Every subset number in benchmarking.md from before this change is a single-run number. Also recorded there: an ad-hoc regex used to toggle a rule off left one extra parenthesis, closing the cond clause early and turning the => continuation into a clause of its own, so emit returned the assq pair instead of rewriting the instruction. The result was a compiler that silently produced wrong bytecode -- it passed the disassembly probe and the base unit tests, and crashed only later in mkboot2. Variants are now gated on make test and a successful mkboot2, not just on the probe, since the probe only proves an opcode is absent and says nothing about whether the compiler is still correct. make test, test7 at 983/983, r7rs-coverage-test at 21, selfhost-test, selfhost-expander-test and checklib --exports at 20 all pass, both self-hosting fixed points still reproduce byte-identical images, and the seven benchmarks eflisp cannot run are unchanged. Fixnum and float paths of all three comparison fusions are spot-checked directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: 77419cb867 user: e tags: trunk | |
| 17:59 | benchmarking.md: phase 2 -- back out each 2017 peephole rule, keep all nine. Nine rules disabled one at a time in emit, leaving the Instructions entry, the opcodes.h enum slot, the VM_LABELS entry and the OP() body in place so the opcode merely stops being emitted. No renumbering: bootstrap.sh runs the old binary against the old image and both images have the current numbering baked in. Each variant rebuilt flisp.boot and efscm.boot and had to prove the rule had stopped firing, by disassembling a probe expression, before its timing was believed. Taken one at a time, eight of the nine land inside the 1.2% control spread and four look faintly faster when removed, which reads as eight rules of dead weight. Removing all eight together costs 2.3%, reproducibly, at 3.753 and 3.750 against a control mean of 3.670. Individually unmeasurable effects still add up, so the per-rule table is not a decision procedure and the plan's keep-if-it-beats-1% rule would have deleted eight rules that collectively pay. All nine are kept and no code changes. That also settles the dispatch-overhead question: if the added opcodes were costing the interpreter anything at dispatch, retiring eight rules' worth of them would have helped. It hurt, so there is nothing there to recover and the planned retired-opcode experiment is unnecessary. The image-size column turned out not to be monotonic in bytecode length -- removing the dup+brf fusion shrinks efscm.boot by 1395 bytes while removing null?+brf grows it by 30. print.c biases every bytecode byte by +35 before printing it as a string, so opcodes 0 through 91 cost one character and 92 upward cost a multi-character escape. Every peephole opcode was appended to the end of the enum and is therefore in the expensive range: one dbrf costs more image bytes than dup plus brf, despite being one byte less of bytecode. Enum position is thus a lever on image size independent of semantics, and reordering so the hot fused opcodes sit below 92 and the rarely-executed .l forms above it would shrink the image for free. Worth doing before phase 3 spends image budget on new opcodes. The probe machinery earned its keep twice: the first sweep reported every rule as still firing because the probe's marker line printed the opcode name and the check matched itself, and the combined variant initially failed to build because removing the not+brf clauses collapses the eq? clause onto the cond line, which stopped the next pattern from matching. Both would have produced confident measurements of nothing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: bbd5035111 user: e tags: trunk | |
| 17:18 | benchmarking.md: record 676ab3fd56 as the check-in the phase 1 changes landed in. The table rows cite the baseline they were measured against, per the plan for this phase; the id of the commit that carries the kept set could only be filled in afterwards. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: 76334eadb3 user: e tags: trunk | |
| 17:17 | Optimization phase 1: build and VM level, 25.4% faster with no image growth. Restores a working benchmark harness first, because nothing here could be decided without one. The r7rs-benchmarks make target pointed at ../r7rs-benchmarks, a directory that does not exist, and the in-tree copy of ecraven's suite had no eflisp port and was not tracked by fossil at all -- so it is vendored here, 200-odd files, which is what makes every number in wiki/DevNotes/benchmarking.md reproducible from a checkout. The eflisp port is small: the r7rs layer already provides everything the benchmarks need except this-scheme-implementation-name, so src/Eflisp-postlude.scm supplies just that. It is a postlude and not a prelude because an -s script starts with no bindings at all, so nothing may precede the benchmark's own import. bench gains an eflisp system driving eflisp-r7rs, the binary with the image compiled in, so there is no boot file to locate from the benchmark directory. The ignore-glob entry for results.* was narrowed to results.Eflisp*: it was hiding upstream's reference results for 27 other Schemes, which are part of the suite, not our output. all.csv is generated by `make csv` and is now ignored. tools/bench-record.sh measures one configuration and prints its table row: image size, image build, self-hosted rebuild, conformance, and the geometric mean over benchmark completions. Geometric because the times span two orders of magnitude and an arithmetic mean would only report whichever benchmark is slowest. The three short timings are best-of-five -- at 0.1 to 0.3 s a single sample is mostly process startup, and the noise is one-sided. QUICK_BENCHMARKS is a twelve- benchmark subset, about 80 s against the full suite's twenty minutes, chosen from the baseline run rather than guessed. Three back-to-back runs of the unchanged tree put the noise floor at 1%, so 1% is the threshold for believing anything. make eflisp-peep builds the interpreter with efsrc/peepcount.c active, which counts every consecutive opcode pair executed. That code and the peep.counts builtin were already present but nothing built them, and they could not have worked: efsrc/flisp.c unconditionally defined eFLISP_PEEP_COUNT to 0 after the command line, silently winning over -DeFLISP_PEEP_COUNT=1, and lib/peep-analyze.lsp called sqrt, which base mode does not have. Both fixed, plus an out-of-range index when fewer pairs exist than the requested count. The four optimizations, measured one at a time against the quick subset: -flto, 20.6%, the largest single win. The VM calls constantly into table.c, equalhash.c, htable.c and ios.c, which are separate translation units, so before this none of those calls could inline. The existing idiom of #include-ing cvalues.c, types.c, print.c, read.c and equal.c into flisp.c was buying exactly this for those five files; LTO extends it to the rest. Confirmed over three runs (3.869, 3.926, 3.900) against three controls (4.881, 4.918, 4.928). 16 MB initial heap, 7.5%. 512K was small enough that gc()'s doubling policy grew past it during a run anyway, paying for two semispace reallocs and a bitvector resize to get there -- so 1 MB is both faster and 18% smaller in max RSS than the old default, a strict improvement with no trade-off. Past 1 MB it is memory for speed; 16 MB costs 36.8 MB RSS against 6.1 MB. The measurements are in the comment on INITIAL_HHEAP_SIZE_BYTES, which is now overridable. apply_cl's locals automatic instead of static, 2.8%. These six were static from femtolisp onwards, which looks deliberate, since apply_cl is re-entered and statics would then be shared state. Nothing depends on that, and sharing them is what would be wrong: the running closure is read from Stack[bp-1] rather than from func precisely so it survives a call; func's every use is a write followed by reads in the same block; accum, pv, e and c are scratch inside one opcode body; the state that genuinely must be per-invocation -- ip, bp, n, s, t -- is automatic already, and OP_FOR calls apply_cl in a loop while depending on s, t and n being frame-private, so statics there would break it. apply_cl contains no setjmp, FL_TRY living in do_trycatch, so the indeterminate-after-longjmp rule that would force static or volatile does not apply; and gc() does not scan them, so static never made them GC roots either. The reasoning is in the comment so it need not be worked out again. -DNDEBUG, 1.0%. Nothing defined NDEBUG anywhere in the tree, so -O3 shipped every assertion, including one on each entry to apply_cl and more in its LOADV, LOADG, SETG and AREF paths. DISABLE_ASSERTIONS existed for exactly this and was never set. Rejected with numbers rather than opinion: switch dispatch is 2.4% slower than computed goto, so USE_COMPUTED_GOTO is confirmed correct on M4 and clang 21, and DISPATCH is now overridable so it can be re-checked; -Os and -mcpu=apple-m4 are both inside noise; -O2 is 5.6% on its own but ties -O3 once LTO is on, so -O3 stays. efscm.boot is unchanged at 202710 bytes -- none of this touches bytecode, so the image-size budget is untouched. Full suite 12.225 to 9.124, 25.4% faster, with the same seven non-completions as the baseline. Image build, self-hosted rebuild and conformance time improved 15 to 21%, so the compiler got faster along with the synthetics. make test, test7 at 983/983, r7rs-coverage-test at 21, selfhost-test, selfhost-expander-test and checklib --exports at 20 all pass; both self-hosting fixed points still reproduce byte-identical images, which is the real check on the apply_cl change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> check-in: 676ab3fd56 user: e tags: trunk | |
| 15:22 | Add .claude/skill-r7rs.md and tools/checklib.lsp. skill-r7rs.md is the r7rs-mode counterpart to skill-base.md: the library layout, how to write and modify a library, the debugging tools, and the self-hosting build chain. Every command in it was run against this tree while writing it. tools/checklib.lsp is the part that is not prose. It does the check that matters and that nothing else does: ./eflisp -s tools/checklib.lsp lib7/|init|/scheme/base.sld expands a library without evaluating it and reports identifiers it references but never defines. A library that builds is not a library that works -- an unimported name is silently renamed into the library's own namespace, becomes a prefixed global with no definition, and fails at the first call, while make efscm.boot succeeds. That is how map-int slipped through (scheme list) and how three names slipped through (r7expander library). Verified it catches a real one by deleting *print-readably* from (scheme write)'s import list and watching it name exactly that. ./eflisp -s tools/checklib.lsp --exports audits every loaded library for exports bound to nothing. It found one on a tree I had already audited by hand: (eflisp native) exported disassemble, which mkboot2 excludes from the image, so it resolved to nothing. (eflisp debug) defines its own, which is the one that works. Dropped from the shim list with a comment, same as the eight names dropped earlier. Clean tree now reports 20 unbound exports, all syntactic literals and compiler special forms, which have no global value by design. The skill's Gotchas are the things that cost time this session rather than anything derivable from the source: that an -s script starts with no bindings at all, that toplevel names are mangled so trace needs 'r7rs.repl:f, that defining a name a shim exports by identity overwrites the base global (naming a function double clobbers the C constructor), that bound? on a prefixed name is not an existence test, and that two copies of the expander fail later rather than at the boundary. CLAUDE.md now references both skill files. wiki/DevNotes/benchmarking.md has uncommitted edits that are not mine and is deliberately left out. make test and test7 pass; efscm.boot is rebuilt for the native.lsp change. check-in: 37fde321ab user: e tags: trunk | |
| 15:10 | Merge selfhost into trunk: R7RS self-hosting. Integrates the selfhost branch, which also carries the r7rs branch (its tip f15b79b961 is an ancestor), so both are closed by this merge. trunk was already fully contained in selfhost, so this is a clean integrate -- no conflicts. The R7RS runtime now lives inside the library system: - lib/scheme and lib/srfi are gone. Each (scheme x) defines its own procedures as scheme.x: globals instead of (r7expander native) re-exporting base globals by identity, so every definition has one site. Base mode loses (import (scheme base)); the expander is written in eflisp, not r7rs. - (eflisp syntax) holds the derived syntax that both (scheme base) and the expander need, sitting below both so the expander's own source does not have to import (scheme base) and close a cycle. - mkimage.lsp builds the image from named library specs plus a reachability walk over function:vals, rather than dumping every bound symbol: 552 bindings where the old dumper had 569. - mkboot3.lsp rebuilds the library universe from inside a running image and reaches a fixed point; mkboot4.lsp additionally moves the expander onto lib7/r7expander/*.sld. make selfhost-test and make selfhost-expander-test check both. Bugs fixed along the way, each committed separately on the branch: the base-mode case macro's else clause; catch in (eflisp exceptions), whose expansion needs caddr; floor//floor-quotient/floor-remainder for exact division, so (modulo -4 2) is 0; and the missing (scheme write) import in (eflisp debug) that had broken trace. Verified on the merged tree, with lib/scheme and lib/srfi removed from disk as well as from the manifest: efscm.boot rebuilds byte-identically from the merged sources, make test, etest, test7 and etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test at 20 errors, and both self-hosting fixed points hold. A fresh checkout of the branch was built and tested the same way before the merge. check-in: ce36dc03a6 user: e tags: trunk | |
| 15:07 | Track tests/hygiene.scm and wiki/DevNotes/benchmarking.md. Both were sitting untracked in the working directory and would have been absent from any other checkout. Found while checking the branch over before merging -- the same class of omission as (eflisp syntax), which was described by a commit that did not contain it. tests/hygiene.scm is a 15-case acceptance battery for nested macros and free-identifier identity; it passes 15/15. Note it has no (import ...) of its own, so it cannot run as an -s script -- ./eflisp -s tests/hygiene.scm dies with r7rs.repl:define has no value. It has to be loaded from an environment that has already imported (scheme base) and (scheme write). Left as found; adding the import line would make it a candidate for a Makefile target. The .fossil-settings/ignore-glob change is not mine -- it was already in the working directory, adding the benchmark output globs that go with benchmarking.md. Committed here so the tree is clean for the merge rather than left dangling. Closed-Leaf check-in: 0a585c50fb user: e tags: selfhost | |
| 14:57 | DevNotes-eflisp-r7rs: bring up to date with this session. Two TODOs are done and are removed rather than annotated: "remove redundancies" and "self host?". The latter had guessed at the mechanism -- "perhaps a new version of make-system-image should be passed a list of library specs to dump instead of dumping all bound symbols" -- which is what mkimage.lsp turned out to be. History items 35-40 cover the chi-body* refactor, the r7e split, retiring the lib/scheme and lib/srfi veneers, mkimage.lsp, mkboot3 and mkboot4. Kept to a few lines each; the notes worth having are the ones that cost time to find -- that *builtins* has to be named because the C core reads it and its absence segfaults rather than raising, and that what makes the expander switchover hard is state rather than record types. Also updated: (eflisp syntax) added to the Grokking section, since it is not obvious why a library exists purely to sit below (scheme base); the string mutators noted as no longer exported; the porting-notes list of macros awaiting translation, all three now dealt with; the coverage block regenerated (the inexact error is gone, current-second is new); and the make targets for the two fixed-point checks. Dropped the "perhaps not TODO" about unchecked library import/export, which the Grokking section already records as fixed and which I saw give a clean message this session: "Library (only (r7expander native) for-each) doesn't export for-each". The benchmark block is left alone -- those results are from June and were not re-run. Net +33 lines for six phases of work. check-in: 62ea8d4315 user: e tags: selfhost | |
| 14:15 | Retire (eflisp syntactic-closure); move letrec to (example letrec). lib7/eflisp/syntactic-closure.sld was a hand transcription of the expander from 2026-03-25, superseded by lib7/r7expander/syntactic-closure.sld, which (include)s r7e/syntactic-closure.lsp and so cannot drift from the copy base mode loads. The old one had drifted: - exports expand, from before the r7expand rename - no keyword self-evaluation case, so (file name :read) would expand to (file name scheme.file::read) - still spelled in r7rs -- string->symbol, string-append, symbol->string, number->string, vector-map -- which is what made it need (scheme base) and reintroduce the import cycle that (eflisp syntax) exists to break - generate-name still a closure over n, so no reset-generated-names! and no fixed point - and er-macro-transformer was broken: (proc form rename compare) env where the shared source has (r7expand (proc form rename compare) env), so it called the transformer for effect and returned the environment, discarding the expansion Nothing imported it and it is in no boot image. expand-1, the one thing it had that the shared source does not, is also dead -- the chi-body* refactor replaced that approach and the name appears nowhere else. lib7/eflisp/letrec.sld was written to test it, but already imports (r7expander syntactic-closure) and works unchanged: mutual recursion and letrec* both check out. Moved to lib7/example/letrec.sld as (example letrec), with a header explaining what it demonstrates -- rename for identifiers closed in the macro's own environment, compare for asking whether an init is literally a lambda so the recursive case can use fix! rather than set!. Those are the two things syntax-rules cannot express, which is the point of having the example. (eflisp uses) still works; both uses and report were exercised. efscm.boot is unchanged, which confirms none of this was on the boot path. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test identical at 20 errors, selfhost-test still reaches its fixed point. check-in: b69b625420 user: e tags: selfhost | |
| 13:54 | Move the expander's own code onto lib7/r7expander/*.sld. New mkboot4.lsp switches the expander over: afterwards the code running the image is r7expander.syntactic-closure:* and r7expander.library:*, compiled through the library system by the previous generation, rather than base globals compiled straight from the r7e sources. Proof, from inside the switched image: assq-environment's vals vector holds r7expander.syntactic-closure:environment-frame, :enclosing-environment, :toplevel-environment? and friends. The switched image passes chibi 983/983, r7rs-coverage-test at the same 20 errors, and all 59 probe cases, and it rebuilds itself to a fixed point. make selfhost-expander-test does the whole thing. FIRST: lib7/|init|/eflisp/syntax.sld was never actually committed. The fossil add in the previous commit failed and I had suppressed its output, so that commit describes a library it does not contain and does not build. It is added here. The blocker was never really record types. It is that the expander has state, and each copy of the code has its own: its own current-meta-environment and current-toplevel-environment parameters, its own library-table, its own %name.<n> counter, its own <environment> record type. Mixing copies does not fail at the boundary -- it fails later, when a transformer from one copy reads a parameter the other copy set, gets #f, and expands against it. Every symptom traced back to that: - (scheme base) taking er-macro-transformer from the new library while the old one still drove expansion. Hence stage 1 compiles the libraries with the shims still resolving, and library.sld loads before syntactic-closure.sld, whose real library would otherwise shadow the shim it needs. - reloading the expander libraries during the rebuild, which made a third copy while the base names still referred to the second. Hence the shims stay: they are identity bindings onto the base names, which now hold the library-compiled closures, and that is what guarantees one copy. - features returning #f for r7rs, because init-libraries! set! the base feature-list while features read the library's. Fixed properly: the list gets an add-feature! operation, which is exported and so adopted. - the image growing 100k a generation, because mkboot3 cleared the base library-table while make-library appended to the library's. - and %name.<n> counters diverging, because reset-generated-names! was not exported and so never adopted. Adoption itself has to be one pass: collect every (base name, library value) pair with the old code still live, then assign. One at a time hands the new environment-frame an old-typed record halfway through the loop doing the repointing. mkimage.lsp now computes a library's prefix from its spec instead of asking its environment's renamer. environment-renamer is a record accessor, and after the switchover the environments are the new record type; the spec is just data. Not wired into the default build. mkboot2 still produces the image the Makefile installs, and mkboot4 is a separate target -- the switched image is 229672 bytes against 202524, because (r7expander library) imports (eflisp compiler) and pulls the compiler in as library globals. Whether that is worth it is a judgement about what the image should contain, not something to decide by making it the default. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test identical at 20 errors, 59 probe cases identical, selfhost-test and selfhost-expander-test both reach their fixed points. check-in: 1100b05e2d user: e tags: selfhost | |
| 13:30 | (scheme base) imports its derived syntax from (eflisp syntax) and re-exports it. The seven macros (eflisp syntax) was created to hold -- cond, case, when, unless, define-record-type, let-values and parameterize -- were still defined a second time in (scheme base). They are now defined once, in (eflisp syntax), imported here and re-exported, so (scheme base)'s interface is unchanged and each macro has a single definition site. That removes the duplication the previous commit deliberately took on. It also means the expander and (scheme base) now share the same macro bindings rather than two structurally identical copies, which matters for syntax-rules literals: cond's else is the same identifier that (scheme base) re-exports, so a use of else expanded against either library matches. cond-expand, guard and guard-aux stay here -- they use else and => as literals but are not part of what the expander needs -- as do let, let*, letrec, letrec*, let*-values, do, define-values and the procedures. make test, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test identical at 20 errors, 59 probe cases identical, no unresolved scheme.base: references, selfhost-test still reaches its fixed point. check-in: 41c9a14135 user: e tags: selfhost | |
| 13:23 | Add (eflisp syntax) below (scheme base) and break the expander import cycle. The cycle phase 5 stopped at: (scheme base) imports unwrap-syntax, identifier? and er-macro-transformer from (r7expander syntactic-closure), while the real syntactic-closure.sld imported (scheme base) for cond, case, define-record-type, parameterize, let-values, when and unless. The shim hid it only because identity bindings have no body to expand. New (eflisp syntax) holds exactly that derived syntax and sits below both. It imports nothing but (r7expander builtin) for the core forms, (eflisp exceptions) for unwind-protect, and the native shims for primitives -- which is why er-macro-transformer, identifier? and unwrap-syntax are now also exported by (eflisp native). Taking them from the shim rather than from (r7expander syntactic-closure) is what keeps this library below the expander instead of beside it. (r7expander syntactic-closure) now imports (eflisp syntax) instead of (scheme base), and expands and evaluates with nothing unresolved. The cycle is gone. Also added to (eflisp native): for-each, which moved to (scheme base) in phase 3 and so was no longer reachable from (r7expander native), and string.join and with-input-file from the earlier commit. Not finished. Loading (r7expander library) still fails with "environment-frame: wrong record type" -- that is blocker 1, the record-type identity problem, which is solved in principle (collect the adoption pairs with the old code live, assign in one pass, rebuild the universe immediately) but is not yet wired into a bootstrap script. The two blockers were independent; this commit removes the structural one and leaves the mechanical one. Two pieces of debt this creates, both deliberate: (eflisp syntax) and (scheme base) now define the same seven macros, and the intended end state is that (scheme base) imports them from here and re-exports so each exists once. And (eflisp syntax) is preloaded in init-libraries!, so the image carries both copies. make test, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test unchanged at 20 errors, selfhost-test still reaches its fixed point. check-in: f9e9055a97 user: e tags: selfhost | |
| 13:13 | Make lib7/r7expander/*.sld actually loadable, and prove the switchover works. The two expander libraries have been inert since phase 2b -- written but never expanded, never evaluated, never imported. Exercising them turned up three unresolved references in (r7expander library), all in the class the static free-variable check exists to find: names the base-mode expander gets as base globals and that no import brings into the library. with-bindings -- (eflisp exceptions), now imported string.join -- a system.lsp function, added to (eflisp native) with-input-file -- r7e/prelude.lsp's reader helper, added to (eflisp native) (r7expander syntactic-closure) was already clean. Both libraries now expand and evaluate, and the static check reports nothing unresolved. That was enough to test the switchover itself, which is the step phase 5 left open. Two blockers were named there; the first turns out to be mechanical and the second is not: 1. Record-type identity. The library's <environment> is tagged r7expander.syntactic-closure:<environment>, so the new environment-frame rejects every environment the running image is made of. The fix is to collect all the (base name, library value) pairs with the old code still live and assign them in one pass -- flipping them one at a time hands the new accessor an old record halfway through -- and then immediately rebuild the universe so nothing old-typed survives. Done that way it works: the library-compiled expander installs the shims, installs (r7expander builtin), and rebuilds all 20 libraries. The rebuilt universe lands in r7expander.library:library-table, not the base one, which is correct and is what mkimage.lsp would have to be taught. 2. An import cycle that is structural, not wiring. (scheme base) imports unwrap-syntax, identifier? and er-macro-transformer from (r7expander syntactic-closure), while the real syntactic-closure.sld imports (scheme base) for define, let, cond, define-record-type and parameterize. The shim breaks the cycle precisely because it is identity bindings onto base globals and so has no body to expand. Replacing the shim with the real library reintroduces the cycle, and no amount of import juggling removes it -- it needs a decision about how the expander gets its own syntax, which is a design question rather than a missing line. So the expander's code is not moved yet. What this commit changes is that the libraries are now correct and loadable rather than untested source, and the path is mapped: adoption is solved, the cycle is what is left. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test identical at 20 errors, 59 probe cases identical, selfhost-test still reaches its fixed point. check-in: 12d33c6d6c user: e tags: selfhost | |
| 13:03 | Self-hosting phase 5: the image rebuilds itself and reaches a fixed point. New mkboot3.lsp runs under -s and rebuilds the whole library universe from the .sld sources, expanded and compiled by the expander and compiler already in the image, then dumps with the phase 4 dumper. Generation n builds generation n+1. gen2, gen3 and gen4 come out byte-identical; make selfhost-test checks it. Three things had to become callable rather than load-time side effects, because mkboot3 runs inside an image where the r7e sources are not available to re-load: - r7e/native.lsp's four shim libraries are now *native-libraries*, a list of (spec . names), installed by install-native-libraries!. This also collapses four near-identical copies of install-native! into one. - r7e/builtin.lsp's block becomes install-builtin-library!. Its body is unchanged and left at its original indentation, so the diff is the head and the tail. - r7e/main.lsp's init-library sequence becomes init-libraries!. Each of those was verified to produce a byte-identical image before anything else changed, so they are refactors and nothing else. Four things kept the image from settling, all of them state that records how much work the build happened to do rather than what the image is: - repl-environment. mkboot3 runs as an -s script, so its own imports land in that frame -- and library-import copies every name it brings in, which is 42k of (scheme base) and (eflisp native) per generation, accumulating without bound. mkboot3 hands the next generation a fresh one, which also keeps its own rebuild! out of the image. - feature-list grew by one r7rs per rebuild. init-libraries! now checks before consing. - generate-name's %name.<n> counter. It was a variable closed over by generate-name, so nothing could reset it; it is now a global with reset-generated-names!, which mkboot2 and mkboot3 both call once expansion is over. - Tables. A table prints in hash order, and reading one back inserts its entries in that order, which lays them out differently, which prints differently again -- so Instructions never settled. The dumper now rebuilds a table with its keys inserted in sorted order, making the printed form a function of the key set alone. Only tables that are directly a global's value are canonicalised. *search-path* joins the never-dump list: __init_globals rebuilds it from *install-dir*, so dumping it bakes in the build machine's path. gen1 -- the base-mode mkboot2 build -- still differs from gen2, at one byte: a *print-circle* label reads #99= where gen2 has #100=, because the two builds produce slightly different sharing graphs. Same size, same bindings, same behaviour. The fixed point that matters is gen2 == gen3, and that holds. What is NOT yet self-hosted: the expander's own code. r7expand, expand-library and the rest are still base globals carried forward from the mkboot2 build rather than r7expander.*: globals compiled through the library system, and lib7/r7expander/*.sld stays inert. Rebuilding the library universe is what makes that step testable, not a substitute for it. Verified on both images -- the make-built one and the self-hosted one: chibi 983/983, r7rs-coverage-test identical at 20 errors, 59 probe cases identical. make test, etest, etest7 pass, flisp.boot untouched. check-in: e244405159 user: e tags: selfhost | |
| 12:42 | Self-hosting phase 4: build the image from library specs, not from every bound symbol. New mkimage.lsp defines make-library-image, which takes a list of library specs, a list of base globals, and an exclude list, and dumps the reachable closure of those instead of "every symbol that happens to be bound and is not a constant, a builtin alias or an iostream". Reachability is over globals. loadg/setg intern every global a compiled function references into its vals vector, so scanning (function:vals f) for symbols finds its references -- along with its quoted constants, which over-approximates. That is the safe direction: a symbol that is really a datum and also names a bound global just dumps a binding nothing looks up, whereas missing one fails at image load or at the first call. The walk also goes through data, because the library system keeps its state in records: library-table maps specs to <library-object>s holding an <environment> whose frame maps identifiers to mangled global names, and expander records whose transformers are closures. The visited set is a table even though its keys are compared with equal?. That is sound here: two records that are equal? have equal? fields, so they reach the same set of symbols, and conflating them cannot lose one. Checked that equal? and the hash behind table survive cyclic records and cyclic pairs before relying on it. Two things the walk structurally cannot see, so they are named: __start -- flmain.c applies it after the image loads *builtins* -- flisp.c:847, the table apply_cl indexes to get a closure wrapper when a builtin is applied indirectly *builtins* is the one that made the case for validating by running rather than by diffing binding sets. No Lisp function references it, so nothing puts it in a vals vector; leaving it out does not raise unbound-error, because vector_elt reads straight through the unbound value. The image loaded, ran hello-world, and segfaulted the moment a test applied a builtin indirectly. Also kept out unconditionally: the globals the C core or __init_globals sets at startup. Dumping those writes the build machine's values over the running ones -- *install-dir* would bake in the path the image was built at. Result against the old dumper: 552 bindings where mkboot2 had 569, a strict subset -- nothing was added. Dropped are array?, compile, confirm-bound, confirm-syntax, div, init-library, io.readlines, lambda-bind?, load-process, mod, printd, quote-value, read-all, read-all-of, revappend, start-repl and vinfo:sym: the loader and the expander's own helpers, which built the image and have no business in it. efscm.boot 192354 -> 190425. mkboot2.lsp now names (map car library-table) rather than a written-out list, so r7e/main.lsp's init-library calls stay the single source of truth for what is preloaded. Selecting a subset does work and does shrink the image -- eight libraries give 401 bindings and 127k, and hello-world runs on it -- but the |init| libraries cannot be thinned out that way, because they live under lib7/|init|/ and load-library-from-spec searches lib7/, so one left out of the image is not found on disk either. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test byte-identical at 20 errors, 59 probe cases identical, flisp.boot untouched. check-in: fd32f1f32d user: e tags: selfhost | |
| 12:28 | Drop unimplemented names from library export lists. Audited every export of every library -- 804 of them -- by resolving each through its library environment and testing the result with bound?. The audit script skips syntax exports, whose bindings are expander records rather than globals. 33 exports had no binding. Most were legitimate: - Syntactic literals with no value by design: else, =>, _, ..., unquote, unquote-splicing and cond-expand's library. r7rs requires (scheme base) to export these; they exist to be matched as syntax-rules literals. - Compiler special forms, which the compiler handles directly and which therefore have no global value: fix!, for, prog1, return, trycatch, while. Each verified working. Eight were genuinely unimplemented and are now dropped, with a comment at each site saying so: (scheme base) and (r7expander native) numerator denominator rationalize -- eflisp's tower is fixnums and doubles with no rationals, so there is nothing for these to return string-set! string-copy! string-fill! -- eflisp strings are immutable (eflisp native) proper-list? -- was srfi-1's and moved with it into (scheme list), which exports it; no base global of that name survives string.trim -- system.lsp's definition is commented out at line 721 and there is no C builtin install-native! binds a keyword to itself without checking that anything is bound, which is how these survived expansion. Calling one now reports r7rs.repl:numerator rather than a bare unbound numerator, so the diagnostic at least names the namespace the reference was renamed into. It is still a call-time error, not an expansion-time one -- the expander renames free identifiers into the toplevel namespace rather than rejecting them. Separately, (eflisp debug) exported write and resolved it to eflisp.debug:write, because its (import (scheme write)) was commented out. That is a missing import rather than an unimplemented procedure -- the library's own trace calls write -- so it is fixed by restoring the import rather than by dropping the export. Remaining unbound exports after this: 18, all syntactic literals and special forms. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test byte-identical at 20 errors -- ecraven guards each probe, so a now-unbound identifier still reports as one error per probe -- 59 probe cases identical. efscm.boot 193450 -> 192354. check-in: f80c9d48a2 user: e tags: selfhost | |
| 12:18 | Fix floor/, floor-quotient and floor-remainder for exact division. The implementation inherited from lib/scheme/base.scm computed the floor quotient as (if (< x 0) (1- x) x) over the truncating quotient, which decrements whenever the truncating quotient is negative -- whether or not there was a remainder to justify it, and never when the quotient rounds to zero. So: (floor/ -4 2) => -3 and 2 should be -2 and 0 (floor-quotient -4 2) => -3 should be -2 (modulo -4 2) => 2 should be 0 (modulo -13 4) => 3 correct, by luck (floor/ -1 2) => 0 and -1 should be -1 and 1 The defining identity n1 = n2*q + r held throughout -- the remainder was computed from the wrong quotient, so both were wrong together -- but the r7rs requirement that floor-remainder take the sign of the divisor did not. modulo is an export rename of floor-remainder, so it was wrong wherever floor-remainder was. Floor division differs from truncating division in exactly one case: the division is inexact and the operands have opposite signs. Then the quotient is one lower and the remainder is shifted by the divisor. (* r n2) is negative precisely in that case, which is what the new floor-adjust? predicate tests. Found while relocating this code into the library and deliberately left alone through the six relocation commits, so that the move stayed a pure motion diff and this fix could be read on its own. Checked against the r7rs 6.2.6 identities over a -12..12 by -6..6 sweep, both the reconstruction n1 = n2*q + r and the sign rule, plus the worked examples from the report. chibi 983/983 in both modes, r7rs-coverage-test unchanged at 20 errors -- neither suite covered this. check-in: 27fe250680 user: e tags: selfhost | |
| 12:16 | Self-hosting phase 3 complete: delete lib/scheme and lib/srfi. Last slice. The (scheme base) veneer is gone, and with it the whole lib/scheme and lib/srfi tree. What was left in lib/scheme/base.scm after the previous five slices was display, three includes, and a handful of base-mode shims: - display moves to (scheme write). It lived in the base veneer rather than the write veneer because ecraven's r7rs-coverage suite reaches for it as an r5rs name, and (scheme base) does not export it -- so (scheme write), which does, is where it belongs. - (include "bytevector.lsp") and the current-*-port parameters from parameterize.scm were absorbed by earlier slices. - The case macro, define-values, the guard macro and char-upcase / char-downcase were base-mode shims with library-layer equivalents already in place. system.lsp's case is correct again now that nothing shadows it, which retires the fix from f15b79b961. - The (bound? ...) guards around make-list, list-copy and %cdrs were already dead: they existed because srfi-1 defined the same names, and srfi-1 became (scheme list) several commits ago. r7e/prelude.lsp no longer says (import (scheme base)). It loads lib/parameterize.scm and lib/let-values.lsp directly instead -- the two facilities eflisp genuinely lacks, as opposed to r7rs spellings of things it has. The expander is written in eflisp now, not in r7rs. lib/srfi/0.scm goes too: it supplied cond-expand to the veneer and nothing else referenced it, while (scheme base) has had its own syntax-rules cond-expand all along. Base mode loses (import (scheme base)). This was the accepted cost when the approach was chosen -- verified then that r7expander.lsp was its only consumer in the tree and that efml does not use it. The transcript in lib/symtab.lsp that shows it is now stale documentation, not broken code. lib/bytevector.lsp survives as an opt-in base-mode extra alongside quicksort.lsp and disassembler.lsp; it is no longer on any boot path. Net effect of the six slices: (scheme base)'s ~110 procedures each have one definition site, in the library that exports them, reached through the library system rather than through the base global namespace. efscm.boot 191881 -> 193450, so the whole relocation cost 1569 bytes -- the peak during the move was 207180, when both copies were resident. Verified after a clean rebuild: make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test byte-identical at 20 errors, all 59 probe cases identical, flisp.boot untouched, and the static free-variable check reports no unresolved scheme.base: references. check-in: 0112c0ede8 user: e tags: selfhost | |
| 12:12 | Self-hosting phase 3: move (scheme base) section 6.13 into the library. Fifth slice: input and output. Everything in 6.13 except eof-object, eof-object? and newline -- which are eflisp primitives -- moves out of lib/scheme/base.scm and into lib7/|init|/scheme/base.sld's body. That is the port predicates, close-port and friends, call-with-port, read-char, peek-char, char-ready?, read-line, read-string, write-char, write-string, flush-output-port, the string and bytevector ports, the u8 operations and the bytevector reads and writes. The three current-*-port parameters come too. They are the last thing this library needed from lib/parameterize.scm besides make-parameter itself, and they are r7rs names, so they belong here: each is a make-parameter whose third argument names the eflisp global behind it, so the parameter and the global stay in step. make-parameter and parameterize stay in base -- the expander uses them, and they are facilities eflisp lacks rather than r7rs spellings of things it has. (eflisp native) gains *error-stream* and the io.* primitives the port procedures are built from -- io.getc, io.peekc, io.putc, io.flush, io.closed?, io.read, io.copy, io.pos, io.readall, io.eof?, io.readline -- which were reachable as base globals before and now have to be imported by name. Two comments carried over from the veneer mark real bugs, left as found: output-port? answers iostream? and so says yes to read-only ports, and peek-u8 peeks a utf-8 character rather than a byte. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test byte-identical at 20 errors, 59 probe cases identical, no unresolved scheme.base: references. efscm.boot 202864 -> 207180. check-in: 746ca2e0fb user: e tags: selfhost | |
| 12:10 | Self-hosting phase 3: move (scheme base) sections 6.10-6.11 into the library. Fourth slice: control features and exceptions. Moved: dynamic-wind, call-with-current-continuation and its -a-cont- tag, with-exception-handler, raise-continuable, error-object?, error-object-message, error-object-irritants, file-error?, read-error?. Left native: values, call-with-values, raise, error, procedure?, apply, map -- all genuinely base. One shape change. The veneer wrapped with-exception-handler and raise-continuable in a let* so they could share the *exception-handlers* stack, and installed them with set! on globals that were otherwise never defined. A library body cannot express that, so the stack is now a private library global -- not exported, same lifetime, same contents. That is what it always was; it just has a name now. call/cc did not move: (scheme base) already exports it as a rename of call-with-current-continuation, so the veneer's separate define was redundant. Neither did the veneer's define-macro guard, which was base mode's; the library has had its own syntax-rules guard all along. make test, etest, test7, etest7 pass, chibi 983/983 in both modes, r7rs-coverage-test byte-identical at 20 errors, 59 probe cases identical, no unresolved scheme.base: references. efscm.boot 201467 -> 202864. check-in: ed46925342 user: e tags: selfhost | |