gnu: chez-scheme: Bootstrap from source.

* gnu/packages/chez.scm (chez-scheme)[source]<snippet>: Also unbundle
pre-built boot files.
(chez-scheme-bootstrap-bootfiles): Stop inheriting from 'chez-scheme'.
[native-inputs]: Add 'zuo', 'chez-nanopass-bootstrap', and either
'chez-scheme-for-racket' or, for cross builds, 'chez-scheme'.
[build-system]: Change to gnu-build-system.
[arguments]: Change to run 'configure' and then bootstrap via
`make re.boot` or, for cross builds, `make cross.boot`
[synopsis, description]: Update accordingly.
(chez-scheme-for-racket-bootstrap-bootfiles)[synopsis]: Write "boot
files" as two words in prose.
[description]: Likewise.
* gnu/packages/racket.scm: Update comments.

Change-Id: I55d9b57090ecaca0aa2ab19c2c3ebae2243a9173
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

2 files changed, 112 insertions, 35 deletions

diff --git a/gnu/packages/chez.scm b/gnu/packages/chez.scm
index 4d3f79d9a72..db582fa8210 100644
--- a/gnu/packages/chez.scm
+++ b/gnu/packages/chez.scm
@@ -57,15 +57,54 @@
 ;; Commentary:
 ;;
 ;; The bootstrapping paths for Chez Scheme and Racket are closely
-;; entwined. Racket CS (the default Racket implementation) is based on (a fork
-;; of) Chez Scheme. Racket's variant of Chez Scheme shares sources for
-;; nanopass and stex with upstream Chez Scheme.
+;; entwined. See "Bootstrapping Racket" in the commentary on "racket.scm" for
+;; details on the Racket portion of Chez Scheme's bootstrapping path.
 ;;
-;; Racket's variant of Chez Scheme can be bootstrapped by an older Racket
-;; implementation, Racket BC, which can be bootstrapped from C. Porting that
-;; code to work with upstream Chez Scheme (or finding an old version that
-;; does) is our best hope for some day bootstrapping upstream Chez Scheme from
-;; source.
+;; Chez Scheme is a self-hosting compiler. A small kernel implemented in C
+;; loads "boot files" (in a custom object file format) compiled from the parts
+;; of the system implemented in Chez Scheme. (While Chez Scheme generates
+;; native machine code, it implements its own linker and loader.)
+;;
+;; As of Chez Scheme 10.0.0 (and the pre-release versions that preceded it on
+;; the Racket branch), there are several ways to obtain boot files:
+;;
+;;  1. The Racket package "cs-bootstrap" (in the "racket/src/rktboot/"
+;;     directory of the main Racket Git repository) simulates enough of Chez
+;;     Scheme to load the Chez Scheme compiler purely from source into Racket
+;;     and apply the compiler to itself, producing the needed boot files
+;;     (albeit very slowly).
+;;     Any variant of Racket since version 7.1 can run the simulation. Using
+;;     the older Racket BC implementation, which does not depend on Chez
+;;     Scheme, breaks the dependency cycle.
+;;     However, the simulation relies on implementation details of Chez
+;;     Scheme, so a given version of Chez Scheme can only be bootstrapped by
+;;     the corresponding version of the "cs-bootstrap" package.
+;;
+;;  2. The Chez Scheme makefile provides a "re.boot" target for bootstrapping
+;;     via a different version of Chez Scheme (9.5.4 or later).
+;;     This path manages potential differences in implementation details
+;;     across Chez Scheme versions using a strategy similar to "cs-bootstrap",
+;;     but the compatibility shim is maintained with the Chez Scheme source
+;;     code (in "s/reboot.ss"). Also, it's faster, since less indirection is
+;;     needed.
+;;
+;;  3. For cross-compilation, or with an extremely similar Chez Scheme, the
+;;     makefile provides "cross.boot" and related targets.
+;;
+;;  4. The Chez Scheme Git repository includes pre-built "pb" (portable
+;;     bytecode) boot files, which can be used for bootstrapping on any
+;;     platform, but these binary files are removed from the source Guix uses.
+;;
+;; Concretely, we use racket-vm-bc to bootstrap chez-scheme-for-racket, which
+;; we then use to bootstrap both chez-scheme and racket-vm-cs.
+;;
+;; In principle, it would be possible instead to use chez-scheme to bootstrap
+;; chez-scheme-for-racket. However, since Racket is ultimately used for
+;; bootstrapping, chez-scheme would still need to be rebuilt when Racket
+;; changes, whereas treating chez-scheme as a leaf avoids having to rebuild
+;; Racket when upstream Chez Scheme changes. Furthermore, since "cs-bootstrap"
+;; is developed in the Racket source repository, we don't have to look for the
+;; version of "cs-bootstrap" compatible with the upstream Chez Scheme release.
 ;;
 ;; Code:
 
@@ -479,7 +518,8 @@ version of Chez Scheme.")
                            (for-each (lambda (dir)
                                        (when (directory-exists? dir)
                                          (delete-file-recursively dir)))
-                                     '("lz4"
+                                     '("boot"
+                                       "lz4"
                                        "nanopass"
                                        "stex"
                                        "zlib"
@@ -575,10 +615,10 @@ with reliability taking precedence over efficiency if necessary.")
                                         "/opt/racket-vm/bin/racket")
                      "../rktboot/main.rkt"))))))))
     (home-page "https://pkgs.racket-lang.org/package/cs-bootstrap")
-    (synopsis "Chez Scheme bootfiles bootstrapped by Racket")
-    (description "Chez Scheme is a self-hosting compiler: building it
-requires ``bootfiles'' containing the Scheme-implemented portions compiled for
-the current platform.  (Chez can then cross-compile bootfiles for all other
+    (synopsis "Chez Scheme boot files bootstrapped by Racket")
+    (description "Chez Scheme is a self-hosting compiler: building it requires
+``boot files'' containing the Scheme-implemented portions compiled for the
+current platform.  (Chez can then cross-compile boot files for all other
 supported platforms.)
 
 The Racket package @code{cs-bootstrap} (part of the main Racket Git
@@ -591,28 +631,57 @@ long as using an existing Chez Scheme, but @code{cs-bootstrap} supports Racket
 
 (define-public chez-scheme-bootstrap-bootfiles
   (package
-    (inherit chez-scheme)
     (name "chez-scheme-bootstrap-bootfiles")
-    (inputs '())
-    (native-inputs '())
+    (version (package-version chez-scheme))
+    (source (package-source chez-scheme))
+    (native-inputs (list chez-nanopass-bootstrap
+                         (if (%current-target-system)
+                             chez-scheme
+                             chez-scheme-for-racket)
+                         zuo))
     (outputs '("out"))
-    (build-system copy-build-system)
-    ;; TODO: cross compilation
+    (build-system gnu-build-system)
     (arguments
-     (list #:install-plan
-           #~`(("boot/" "lib/chez-scheme-bootfiles"))))
-    (synopsis "Chez Scheme bootfiles (binary seed)")
+     (list
+      #:configure-flags
+      #~`("--force" ; don't complain about missing bootfiles
+          "ZLIB=-lz" "LZ4=-llz4" "STEXLIB=/GuixNotUsingStex" ; ignore submods
+          "ZUO=zuo"
+          ;; could skip -m= for non-cross non-pbarch builds
+          #$(string-append "-m=" (or (nix-system->native-chez-machine-type)
+                                     (nix-system->pbarch-machine-type))))
+      #:make-flags
+      #~(list (string-append "SCHEME="
+                             (search-input-file %build-inputs "/bin/scheme"))
+              #$(if (%current-target-system)
+                    "cross.boot"
+                    "re.boot"))
+      #:phases
+      #~(modify-phases %standard-phases
+          (add-after 'unpack 'unpack-nanopass
+            #$unpack-nanopass)
+          (replace 'configure
+            #$chez-configure)
+          (delete 'check)
+          (replace 'install
+            (lambda args
+              (mkdir-p (string-append #$output "/lib"))
+              (copy-recursively
+               "boot"
+               (string-append #$output "/lib/chez-scheme-bootfiles")))))))
+    (home-page "https://cisco.github.io/ChezScheme/")
+    (synopsis "Bootstrapped Chez Scheme boot files")
     (description
      "Chez Scheme is a self-hosting compiler: building it requires
-``bootfiles'' containing the Scheme-implemented portions compiled for the
+``boot files'' containing the Scheme-implemented portions compiled for the
 current platform.  (Chez can then cross-compile bootfiles for all other
 supported platforms.)
 
-This package provides bootstrap bootfiles for upstream Chez Scheme.
-Currently, it simply packages the binaries checked in to the upstream
-repository.  Hopefully we can eventually adapt Racket's @code{cs-bootstrap} to
-work with upstream Chez Scheme so that we can bootstrap these files from
-source.")))
+This package provides boot files for the released version of Chez Scheme
+bootstrapped by @code{chez-scheme-for-racket}. Chez Scheme 9.5.4 or any later
+version can be used for bootstrapping. Guix ultimately uses the Racket package
+@code{cs-bootstrap} to bootstrap its initial version of Chez Scheme.")
+    (license asl2.0)))
 
 ;;
 ;; Chez's bootstrap dependencies:
diff --git a/gnu/packages/racket.scm b/gnu/packages/racket.scm
index f50f6f3eab3..2b203531592 100644
--- a/gnu/packages/racket.scm
+++ b/gnu/packages/racket.scm
@@ -72,6 +72,7 @@
 ;;             ├── bc/
 ;;             ├── cs/
 ;;             ├── ChezScheme/
+;;             ├── rktboot/
 ;;             ├── zuo/
 ;;             └── ...
 ;;
@@ -149,14 +150,12 @@
 ;;   - Racket BC [3M] needs an existing Racket to run "xform",
 ;;     which transforms its own C source code to add additional annotations
 ;;     for the precise garbage collector.
-;;   - Racket CS needs (bootfiles for) Racket's fork of Chez Scheme.
-;;     It also needs an existing Racket to compile Racket-implemented
+;;   - Racket CS needs (boot files for) the corresponding version of Chez
+;;     Scheme. It also needs an existing Racket to compile Racket-implemented
 ;;     parts of the runtime system to R6RS libraries.
-;;   - Chez Scheme also needs bootfiles for itself, but Racket can simulate
-;;     enough of Chez Scheme to load Racket's fork of the Chez Scheme compiler
-;;     purely from source into Racket and apply the compiler to itself,
-;;     producing the needed bootfiles (albeit very slowly).
-;;     Any variant of Racket since version 7.1 can run the simulation.
+;;   - Chez Scheme also needs boot files for itself, but Racket BC can
+;;     bootstrap these using the code in "racket/src/rktboot/".
+;;     See the commentary in "chez.scm" for further details
 ;;
 ;; So, we build CGC to build 3M to build bootfiles and CS.
 ;;
@@ -164,6 +163,14 @@
 ;; often use "BC" to mean "3M", consistent with `(banner)` and the
 ;; suffixes used on executables when more than one variant co-exists.)
 ;;
+;; Since the pre-releases for Chez Scheme 10.0.0, all of Racket's changes have
+;; been merged upstream, and development will be kept in sync going
+;; forward. However, there is no plan to align the Chez Scheme and Racket
+;; release cycles. For the near fulture, a given released version of Racket
+;; will continue to depend on a specific pre-release version of Chez Scheme as
+;; part of Racket CS's "ABI". See upstream discussion at
+;; <https://racket.discourse.group/t/2739/3>.
+;;
 ;; One remaining bootstrapping limitation is that Racket's reader, module
 ;; system, and macro expander are implemented in Racket. For Racket CS,
 ;; they are compiled to R6RS libraries as discussed above. This note from the
@@ -189,7 +196,8 @@
 ;; Zuo is notably *not* a problem for bootstrapping. The implementation is a
 ;; single hand-written C file designed to build with just `cc -o zuo zuo.c`,
 ;; even with very old or limited compilers. (We use the Autoconf support for
-;; convienience.)
+;; convienience.) As of Zuo 1.8, Zuo has tagged releases in its own repository
+;; independent of the Racket release cycle.
 ;;
 ;; CODE: