This fixes a regression introduced in 26d7eb8a4adafc648ef035e91b6bbc4945d4c962
when running ‘guix-daemon’ without root privileges.

On rootless installations, we must abide by the usual file access rules, which
include the requirement that a directory be writable before it be renamed (the
rationale being that its ".." entry needs to be modified).

* guix/scripts/substitute.scm (rename-file*): New procedure.
(guix-substitute): Use it.
* tests/substitute.scm (directory-nar-sha256): New procedure.
(%narinfo/directory): New variable.
(call-with-narinfo): Add '#:directory?' keyword argument.
(with-directory-narinfo): New syntax.
("substitute, authorized key, directory"): New test case.

Fixes: guix/guix#9686
Change-Id: I9d399dd5f1717b7d02854d97e1100a0a464fac75
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #9702

Previously, by serving a valid, signed, and authorized narinfo that does not
match what the client asked for, an attacker could cause users to get the
wrong substitute.

* guix/substitutes.scm (fetch-narinfos)[handle-narinfo-response]: Check
whether the result of ‘read-narinfo’ matches ‘request’ and ignore it if it
doesn’t.
[do-fetch]: Likewise with ‘narinfo-from-file’.
* tests/substitute.scm ("query narinfo that returns different store path")
("substitute, narinfo does not match requested store item"): New tests.

Reported-by: Reepca Russelstein <reepca@russelstein.xyz>
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Merges: #9665

This change safeguards against narinfos that would advertise
“StorePath: /gnu/store/…-foo/../../../etc/passwd” or similar, which could
otherwise propagate down the line and lead to attempts to overwrite said file,
should ‘guix-daemon’ be running as root.

* guix/narinfo.scm (narinfo-maker): Validate ‘path’, ‘references’, and
‘deriver’ and return #f upon failure.
(read-narinfo, string->narinfo): Adjust docstring accordingly.
* guix/substitutes.scm (fetch-narinfos): Remove check for ‘narinfo-path’
prefix; instead check whether ‘read-narinfo’ returned true.
(cached-narinfo): Check whether ‘string->narinfo’ returned true.
* tests/substitute.scm ("query narinfo that contains invalid store path"): New
test.

Change-Id: I0be2ea88b7f762144fca35373363364df3264b5a
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

The mistake was harmless because the ‘Deriver’ and ‘References’ field values
were unchecked so far.

* tests/substitute.scm (%narinfo)
("query narinfo with signature over irrelevant bits")
("query narinfo with signature over relevant subset")
("substitute, invalid hash")
("substitute, two invalid narinfos"): Remove store prefix from ‘Deriver’ field
value.  Add hash part to ‘Deriver’ and ‘References’ values.
* tests/guix-daemon.sh: Likewise.

Change-Id: I0af4b9aa4b64cebf46d221ffab17bf5663b0cc11
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

The hash of a nar is only known once 'download-nar' completes.  By that point
the nar may already have been partially-restored to its destination for some
time.  Additionally, after a hash mismatch is detected, 'guix substitute'
leaves the invalid contents at the specified destination.  This means that
untrusted, attacker-controlled contents are present at what at some point may
have been or at some point may become a valid store path.  It is entirely
possible that a user or program won't check that a given store item is valid
before trying to use it (it may be that 'guix gc' was run while an obscured
reference was held).  Let's protect them in that case by ensuring that
unverified contents are kept elsewhere, in a temporary directory.

While implementing this one such program was discovered: tests/substitute.scm.
It actually incorrectly computes the hashes of the nars it supplies to (guix
scripts substitute), computing the hash of the plain contents instead of the
hash of the nar.  These tests nevertheless passed because they didn't check
what (guix scripts substitute) gave as output, instead only checking that the
expected contents were at the specified destination.

* guix/scripts/substitute.scm (call-with-temporary-directory-in): new
  procedure.
  (guix-substitute): use it to create a temporary directory next to the final
  destination to restore the nar inside of.  Once it is fully restored, if the
  hash matches, move it to the final destination, deleting the file currently
  there if it already exists.
* tests/substitute.scm (plain-file-nar-sha256): new procedure.
  (%narinfo, "substitute, narinfo with several URLS" test): use it.

Change-Id: Ifeb9b4c912f9d9b2e7477d237da04e7d5175ff83
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

There are two potential places where file:// URIs could be used in 'guix
substitute':
1. The substitute urls, which are used for fetching narinfos.  These can come
   from the daemon, via daemon option "substitute-urls", from the client, via
   daemon option "untrusted-substitute-urls", or from 'guix discover'.  The
   latter two must not be allowed to provide file:// URIs, since that could be
   used to cause the daemon to read private files, files in /proc or /dev,
   etc, and various parts of those files may show up in the backtrace from
   'guix substitute'.

2. The narinfos themselves, which may offer arbitrarily many URIs to download
   the nar of the store item in question.  These URIs do not need to be
   in the signed portion of the narinfo, so they may be attacker-controlled.
   These should also not be allowed to provide file:// URIs, with the sole
   exception being in the case of the test suite for the sake of simplifying
   the tests.

To accomplish this, we error out of 'guix substitute' if
"untrusted-substitute-urls" includes a file:// URI, and we filter out all
file:// URIs from those provided by 'guix discover'.  We also ignore all
narinfos containing file:// URIs unless '%allow-unsafe-substitute-uris?' is #t.

* guix/scripts/substitute.scm (%allow-unsafe-substitute-uris?,
  %allow-unsafe-narinfo-uris?): new parameters.
  (process-substitution, process-substitution/fallback): honor
  '%allow-unsafe-substitute-uris?'.
  (uri-safe?, narinfo-uris-safe?, assert-safe-uris): new procedures.
  (%default-substitute-urls): use assert-safe-uris for the
  "untrusted-substitute-urls" daemon option.
  (%local-substitute-urls): filter out file:// URIs.
* tests/substitute.scm ("file:// URI prohibited by default"): new test.  Set
  '%allow-unsafe-substitute-uris?' to #t after it finishes.

Change-Id: I34d7e58b5a01c7cc61b52fcd61b547bba2ce4300
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

This changes the behaviour for the first one or few nars the substitute script
downloads, with uncompressed and zstd compressed nars prefered rather than
picking by file size.

* guix/scripts/substitute.scm: (%default-fast-decompression?): Change to #t.
* tests/substitute.scm ("substitute, preferred nar URL is 404, other is 200"):
Adjust test.

Change-Id: I89202f084cd6b9d506bcb3d46f75de690c6986b5

Fixes <https://issues.guix.gnu.org/67575>.

* guix/scripts/substitute.scm (process-substitution/fallback): Use
‘report-error’ instead of ‘leave’.  Write status line to PORT.
* tests/substitute.scm ("substitute, narinfo is available but nar is
missing"): Adjust accordingly.

Change-Id: Ic7297dbd563c007111ec2167c8d52505a07d4822

Fixes <https://issues.guix.gnu.org/63443>.
Reported by Attila Lendvai <attila@lendvai.name>.

* guix/scripts/substitute.scm (catch-system-error): New macro.
(download-nar): Add call to 'delete-file-recursively'.
* tests/substitute.scm ("substitute, previous partial download around"):
New test.

If a substitute server advertises in its narinfo, for example, both a
/zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip
URL.

Fixes <https://issues.guix.gnu.org/63634>.

* guix/narinfo.scm (narinfo-preferred-uris): New procedure.
(narinfo-best-uri): Rebase on top of it.
* guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure.
Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of
NARINFO.
* tests/substitute.scm (request-substitution): Remove 'parameterize'.
Delete DESTINATION.
("substitute, preferred nar URL is 404, other is 200"): New test.