Also validate that they are in strictly ascending order, which also ensures
that there are no duplicate names.

When 'guix substitute' fetches a nar, it does so with a validly-signed hash
already known from the narinfo.  But it can't verify that the hash of the nar
it's currently fetching matches until the full nar is downloaded.  Until then,
'download-nar' will extract the nar into the specified destination using
'restore-path', which avoids having to keep a file of unbounded size in
memory.  Critically, this means that the input that 'restore-path' (and by
extension 'fold-archive') is processing is untrusted, since substitute server
TLS certificates aren't verified (the narinfo signatures are supposed to make
it unnecessary).

As such, the scope of harm that can be caused by a malicious nar being
processed by 'restore-file' needs to be minimized.

* guix/serialization.scm (valid-nar-file-name?): new procedure.
  (fold-archive): Use it to verify that directory entry names do not contain
  '/' or '\0', and they are not equal to ".", "..", or "".  Also verify that
  they are in strictly ascending order.
  (call-with-port*): new procedure.
  (dump-file): use O_EXCL and O_NOFOLLOW.  This precaution ensures that even
  if 'restore-file' is somehow tricked into writing to a symlink (not
  currently believed to be possible), it will result in an error.
* guix/store/deduplication.scm (call-with-fresh-output-file): new procedure.
  (dump-file/deduplicate): use it for similar reasons as 'dump-file'.
* tests/nar.scm (call-with-tree-port, port-bad-nar?): new procedures.
  ("write-file-tree + fold-archive, unsorted directory entries",
   "write-file-tree + fold-archive, duplicate directory entries",
   "write-file-tree + fold-archive, invalid directory entries"): new tests.
* tests/publish.scm (call-with-temporary-output-filename): new procedure.
  ("/nar/*", "/nar/gzip/*", "/nar/lzip/*", "/nar/zstd/*", "/nar/ with properly
  encoded '+' sign"): use it in these test cases so that the output filename
  doesn't name an already-existing file.

Change-Id: I41f248c13d7af787233afad5cae102056329a68b
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

Fixes guix/guix#2450.

This reverts commit b5745a327e8dae21caaf10b59256dc7b16d54588, which introduced
discrepancies in how substitutes are served; in particular, narinfos of
non-substitutable items would still be served, and likewise for narinfos and
nars of dependents of non-substitutable items.

This commit prevents Guix substitute servers from distributing binaries
which are marked non-substitutable.  This prevents substitute servers
from accidentally committing copyright violations by distributing
binaries that are non-substitutable for copyright reasons.

* guix/scripts/publish.scm (render-nar): Query the derivers of
‘store-path’ and do nothing if one of them does not match
‘substitutable-derivation?’.
* tests/publish.scm ("non-substitutable derivation"): New test.

Change-Id: Iaca81f5bdb430a12a3ad41e9b83e0bcc535af607
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
Modified-by: Ludovic Courtès <ludo@gnu.org>

On systems with more than 64 processors. Otherwise starting so many threads
during the test will trip resource limits and lead to a spurious failure with
SIGABRT.

I'm seeing this on a 128 core system running Guix, and limiting to just 64
processors seems sufficient.

* tests/publish.scm: Call setaffinity when the current processor count exceeds
64.

Change-Id: Id4c6d7dcc9ae8dd8dafa2587def98f538061425c

* guix/utils.scm: Remove re-export of ‘call-with-temporary-output-file’.
Autoload a number of modules.
* guix/download.scm, guix/import/hackage.scm,
guix/import/hexpm.scm, guix/import/opam.scm,
guix/import/pypi.scm, tests/cpio.scm, tests/egg.scm,
tests/opam.scm, tests/publish.scm, tests/store-database.scm,
tests/utils.scm: Adjust imports accordingly.

Change-Id: I3f5e94631397996a30be2ea4ff8b50a3371e8ee7

Fixes <https://issues.guix.gnu.org/54723>.
Reported by Guillaume Le Vaillant <glv@posteo.net>.

Regression introduced in f743f2046be2c5a338ab871ae8666d8f6de7440b.

With commit f743f2046be2c5a338ab871ae8666d8f6de7440b, responses to
pipelined GETs would end up being written concurrently by many threads.
Thus the body of those responses could be interleaved and garbled.

* guix/scripts/publish.scm: Revert
f743f2046be2c5a338ab871ae8666d8f6de7440b.
* tests/publish.scm ("/*.narinfo pipeline"): New test.

This will allow mirror operators to alter these non-normative bits of a
narinfo without having to resign narinfos.

* guix/scripts/publish.scm (narinfo-string): Remove
URL/Compression/FileSize from BASE-INFO.  Move them after "Signature".
* tests/publish.scm ("/*.narinfo")
("/*.narinfo with properly encoded '+' sign")
("/*.narinfo with lzip + gzip")
("with cache, lzip + gzip"): Adjust accordingly.
* tests/substitute.scm ("query narinfo with signature over relevant subset"):
New test.

This is a followup to 575e52ac2b090fd194086e9c1c53bbf8055acbc2.

* tests/publish.scm ("/log/NAME"): Expect a gzip-encoded log.

* guix/scripts/publish.scm (show-help, %options): Add '--negative-ttl'.
(render-narinfo, render-narinfo/cached, make-request-handler): Add #:negative-ttl
and honor it.
(run-publish-server): Add #:narinfo-negative-ttl and honor it.
(guix-publish): Honor '--negative-ttl'.
* tests/publish.scm ("negative TTL", "no negative TTL"): New tests.

Fixes <https://bugs.gnu.org/47239>.
Reported by Konrad Hinsen <konrad.hinsen@fastmail.net>.

* tests/publish.scm ("with cache"): Pass the result of 'stat:perms' to
'logand' to be umask-insensitive.

Signed-off-by: Ludovic Courtès <ludo@gnu.org>