Also validate that they are in strictly ascending order, which also ensures
that there are no duplicate names.

When 'guix substitute' fetches a nar, it does so with a validly-signed hash
already known from the narinfo.  But it can't verify that the hash of the nar
it's currently fetching matches until the full nar is downloaded.  Until then,
'download-nar' will extract the nar into the specified destination using
'restore-path', which avoids having to keep a file of unbounded size in
memory.  Critically, this means that the input that 'restore-path' (and by
extension 'fold-archive') is processing is untrusted, since substitute server
TLS certificates aren't verified (the narinfo signatures are supposed to make
it unnecessary).

As such, the scope of harm that can be caused by a malicious nar being
processed by 'restore-file' needs to be minimized.

* guix/serialization.scm (valid-nar-file-name?): new procedure.
  (fold-archive): Use it to verify that directory entry names do not contain
  '/' or '\0', and they are not equal to ".", "..", or "".  Also verify that
  they are in strictly ascending order.
  (call-with-port*): new procedure.
  (dump-file): use O_EXCL and O_NOFOLLOW.  This precaution ensures that even
  if 'restore-file' is somehow tricked into writing to a symlink (not
  currently believed to be possible), it will result in an error.
* guix/store/deduplication.scm (call-with-fresh-output-file): new procedure.
  (dump-file/deduplicate): use it for similar reasons as 'dump-file'.
* tests/nar.scm (call-with-tree-port, port-bad-nar?): new procedures.
  ("write-file-tree + fold-archive, unsorted directory entries",
   "write-file-tree + fold-archive, duplicate directory entries",
   "write-file-tree + fold-archive, invalid directory entries"): new tests.
* tests/publish.scm (call-with-temporary-output-filename): new procedure.
  ("/nar/*", "/nar/gzip/*", "/nar/lzip/*", "/nar/zstd/*", "/nar/ with properly
  encoded '+' sign"): use it in these test cases so that the output filename
  doesn't name an already-existing file.

Change-Id: I41f248c13d7af787233afad5cae102056329a68b
Signed-off-by: Ludovic Courtès <ludo@gnu.org>

Files smaller than 8 KiB typically represent ~70% of the entries in
/gnu/store/.links but only contribute to ~4% of the space savings
afforded by deduplication.

Not considering these files for deduplication speeds up file insertion
in the store and, more importantly, leaves 'removeUnusedLinks' with
fewer entries to traverse, thereby speeding it up proportionally.

Partly fixes <https://issues.guix.gnu.org/24937>.

* config-daemon.ac: Remove symlink hard link check and CAN_LINK_SYMLINK
definition.
* guix/store/deduplication.scm (%deduplication-minimum-size): New
variable.
(deduplicate)[loop]: Do not recurse when FILE's size is below
%DEDUPLICATION-MINIMUM-SIZE.
(dump-port): New procedure.
(dump-file/deduplicate)[hash]: Turn into...
[dump-and-compute-hash]: ... this thunk.
Call 'deduplicate' only when SIZE is greater than
%DEDUPLICATION-MINIMUM-SIZE; otherwise call 'dump-port'.
* nix/libstore/gc.cc (LocalStore::removeUnusedLinks): Drop files where
st.st_size < deduplicationMinSize.
* nix/libstore/local-store.hh (deduplicationMinSize): New declaration.
* nix/libstore/optimise-store.cc (deduplicationMinSize): New variable.
(LocalStore::optimisePath_): Return when PATH is a symlink or smaller
than 'deduplicationMinSize'.
* tests/derivations.scm ("identical files are deduplicated"): Produce
files bigger than %DEDUPLICATION-MINIMUM-SIZE.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
Likewise.
* tests/store-deduplication.scm ("deduplicate, below %deduplication-minimum-size"):
New test.
("deduplicate", "deduplicate, ENOSPC"): Produce files bigger than
%DEDUPLICATION-MINIMUM-SIZE.
* tests/store.scm ("substitute, deduplication"): Likewise.

This avoids having to traverse and re-read the files that we have just
restored, thereby reducing I/O.

* guix/serialization.scm (dump-file): New procedure.
(restore-file): Add #:dump-file parameter and honor it.
* guix/store/deduplication.scm (tee, dump-file/deduplicate): New
procedures.
* guix/nar.scm (restore-one-item): Pass #:dump-file to 'restore-file'.
(finalize-store-file): Pass #:deduplicate? #f to 'register-items'.
* tests/nar.scm <top level>: Call 'setenv' to set "NIX_STORE".

* guix/serialization.scm (restore-file): Set the permissions and mtime
of FILE.
* guix/nar.scm (finalize-store-file): Pass #:reset-timestamps? #f to
'register-items'.
* tests/nar.scm (rm-rf): Add 'chmod' calls to ensure files are writable.
("write-file + restore-file with symlinks"): Ensure every file in OUTPUT
passes 'canonical-file?'.
* tests/guix-archive.sh: Run "chmod -R +w" before "rm -rf".

* guix/serialization.scm (fold-archive): Call PROC with a
'directory-complete tag when done with a directory.
(restore-file): Handle it.
* guix/scripts/archive.scm (list-contents): Likewise.
* guix/scripts/challenge.scm (archive-contents): Likewise.
* tests/nar.scm ("write-file-tree + fold-archive"): Adjust accordingly.

* guix/serialization.scm (read-contents): Remove.
(read-file-type, fold-archive): New procedures.
(restore-file): Rewrite in terms of 'fold-archive'.
* tests/nar.scm ("write-file-tree + fold-archive")
("write-file-tree + fold-archive, flat file"): New tests.

Fixes <https://bugs.gnu.org/33603>.
Reported by Maxim Cournoyer <maxim.cournoyer@gmail.com>.

* guix/serialization.scm (port-conversion-strategy): New variable.
(restore-file): Parameterize it.
* tests/nar.scm ("restore-file with non-UTF8 locale"): New test.

Fixes <https://bugs.gnu.org/33361>.

* guix/store/deduplication.scm (replace-with-link): Call 'set-file-time'
and 'chmod' after 'rename-file'.
* tests/nar.scm ("restore-file-set with directories (signed, valid)"):
New test.

* guix/tests.scm (canonical-file?): New procedure.
* tests/nar.scm ("restore-file-set (signed, valid)"): Check that every
item of FILES matches 'canonical-file?'.

* tests/nar.scm: Remove test for 'open-sha256-input-port'.