Transactional file edits#
Some programs keep their data in files and use PostgreSQL as an index of those files. Then each write goes to two stores, and the two stores cannot commit together. A request that writes files and then fails can leave data in the files that the database refused. A request that writes several files can stop after only some of them.
The fsedit package in the runtime module prevents this. It makes an edit
to several files atomic, and it commits the edit together with the request’s
database transaction:
go get gitlab.com/terryp/model/go-runtime
fsedit needs golang.org/x/sys/unix. It builds on Linux and the BSDs,
including macOS. MustCreate works only on Linux (see
Preconditions). is the reference.
Use it in a server#
func main() {
ctx := context.Background()
// Finish what a crashed process left, before the rescan and before serving.
rec, err := fsedit.Recover(ctx, dataRoot)
if err != nil {
log.Fatal(err)
}
slog.Info("recovered", "applied", rec.Applied, "discarded", rec.Discarded)
router := mdl.NewRouter()
genmodel.RegisterAllEndpoints(router)
handler := fsedit.Middleware(dataRoot)(genmodel.DBMiddleware(pool)(router))
log.Fatal(http.ListenAndServe(":8080", handler))
}
func saveItem(ctx context.Context, item Item, data []byte) error {
// The truth first, the index second.
if err := fsedit.Write(ctx, item.TOMLPath(), data); err != nil {
return err
}
return item.Upsert(ctx)
}
fsedit.Middleware only names the root. The transaction joins the unit of
work that DBMiddleware owns, so put fsedit.Middleware outside it. A
server without a database uses txn.Middleware in place of DBMiddleware.
The first Write or Remove in a request starts the transaction. Later
edits in the same request join it. Two edits to one path become one edit, and
the last edit wins.
Code with no request, such as a scanner, an importer, or a command, calls
fsedit.Begin(root), edits through the returned *Tx, and then calls its
Commit or Rollback.
Stream a write#
WriteFrom stages what an io.Reader reads, up to its end, instead of a
[]byte, so a large file never has to be held in memory:
func saveUpload(ctx context.Context, path string, r *http.Request) error {
return fsedit.WriteFrom(ctx, path, r.Body)
}
The content is copied into the root’s .txn directory first, and only then
is the directory locked. A slow reader, such as a client sending a large
body, therefore holds no lock and keeps no other transaction waiting. Once
the copy is done, WriteFrom takes the lock, checks its preconditions, and
renames the copy to its staged name. It coalesces with an earlier edit of the
same target as Write does.
A copy that fails, because the reader returns an error or the context is
canceled, leaves nothing staged. An earlier edit of the same target is kept
as it was, and the transaction can still commit. Write is WriteFrom with
a bytes.Reader.
WriteFrom is two steps, which you can also take yourself when the path is
not known until the content is in: Receive copies the content in, and
WriteReceived stages it as a path’s. The REST upload endpoint receives the
body before it locks the row that says where the file goes:
in, err := fsedit.Receive(ctx, r.Body)
if err != nil {
return err
}
defer in.Close() // removes the content unless it was written
row, err := genmodel.GetItemForUpdate(ctx, id)
if err != nil {
return err
}
return fsedit.WriteReceived(ctx, row.FilePath(), in)
The rename needs the target on the same filesystem as the root. A target on
another one, under a mount point inside the root, is copied under the lock
instead. A *Tx has WriteFrom, Receive and WriteReceived methods as
well.
The protocol#
receive copy the content to .txn/<copyid>.part and fsync it
|
edit lock the directory, then rename the content to
.<base>.<txnid>.next (a remove writes a .<base>.<txnid>.del
tombstone); the real files do not change
|
DB commit the gatekeeper: data that the database refuses never gets to
the files
|
manifest .txn/<txnid>.json lists every target. It is written to a
temporary name, fsynced, renamed into place, and .txn is fsynced
<-- the point of no return
|
apply rename each staged file over its target, or remove the target
and its tombstone; fsync each directory
|
finish remove the manifest; release the locks
The database commits first. If the manifest came first and the database commit then failed, the files would contain data that the database refused. Because the files are the source of truth, the next rescan would put that data back into the database. If a process stops between the database commit and the manifest, the write is lost. The rescan then shows the old value in both stores.
Recovery#
Recover(ctx, root) uses two rules. Neither rule has to find out which files
belong together:
The manifest exists. The transaction committed.
Recoverapplies the manifest again and then removes it. Each step can run again safely: a staged file or tombstone that exists is applied, and one that does not exist was applied already.No manifest exists. The transaction did not commit.
Recoverremoves each staged file and tombstone with that transaction id.
Received content, in .txn until a write moves it, has an id of its own
with its own liveness lock. Recover leaves it alone while its process
runs, and removes it once that process has stopped.
Call Recover at start, before serving. It does not touch transactions that
are still running in another process, so it is safe while other instances
serve from the same tree.
When a commit fails after the manifest#
The files can fail after the manifest is in place. For example, the disk has no space for metadata, the filesystem reports an I/O error, or another process changed a directory. Nothing can check for these failures in advance.
The transaction is still committed. It must not be handled as a rollback.
Commit returns the error so that you can log it. The manifest stays, and
each later lock on an affected directory, and each Recover, tries again.
Until one of these tries succeeds, the directory accepts no new edits, because a replay would write over them. Each try fails with an error that names the transaction and the directory. Directories that the transaction did not touch are not affected. When the cause is gone, the next try finishes the work, and the directory accepts edits again.
Names to skip#
A program that scans the tree must skip these names:
the
.txndirectory at the root;hidden files that match
.*.<txnid>.nextand.*.<txnid>.del, where<txnid>is 16 lower-case hexadecimal digits.
A target cannot have one of these names.
Locking#
Several processes can edit one tree. A transaction holds an exclusive
flock on each directory that it edits, from its first edit in the
directory until it finishes. Thus edits in one directory are serialized. The
lock is on the directory itself, so the tree contains no lock files.
Locks are taken in ascending path order. If a transaction needs a directory that sorts before a directory it holds, it releases the later locks, takes the new lock, and then takes the released locks again. After that, it checks its staged files and preconditions again.
An edit never blocks on a lock. It tries again until its
LockTimeout(DefaultLockTimeoutif not set), and then returnsErrLocked. The database commit occurs while the locks are held, and PostgreSQL cannot detect a deadlock that goes through aflock. MapErrLockedto 409 or 503.
A write’s content is received before the directory is locked (see Stream a write), so the time that the lock is held does not depend on how fast the content arrives.
When a transaction takes a lock, it first finishes any committed edits in that directory that a stopped process did not apply. Otherwise a later recovery could replace a newer file with the stopped transaction’s older file.
Preconditions#
Option |
The edit fails with |
|---|---|
|
the target exists |
|
the target is not the version that |
The check occurs under the directory lock, and the lock stays until the
transaction finishes. Thus no other fsedit transaction can change the
target after the check. WriteFrom also checks before it copies its
content, so that it fails early, but only the check under the lock decides.
A MustCreate edit also commits with renameat2 and RENAME_NOREPLACE.
Thus it does not replace a file that a program outside fsedit created after
the check. This needs Linux 3.15 or later and a filesystem that supports the
flag, for example ext4, btrfs, xfs, or tmpfs. fsedit has no fallback:
On other platforms, the
MustCreateedit itself fails with an error that wrapserrors.ErrUnsupported.On a Linux filesystem without the flag, the rename fails during the commit, after the manifest, with an error that wraps
errors.ErrUnsupported. This is the same as any other failure after the manifest: the other files are published, and the directory of the new file accepts no edits until the file can be created.
Durability#
Each staged file is fsynced before the manifest. Received content is readable only by its owner until it is written. It then takes the replaced file’s permissions exactly, or a new file’s under the umask, and is fsynced again. The manifest is fsynced before the first rename. Each directory is fsynced after its renames and removals. Without the directory fsync, a rename can be visible before the data is.