Transactional file edits#

Some programs keep their data in files and use PostgreSQL as an index of those files. Then each write goes to two stores, and the two stores cannot commit together. A request that writes files and then fails can leave data in the files that the database refused. A request that writes several files can stop after only some of them.

The fsedit package in the runtime module prevents this. It makes an edit to several files atomic, and it commits the edit together with the request’s database transaction:

go get gitlab.com/terryp/model/go-runtime

fsedit needs golang.org/x/sys/unix. It builds on Linux and the BSDs, including macOS. MustCreate works only on Linux (see Preconditions). is the reference.

Use it in a server#

func main() {
    ctx := context.Background()
    // Finish what a crashed process left, before the rescan and before serving.
    rec, err := fsedit.Recover(ctx, dataRoot)
    if err != nil {
        log.Fatal(err)
    }
    slog.Info("recovered", "applied", rec.Applied, "discarded", rec.Discarded)

    router := mdl.NewRouter()
    genmodel.RegisterAllEndpoints(router)
    handler := fsedit.Middleware(dataRoot)(genmodel.DBMiddleware(pool)(router))
    log.Fatal(http.ListenAndServe(":8080", handler))
}

func saveItem(ctx context.Context, item Item, data []byte) error {
    // The truth first, the index second.
    if err := fsedit.Write(ctx, item.TOMLPath(), data); err != nil {
        return err
    }
    return item.Upsert(ctx)
}

fsedit.Middleware only names the root. The transaction joins the unit of work that DBMiddleware owns, so put fsedit.Middleware outside it. A server without a database uses txn.Middleware in place of DBMiddleware.

The first Write or Remove in a request starts the transaction. Later edits in the same request join it. Two edits to one path become one edit, and the last edit wins.

Code with no request, such as a scanner, an importer, or a command, calls fsedit.Begin(root), edits through the returned *Tx, and then calls its Commit or Rollback.

Stream a write#

WriteFrom stages what an io.Reader reads, up to its end, instead of a []byte, so a large file never has to be held in memory:

func saveUpload(ctx context.Context, path string, r *http.Request) error {
    return fsedit.WriteFrom(ctx, path, r.Body)
}

The content is copied into the root’s .txn directory first, and only then is the directory locked. A slow reader, such as a client sending a large body, therefore holds no lock and keeps no other transaction waiting. Once the copy is done, WriteFrom takes the lock, checks its preconditions, and renames the copy to its staged name. It coalesces with an earlier edit of the same target as Write does.

A copy that fails, because the reader returns an error or the context is canceled, leaves nothing staged. An earlier edit of the same target is kept as it was, and the transaction can still commit. Write is WriteFrom with a bytes.Reader.

WriteFrom is two steps, which you can also take yourself when the path is not known until the content is in: Receive copies the content in, and WriteReceived stages it as a path’s. The REST upload endpoint receives the body before it locks the row that says where the file goes:

in, err := fsedit.Receive(ctx, r.Body)
if err != nil {
    return err
}
defer in.Close() // removes the content unless it was written
row, err := genmodel.GetItemForUpdate(ctx, id)
if err != nil {
    return err
}
return fsedit.WriteReceived(ctx, row.FilePath(), in)

The rename needs the target on the same filesystem as the root. A target on another one, under a mount point inside the root, is copied under the lock instead. A *Tx has WriteFrom, Receive and WriteReceived methods as well.

The protocol#

receive    copy the content to .txn/<copyid>.part and fsync it
    |
edit       lock the directory, then rename the content to
           .<base>.<txnid>.next (a remove writes a .<base>.<txnid>.del
           tombstone); the real files do not change
    |
DB commit  the gatekeeper: data that the database refuses never gets to
           the files
    |
manifest   .txn/<txnid>.json lists every target. It is written to a
           temporary name, fsynced, renamed into place, and .txn is fsynced
           <-- the point of no return
    |
apply      rename each staged file over its target, or remove the target
           and its tombstone; fsync each directory
    |
finish     remove the manifest; release the locks

The database commits first. If the manifest came first and the database commit then failed, the files would contain data that the database refused. Because the files are the source of truth, the next rescan would put that data back into the database. If a process stops between the database commit and the manifest, the write is lost. The rescan then shows the old value in both stores.

Recovery#

Recover(ctx, root) uses two rules. Neither rule has to find out which files belong together:

  • The manifest exists. The transaction committed. Recover applies the manifest again and then removes it. Each step can run again safely: a staged file or tombstone that exists is applied, and one that does not exist was applied already.

  • No manifest exists. The transaction did not commit. Recover removes each staged file and tombstone with that transaction id.

Received content, in .txn until a write moves it, has an id of its own with its own liveness lock. Recover leaves it alone while its process runs, and removes it once that process has stopped.

Call Recover at start, before serving. It does not touch transactions that are still running in another process, so it is safe while other instances serve from the same tree.

When a commit fails after the manifest#

The files can fail after the manifest is in place. For example, the disk has no space for metadata, the filesystem reports an I/O error, or another process changed a directory. Nothing can check for these failures in advance.

The transaction is still committed. It must not be handled as a rollback. Commit returns the error so that you can log it. The manifest stays, and each later lock on an affected directory, and each Recover, tries again.

Until one of these tries succeeds, the directory accepts no new edits, because a replay would write over them. Each try fails with an error that names the transaction and the directory. Directories that the transaction did not touch are not affected. When the cause is gone, the next try finishes the work, and the directory accepts edits again.

Names to skip#

A program that scans the tree must skip these names:

  • the .txn directory at the root;

  • hidden files that match .*.<txnid>.next and .*.<txnid>.del, where <txnid> is 16 lower-case hexadecimal digits.

A target cannot have one of these names.

Locking#

Several processes can edit one tree. A transaction holds an exclusive flock on each directory that it edits, from its first edit in the directory until it finishes. Thus edits in one directory are serialized. The lock is on the directory itself, so the tree contains no lock files.

  • Locks are taken in ascending path order. If a transaction needs a directory that sorts before a directory it holds, it releases the later locks, takes the new lock, and then takes the released locks again. After that, it checks its staged files and preconditions again.

  • An edit never blocks on a lock. It tries again until its LockTimeout (DefaultLockTimeout if not set), and then returns ErrLocked. The database commit occurs while the locks are held, and PostgreSQL cannot detect a deadlock that goes through a flock. Map ErrLocked to 409 or 503.

A write’s content is received before the directory is locked (see Stream a write), so the time that the lock is held does not depend on how fast the content arrives.

When a transaction takes a lock, it first finishes any committed edits in that directory that a stopped process did not apply. Otherwise a later recovery could replace a newer file with the stopped transaction’s older file.

Preconditions#

Option

The edit fails with ErrConflict when

MustCreate()

the target exists

IfUnmodified(tok)

the target is not the version that Stat returned tok for

The check occurs under the directory lock, and the lock stays until the transaction finishes. Thus no other fsedit transaction can change the target after the check. WriteFrom also checks before it copies its content, so that it fails early, but only the check under the lock decides.

A MustCreate edit also commits with renameat2 and RENAME_NOREPLACE. Thus it does not replace a file that a program outside fsedit created after the check. This needs Linux 3.15 or later and a filesystem that supports the flag, for example ext4, btrfs, xfs, or tmpfs. fsedit has no fallback:

  • On other platforms, the MustCreate edit itself fails with an error that wraps errors.ErrUnsupported.

  • On a Linux filesystem without the flag, the rename fails during the commit, after the manifest, with an error that wraps errors.ErrUnsupported. This is the same as any other failure after the manifest: the other files are published, and the directory of the new file accepts no edits until the file can be created.

Durability#

Each staged file is fsynced before the manifest. Received content is readable only by its owner until it is written. It then takes the replaced file’s permissions exactly, or a new file’s under the umask, and is fsynced again. The manifest is fsynced before the first rename. Each directory is fsynced after its renames and removals. Without the directory fsync, a rename can be visible before the data is.