Internal Mechanism

This guide explains how lip works under the hood.

Pipeline

lip setup, lip install, lip uninstall and lip update run the same steps:

State file: Load the workspace's state from .lip/state.json. A workspace without it has no installed packages, unless it has a tooth_lock.json of lip v0.x, which is read instead (see Workspaces of lip v0.x).
Environment: Determine the side and platform (see Environment).
Resolver: Turn every source into a sequence of candidates: the archives it names, newest version first, each with the entry of the requested variant for the environment.
Graph solver: Choose one candidate per request and per dependency, so that every identity appears once and no two mods have the same mod name.
Plan: Plan uninstalling the packages that leave the set, in reverse installation order, then installing the packages that enter it, dependencies first, each against the workspace as the steps before it leave it. Planning downloads every archive and asset to be written and checks everything that can fail: file ownership, preserve, paths, the src of placements, the mod name in the LeviLamina manifest.json of mods, and, when the operation writes or deletes files, whether mods is a link. A tooth.json of an earlier format has been converted completely to a package file while resolving (see Earlier Formats), so the same installer handles every package, with the rules of lip.toml only.
Apply: When the operation runs hook commands for a Windows platform, or sets up the Windows server, first install what they need of the system (see System Dependencies). When the operation writes or deletes files, create mods if it is missing (see Installing Dependencies), then run the planned hooks and file changes. A file matching preserve is kept when it exists as it is about to be written, after the hook commands before it, and the package owns only the files it wrote.
State file: Save the state file once, after every step has succeeded.

Operations are all or nothing (lip-workspace(7), "Atomicity"). If anything fails before step 6, the workspace is not modified. If a step fails while applying (a file cannot be written, a hook command exits with a non-zero code, or you press Ctrl+C), lip undoes every change of the operation and leaves the state file as it was. What hook commands did cannot be undone: a failing post_install or pre_uninstall command still fails the operation and undoes lip's file changes, but not the effects of the commands that already ran.

Package Resolution

lip does not use a central package registry or module proxy. A package id such as github.com/user/repo maps directly to the git repository https://github.com/user/repo.git, which lip talks to with its own built-in client for the git smart-HTTP protocol (version 2). The git command-line tool is not required and is never run.

Versions: lip lists the repository's tags with ls-refs (only refs/tags/v* is requested). The available versions are the tags named exactly v<SemVer 2.0.0> (e.g., v1.2.3); other tags, including lenient ones such as v1.2 or v01.2.3, are ignored. Annotated tags are peeled to the commit they point to.
Git refs: A source with a git ref has one candidate. @branch:<name> and @tag:<name> are resolved with ls-refs for exactly refs/heads/<name> or refs/tags/<name> (annotated tags peeled), once per command; @rev:<commit> needs no request, and the commit is fetched by its id. Abbreviated commit ids are rejected: resolving one would need the whole history, which lip never downloads.
Lazy candidates: Candidates of a repository are produced newest tag first, and only on demand, so lip opens only the tags the solver actually needs.
lip.toml: To read the lip.toml of a version, lip fetches only the trees of the tagged commit (a shallow, partial fetch with filter blob:none, so no file contents are transferred) and then downloads that one file (or tooth.json, when there is no lip.toml). Resolving dependencies therefore costs a few kilobytes per version, regardless of the repository's size.
Candidates: A version is a candidate only when its lip.toml is valid, its id is the path of the repository, its version equals the tag version, and the requested variant has an entry for the side and platform of the workspace. Otherwise lip moves on to the next version.
Installation: Files are downloaded only when an installer needs them: the files a self asset places from the package's own archive, and the zip and mcmod assets, which are downloaded when the package is installed, like any other HTTP download.
Integrity: Every downloaded git object is checked against its git object id (SHA-1). ZIP archives and assets are checked to be valid ZIP archives; no checksum of their content is recorded or checked.
Caching: Git archives are not cached: only the files an operation reads are fetched, every time. Everything downloaded over HTTP(S) is stored in the cache as http-<SHA-256 of the URL>; a cached file that is not a valid ZIP archive is downloaded again. Downloads are written to .tmp files in the cache directory and renamed into place when complete; lip writes nothing to the system temporary directory. Tag lists and trees are kept in memory for the duration of a command; tags are looked up again on every run, so a moved tag takes effect immediately.
Proxy: If github_proxy is set to a URL, requests for github.com repositories are first sent through the proxy. If a proxied request fails, lip retries it against github.com directly. Release assets downloaded from github.com URLs use the same rule.

A version can only be installed if its tag still exists in the repository. Only public repositories reachable over HTTPS are supported. Package ids are compared ignoring ASCII case.

At a branch, a commit, or a tag not named v<SemVer> given by a git ref, there is no tag version: the version of the lip.toml there is used. A tooth.json of an earlier format at a version tag is installed with the tag version, whatever its version says.

Solving

Pinning: Installed packages enter the solver as they are, rebuilt from the state file without network access. If the request cannot be satisfied that way, lip solves it again with only the explicitly installed packages pinned, so dependencies may change version; their installed versions are tried first, so only the dependencies that must change do. lip update solves this way from the start, so that installed dependencies never hold back the newest version of an updated package.
Preference: Earlier candidates win: the newest version of a repository.
Order: Packages are installed dependencies first; packages in a dependency cycle are ordered by identity.
Clean-up: Dependencies that no explicitly installed package needs any more are uninstalled.
Same package: An installed package is replaced only when the package lip resolves differs in identity, version or commit; a package from a URL or a local path has no commit.
Reinstalling from the state file: When lip needs an installed package's files again, it installs it with the package file recorded in the state file instead of resolving the package again: for a package from a git repository, from the recorded commit; otherwise from the archive its source names now.

Package File Validation

lip.toml is described by the JSON Schema protocol/lip.schema.toml and config.toml by protocol/config.schema.toml, both written as TOML, and the state file by protocol/state.schema.json. The files of lip v0.x were described by JSON Schemas in the former specs/schemas/ directory, under legacy/v<format>/: tooth.schema.yaml for tooth.json formats 1, 2 and 3, tooth_lock.schema.yaml for tooth_lock.json and liprc.schema.yaml for liprc.json (all format 3).

Validation: Beyond the schema, lip checks what lip.toml(5) requires: a valid package id, version, dependency keys and ranges, sides and platforms, no two entries of a variant for the same side and platform, at most one mcmod asset per entry, and valid globs. An archive whose lip.toml is invalid is not a package and yields no candidate, and lip reports why.
lip v0.x files: An archive without lip.toml is read from its tooth.json of format 1, 2 or 3: formats 1 and 2 are converted to format 3 in memory, and format 3 is converted completely to a package file for the environment (lip-compatibility(7), "tooth.json"); run lip pkg migrate in a package repository to replace its tooth.json by lip.toml ("lip pkg migrate"). A tooth_lock.json is migrated to .lip/state.json ("tooth_lock.json"), and a liprc.json to config.toml ("liprc.json"), automatically.

Workspace State

lip maintains the state of your installed packages in the state file kept in the workspace, at .lip/state.json (see State File). This tracks:

Which packages are explicitly requested versus implicit dependencies.
The package file each package was installed with (and so its exact version), its source, and, for a package from a git repository, its commit.
The files each package owns.
The side and platform of the workspace.

Files written to make an operation atomic are hidden and sit next to their target: the state file is first written to .lip/state.json.tmp and then renamed over .lip/state.json, and a file that an operation overwrites or deletes is first renamed to .<name>.bak (or .<name>.bak1, .<name>.bak2, … if that name is taken) in the same directory. The backups are deleted once the state file is saved (one that cannot be deleted, for example because a program has it open, is left and named in a warning), or renamed back if the operation fails. Directories that an uninstallation empties are deleted at the same time. On Windows these files also get the hidden attribute.

The journal and the workspace lock below are kept in .lip next to the state file, so they move with the workspace: a moved or copied workspace takes an interrupted operation with it, and the journal records paths relative to the workspace. Their formats are defined in protocol/journal.md and protocol/lip-workspace.md, so that other programs can take part.

Journal: While an operation changes the workspace, lip records every change, before making it, in .lip/journal. If lip is killed, the next lip setup, install, uninstall, update or run in the workspace reads the journal first: it undoes the interrupted operation (restoring the backups) and reports it, or, if the operation had already reached the point of saving its state file, finishes it by deleting the backups. A dry run takes no lock and only reports the journal.
One operation at a time: An operation holds .lip/lock open exclusively (an exclusive flock on Linux and macOS; the file is kept). A second lip setup, install, uninstall or update of the same workspace fails at once with "Another process is changing the workspace"; run it again when the first one has finished. lip run holds the lock only until it starts the server.

System Dependencies

The programs of a workspace whose platform starts with win (hook commands and the Windows server, with or without LeviLamina) need two things of the system, which lip installs itself (lip-workspace(7), "System dependencies"):

Wine, on Linux and macOS: a build of Wine that the version of lip names, downloaded once and extracted into wine/ in the data directory, shared by every workspace: Kron4ek/Wine-Builds (amd64-wow64) on Linux on x64, and Gcenx/macOS_Wine_builds (wine-devel) on macOS, which Rosetta 2 runs on arm64. lip never runs another wine, such as one on PATH.
The Microsoft Visual C++ Redistributable (x64): when the registry has no value Installed = 1 under HKLM\SOFTWARE\Microsoft\VisualStudio\14.0\VC\Runtimes\x64, lip downloads vc_redist.x64.exe and runs it with /install /quiet /norestart: on Windows as an administrator, which may show a User Account Control prompt; elsewhere through Wine, in the Wine prefix of the workspace, whose registry it reads with wine reg query.

Every download has its URL and SHA-256 written into lip, and is checked before it is used: a download whose SHA-256 differs fails without being extracted or run. Downloads are written to the data directory and deleted once used; Wine is extracted into a .tmp directory and renamed into place, so a build that was cut off is never run. Processes of lip that install at the same time wait for each other through the file lock in the data directory.

lip installs them before an operation that runs hook commands, or a lip setup of the build windows, starts to change the workspace, so that a failure there leaves the workspace as it was, and before lip run starts the Windows server. The Linux server needs neither. When everything is installed, this costs a registry lookup.

Daemon

Every command except lip daemon, and the interface, are carried out by the daemon of the user, a lip process started with the single argument --daemon (lip-daemon(1)). The lip you start is its client:

It connects to the daemon: on Windows through the named pipe \\.\pipe\lip-<SID>, which only your user can open; on Linux and macOS through the Unix domain socket $XDG_RUNTIME_DIR/lip/daemon.sock, or daemon.sock in the state directory when XDG_RUNTIME_DIR is not set, with the mode 0600 in a directory of mode 0700.
When nothing listens, it starts the daemon with its own executable, detached from the terminal and without its standard streams, and connects once the daemon listens. Two clients that start a daemon at the same time end up with one: the second daemon cannot create the endpoint and exits.
When the daemon is of another version of lip, or runs another executable, as after lip self-update, the client asks it to exit and starts a new one. A daemon that is carrying out a command or runs a managed server keeps running; the client then carries out its command itself, except lip run --managed, which the old daemon carries out.
It sends the command with its working directory, its environment variables, whether its standard streams are terminals and the size and colors of the terminal. The daemon carries the command out with them, as lip would: commands of different clients run side by side, each in its own directory, with its own environment, proxy and failure report.
The daemon reads the standard input of the client only when the command needs it, such as the input of the server of lip run, and sends back what the command and the programs it runs print. Signals that the client receives (Ctrl+C, SIGTERM, closing the terminal) are passed on: Ctrl+C cancels and undoes an operation that changes the workspace, and asks the server of lip run to stop. When the client goes away while the server runs, the server is asked to stop, unless it is a managed server (lip run --managed), which the daemon keeps running: the client then only detaches from it. The client exits with the exit code of the command.

The interface (lip without arguments) is carried out the same way. The daemon draws it, and the client writes what it draws to the terminal; the client reads the keys you press and passes them on, with the size of the terminal whenever it changes. Every terminal in which you open the interface gets its own, while the list of workspaces (workspaces.json in the state directory) and the servers started from the interface are kept by the daemon and shared by all of them. The servers started from the interface are managed servers: they keep running when you quit the interface or close its terminal.

When the daemon cannot be started or reached, the client says so and carries out the command, or the interface, itself. The daemon exits after it has had no connection and run no managed server for a while, after lip self-update replaced lip, or on lip daemon stop.

Notes on the Specification

lip manages server workspaces only: it never determines the side from the workspace, but always uses server (lip(1), "Environment"). Client workspaces are left to launchers implementing the same protocol, such as LeviLauncher.

lip implements the lip protocol.