Orphan stores & the .archived archive

Deleting a site removes its docroot — including the in-site files/private symlinks — and never the uploaded data in its per-site store under static/files/<url>/: the Delete task sets that store aside into the dated .archived directory itself, before the site's alias goes, and a migrate under a new name does the same for the old name's store. What a task leaves at its live name — a store named by a share control file, a store some live site still reads through its link, a store kept by the operator's opt-out switch below, a leftover from an older backend or an interrupted delete — is a ghost store: uploaded data with no live site of its own. This page covers how the subsystem detects ghosts, the three situations in which a store is moved aside (at task time, at name reuse and in the nightly sweep), the .archived layout they land in, the accumulation alert, and why pruning the archive is deliberately left to you. The store layout itself belongs to the overview, the tools and their logs to the tools reference, and the cron/opt-in plumbing to nightly automation.

One rule governs everything here: nothing on this page ever deletes data. Every path archives — normally a free same-filesystem rename into a hidden directory — and every archiving logs a du/df disk-space snapshot, even when space is ample.

Do not confuse these ghost stores with the registration ghosts (platforms, vhosts, site aliases) handled by the opt-in ghost & empty-artefact cleanup — that family moves stale registrations to undo/; this page is about the site's data.

The strict orphan test

The account sweep scans static/files/ for entries that look like site storage (a directory containing files/ or private/, or any symlinked entry) and have no matching active site. For each candidate <name> it checks two registration artefacts inside the account:

• the Drush alias — /data/disk/<account>/.drush/<name>.alias.drushrc.php; for the account's own control panel, which has no per-site alias, hostmaster.alias.drushrc.php whose uri is the panel's name
• the nginx vhost — /data/disk/<account>/config/server_master/nginx/vhost.d/<name>

Only a store whose name has neither is archived — a genuinely deleted site — and even then not while a share control file names it, nor while some registered site (one that still has its alias or vhost — a leftover directory's link does not count) reads it through its own files/private link: a clone whose unshare was refused for disk space, a hand-made link. Such a store is left in place with an [ALERT] naming the file or the link, for you to review. The double test exists because of disabled sites: a merely disabled site keeps its alias and a placeholder vhost, so it fails the test on both counts, is treated as active, and its store is left in place — the files stay live for a later re-enable. A partial/broken state (only one of alias/vhost survives) is reported but also left alone, with an explicit line in the daily report:

[REPORT] ORPHAN static/files entry in account <acct>: <path> (size=<N>K) – a surviving alias/vhost remains (disabled or partial state); left in place, NOT archived

Hidden entries (.archived itself), lost+found, and the known non-site entries dbackup and foo.com are skipped outright. symlinkinfo <site> (see the tools reference) prints the same classification — active / disabled / deleted / partial — for any name you ask about.

The .archived layout

Every archive, from any of the three triggers below, lands under the store itself:

/data/disk/<account>/static/files/.archived/<UTC-stamp>/<site>/{files,private}

The stamp is %Y%m%dT%H%M%SZ-<pid> and is run-scoped, so a site's files and private (and all orphans of one sweep) share one stamp directory. Two deliberate properties of the location:

• Under static/files, not a sibling. static/files may be a symlink onto attached storage; a static/files-* sibling would land on the possibly-tiny root filesystem and turn a free rename into a cross-FS copy that can fill the disk. Under the store, the move is always a same-filesystem rename.
• Leading dot. The orphan/site scan ignores .* entries, so the archive is never itself mistaken for a live or orphaned site — yet the tools still track it (the accumulation report below, and symlinkinfo's archive history). The same convention hides the relocated .backups/.backup-exports dirs — see backups on the static filesystem.

The move degrades safely: if the archive base somehow sits on a different device and the ghost would not fit in its free space, or if the mv fails, the reuse-time path renames the store aside in place and the nightly path leaves the orphan where it is — a [WARN] line either way, never a disk-filling cross-FS copy, and never a blocked conversion.

Task-time archiving: delete and rename

The Delete task does not leave the store for the sweep: before the site's alias goes it asks the root-side wrapper (fix-drupal-site-symlinks.sh --site=<uri> --account=<acct> --archive-store) to set static/files/<uri>/ aside into the same dated archive — DELETE/STORE/ARCHIVED in the task log, [APPLY] Archive-store: … in autosymlink.log, never a deletion. A migrate under a new name (the rename path) does the same for the old name's store after the renamed site has been re-homed into its own copy, logged as RENAME/STORE/ARCHIVED. Both keep the store at its live name when the wrapper declines or fails, with a …/STORE/LEFT warning in the task log and the reason in autosymlink.log: the deliberate case is a store named by a share control file (static/control/share.*.<site>.info — another site reads it); the protective case is a store some registered site (one that still has its alias or vhost) reads through its files/private link — the renamed site itself when its re-home was refused for disk space, a clone whose unshare was refused; the [ALERT] names the link, and the fix is to re-run the unshare, never to move the store, while a link left by an unregistered leftover directory is reported and does not hold the archive; and a failed move counts too (no room on a cross-filesystem archive target, a failed mv). The opt-out switch below holds the task's hand before the wrapper is called at all (…/STORE/KEPT); …/STORE/NONE says the site had no store. Whatever a task leaves behind for any reason is a normal reported orphan for the nightly sweep, which applies the same reader guard and honours a share control file the same way. The task-side detail is on task failures.

Reuse-time archiving

Ghost stores used to block reusing a deleted site's name. Now, when an install, clone, or migrate/rename reuses a name whose stale store still exists, autosymlink archives it aside automatically and the new site converts cleanly — no manual step. This archive never refuses (the conversion needs the name): a stale store some registered site still reads through its own files/private link is moved anyway, with an [ALERT] naming that reader, whose link dangles until its unshare is re-run. The dry-run announces it beforehand:

[PLAN] Existing (stale) store target will be archived before move: <...>/static/files/<site>/files -> <...>/static/files/.archived/<stamp>/<site>/files

The same archiving covers break-sharing collisions: when a clone or migrate whose deployed link still points at another site's store needs its own copy (--force-unshare), a pre-existing store at the target name is archived before the copy — handled identically to the local-convert path, so it lands in the same reported, prunable archive instead of cluttering the live store. See site cloning and cross-host migration for the task-side view.

On migrate/rename both ends are handled at task time: the target name's stale store is archived at reuse time, and the old name's store is set aside by the task's post hook (RENAME/STORE/ARCHIVED, the section above). Only what a task left in place, or a leftover from an older backend, waits for the nightly sweep below.

The nightly sweep

With the nightly automation enabled (_AUTOSYMLINK_NIGHTLY=YES, updatesymlinks --auto-fix — cadence and opt-ins on nightly automation), the apply modes auto-archive each deleted-site leftover that passes the strict orphan test, logging one incident per store:

[REPORT] ORPHAN archive incident: <site> deleted-site leftover size=<N>K -> static/files/.archived/<stamp>/ (target FS avail=<M>K)

(The reuse-time variant reads <site>/<type> stale store instead of deleted-site leftover.)

The sweep runs with the Ægir task queue paused and drained (the self-healing /run/boa_queue_stop.pid honoured by runner.sh — see task queue), so an in-flight install or clone can never make a live site momentarily look deleted to the scan.

The read-only paths are unchanged: updatesymlinks --orphan-report and autosymlink report still only list orphans — useful as a preview, or as the steady state when the auto-fix is off. Each listed orphan also says what the sweep would do with it: left in place when a share control file names it or a registered site still reads it through its own files/private link, archived otherwise (a link left by an unregistered leftover directory is named but does not hold the archive).

Opt-out: create /data/conf/disable_orphan_store_archiving.cnf to revert the nightly sweep to report-only; remove it to re-enable. It also holds the task-time archive of a delete or a rename (the task logs DELETE/STORE/KEPT or RENAME/STORE/KEPT and leaves the store at its live name) — reuse-time archiving is unaffected, since without it a reused name could not convert at all.

Accumulation alert

Archiving never reclaims space: a big site whose name is reused many times leaves many copies. The daily report therefore sums each account's static/files/.archived/ pile (read-only, moves nothing) and, once the total crosses the threshold, emits an ORPHAN-class line:

[REPORT] ORPHAN archived-store pile in account <acct>: <...>/.archived total=<N>K >= threshold <T>K - prune old entries to reclaim space

updatesymlinks emails the report whenever it contains a [REPORT] ORPHAN line (subject ORPHANS detected), so crossing the threshold means an email; below it, a plain informational Archived-store pile ... (below threshold ...) line is logged but does not itself trigger the email.

The threshold defaults to 1 GiB (1048576 KB) and is tunable box-wide via /data/conf/native_files_archive_alert_kb.cnf — a single KB integer on the first line; remove the file to restore the default. It controls only the alert, never any archiving behaviour. The control files are catalogued on the configuration leaf.

Pruning is operator-only

No age-based auto-clean is applied to .archived/ — deliberately. The age-based purge BOA uses for /data/disk/<account>/backups is safe there because backup timestamps are always fresh; an archived orphan may hold data that is already very old yet still wanted, so purging it by age cannot be guaranteed safe. The tools archive and alert; deleting is your call, after review:

du -sh /data/disk/<account>/static/files/.archived/*        # oldest-first by stamp
rm -rf /data/disk/<account>/static/files/.archived/<UTC-stamp>

Before pruning a stamp, symlinkinfo <site> shows which archive events produced it (reuse vs deleted-site orphan) and which archived copies are still on disk, with sizes.