Solr core lifecycle (manage_solr_config.sh)

/var/xdrago/manage_solr_config.sh is the runtime Solr-core reconciliation daemon (~1415 LOC). It walks every Drupal and Backdrop site (Grav and Textpattern sites carry no INI and are skipped), reads each boa_site_control.ini, syncs the core set, optimises indices, archives orphans, and runs a read-only health check. This is the script behind the "changes take a few minutes to take effect" behaviour: the INI-driven reconciliation runs on every four-minute pass; the heavier maintenance is self-throttled inside that invocation.

Scheduled runs

Invoked directly from the root crontab every four minutes — not via minute.sh:

*/4 * * * * /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/manage_solr_config.sh >/dev/null 2>&1

Three role markers stop the daemon before it does anything at all, ahead of every other check: /root/.proxy.cnf, /root/.standby.cnf and /var/log/boa/.xmass_solr_hold.pid. The last two exist because a replication standby holds Solr down until promotion, so creating, deleting or restarting a core would act on an index still being synced; the xmass Solr hold is a separate gate with its own lifetime: java.sh — launched every minute from minute.sh — keeps the Solr init scripts disarmed for as long as the hold marker is present, until cutover step 14 (or post-mig) clears it, re-arms the init scripts and starts Solr on the target, and because it is tested independently of the standby marker, a box whose standby marker has already gone — self-removed by second.sh after a hand promotion or an abandoned init — still holds Solr down until one of those two clears the hold. On a box carrying any of the three, no core is created, deleted or refreshed at all, which is the expected state on a migration target mid-window.

The whole pass is skipped during a BOA install/upgrade: at startup it scans for a running autoinit/automini/barracuda/boa/octopus process and the /run/octopus_install_run.pid, /run/boa_run.pid, /run/boa_wait.pid lock files, and sets _protectedRun=TRUE to exit without touching anything.

Force an immediate pass as root:

bash /var/xdrago/manage_solr_config.sh

The forced run still honours the 6 h / 12 h throttles; remove the relevant sentinel first to force the maintenance (below).

What runs in each pass

Pass start (every 4 min, root crontab)
  ↓ proxy / standby / xmass Solr hold marker → exit
  ↓ install/upgrade running → exit (_protectedRun)
For each Octopus instance under /data/disk/:
  For each site:
    Read boa_site_control.ini → target Solr-core state
    Compare to on-disk core
      core needed but missing      → create
      core dir present, unlisted   → re-register in place (CoreAdmin CREATE, index kept)
      core unneeded (INI removed)  → delete within 15 min (archive, then remove core + solr.php)
      integration-module changed   → re-deploy config family
      custom_config=NO & update=YES → refresh schema/solrconfig
    Sync sites/<domain>/solr.php
  ↓
Orphan cleanup (_cleanup_orphan_cores)  — throttled 6 h, after site reconcile
  ↓
Per-core health check (_check_solr_core_health) — every pass, read-only
  ↓
Index optimise (_run_optimize_if_due)   — throttled 12 h

The load gate _O_LOAD_MAX (_CPU_TASK_RATIO * 100, default _CPU_TASK_RATIO=3.1) lets a busy host skip per-site work, so a core change can be deferred a pass under high load.

Core deletion (INI removal)

Deletion is explicit, not age-based: once the site INI no longer names a recognised solr_integration_module, a subsequent pass removes the computed core ID (and its _Old_/_Legacy_ aliases) across every installed version's data dir and removes sites/<domain>/solr.php. Each populated core is archived first: Solr 9 and Solr 7 cores are unloaded without deleting any file and moved whole to /var/backups/solr9/<timestamp>-<core> or /var/backups/solr7/<timestamp>-<core> (the same shape and recovery recipe as the orphan-core archive, so the move is the delete and costs no extra disk); a Solr 4 core is tarred into /var/backups/solr-archive/ under a free-space precheck and pruned after 30 days. A core whose archive cannot be written is left in place rather than deleted. A site INI that went missing and was reseeded from the template does not count as a cleared directive: the reseed leaves a root-only sentinel, and the placeholder tears nothing down until the operator edits that line (a valid directive clears the sentinel again). The mechanism differs per version: Solr 9 is unloaded through the Core Admin API on its own port (action=UNLOAD&deleteIndex=true&deleteDataDir=true&deleteInstanceDir=true on 9099), because the Solr 9 CLI ignores -p on delete; Solr 7 still goes through solr delete -p 9077; Solr 4 is a solr.xml edit plus a directory removal and a Jetty 9 restart. BOA cleanly deletes the core within 15 minutes of the line being commented out (a four-minute pass reconciles it, subject to the load gate). The associated Drupal module stays in the codebase; disable it manually if unneeded.

Re-registering a core Solr stopped listing

Solr 7 and Solr 9 discover their cores by the core.properties file inside each core directory. When that file is gone — the classic cause is a core's conf/ replaced from an archive that did not carry it — the directory and its index are still there, but Solr starts without the core ("Found 0 core definitions"), /solr/<core>/admin/ping answers 404 and the site's search server reports errors. Nothing in the pass used to notice: the directory existed, so the update path only ever issued a RELOAD, which Solr refuses for a core it never loaded, silently, every four minutes.

The pass now repairs the missing file for every site whose solr_integration_module names a supported module (search_api_solr, search_api_solr7, search_api_solr9) whose Solr instance is installed. It runs after the config refresh, so the attempt loads the conf/ the pass has just repaired. When the core directory exists without core.properties, it asks Solr's STATUS API whether the core is registered:

• Not registered — the directory is re-registered in place with a CoreAdmin CREATE on that instanceDir. Solr writes only core.properties; conf/ and the index are untouched, and the document count comes back as it was.
• Still served from memory — the file was lost while Solr kept running (the latent phase of the same outage: search works until the next restart, then the core is gone). The pass writes core.properties back itself (name=<core>, which is all Solr writes for a core created here), so the next Solr start finds it.
• Recorded by Solr as failed to load — retried once its conf/ changes (the pass's own refresh, or a hand edit) or after an hour, never on every pass; Solr's failure record is cleared first with an UNLOAD that deletes nothing.

The repair is gated on the missing file on purpose: Solr refuses a CREATE into a directory that already carries core.properties and deletes that file on the way out, so a core that merely failed to load at startup is never touched by this path (it stays visible as HEALTH-INIT-FAIL, and a RELOAD recovers it once its config is fixed). The mirror shape — core.properties present but Solr does not list the core (a tree copied in from another box, an archive taken while Solr was down, a recovery that stopped before the CREATE) — is therefore reported, not repaired: the health check prints HEALTH-WARN for it every pass. Move that file aside and the next pass re-registers a bound site's core, or restart Solr. Solr 4 keeps its registry in solr.xml and is not covered.

With Solr down or unresponsive nothing is attempted and nothing is logged by this step (the health check's HEALTH-ERROR line is the signal). Because a CREATE is the pass's one write that cannot be repeated (a second one for the same core is refused and takes the first one's core.properties with it), the script now holds the single-instance lock every other four-minute daemon holds (lock.inc, /run/manage_solr_config.lock): a hand run during the cron tick prints "another instance is running; exiting." and does nothing. Pass logs under /var/backups/solr/log/ are pruned after a day, so a CORE-REREGISTERED line is there for about 24 hours.

Orphan-core cleanup (3-tier, 6 h)

Orphan cleanup is fully automated — you do not normally archive cores by hand. On each pass (throttled to once per 6 h via /var/backups/solr/.orphan_cleanup_last_run.pid, skipped during install/upgrade), _cleanup_orphan_cores classifies every core on disk and retires those nothing references. It matches all three historical name formats, not just the current one: oct.<user>.<domain> (current), the legacy solr.<user>.<domain>, and the bare <user>.<domain> form — so a pre-oct. core left behind by an old rename or platform migration is archived too. It runs after per-site reconciliation so a just-recreated managed core is on disk before classification.

Cores are never deleted. A qualifying core is first unloaded from Solr's registry — action=UNLOAD&deleteIndex=false&deleteDataDir=false&deleteInstanceDir=false — then the directory is moved to /var/backups/solrN/<YYYYMMDD-HHMMSS>-<core>/.

Classification

The active-core set is built from the host's nginx vhosts and Ægir drush aliases, but _build_active_core_set registers a vhost only for an enabled, non-hostmaster site: it skips any disabled/parked vhost (one carrying the "Do not reveal Aegir front-end URL here" marker) and any hostmaster (aegir/distro) vhost before recording it. A disabled, parked, or hostmaster site therefore registers no vhost, so its core is treated as having no vhost and lands in Tier 1. Each core is sorted into one tier — Tier 2 requires vhost and alias, everything else falls to Tier 1:

Staleness is measured on data/index/ mtime (Lucene segment-commit time), falling back to data/ mtime; a core with no data/ directory was never indexed and counts as immediately stale. data/index/ is the primary signal for a reason: its mtime moves only on real Lucene segment commits, whereas data/ mtime is kept perpetually fresh by tlog and write.lock bookkeeping even on an idle core — measuring on data/ alone would make every core look active.

Actively-managed cores are age-immune

A core whose site has solr_integration_module set is never archived on age alone, even if its index looks stale — _check_sites_list actively manages it, and archiving would let the next pass recreate it empty and lose the index. These log ORPHAN-SKIP. A Grav or Textpattern site is never actively managed (it carries no INI, and the pass skips it), so a core left over for one falls to the ordinary orphan tiers.

Log markers (one per core, in the pass log)

Recovering an archived core

The core was unloaded before the move, so Solr has no registry record — re-add with CREATE (not RELOAD), which re-registers the existing directory without touching index files. For a site whose solr_integration_module directive is set, moving the directory back is enough: the next four-minute pass registers it itself (see re-registering a core Solr stopped listing). The curl below is the manual form for any other case:

port=9099                          # 9077 for Solr 7
core="oct.o1.example.com"
ts="20260418-222802"               # timestamp prefix from the backup dir name
bkp="/var/backups/solr9/${ts}-${core}"
dest="/var/solr9/data/${core}"     # /var/solr7/data/ for Solr 7

mv "${bkp}" "${dest}"
chown -R solr9:solr9 "${dest}"     # solr7:solr7 for Solr 7
curl "http://127.0.0.1:${port}/solr/admin/cores?action=CREATE&name=${core}&instanceDir=${dest}"

Pruning old archives

Archived cores are never auto-pruned — only the pass logs under /var/backups/solr/log/ are cleaned on each run. The archive dirs /var/backups/solr7/ and /var/backups/solr9/ therefore grow over time, so reclaiming that space is a manual task. Review the archive listing and remove entries archived more than ~90 days ago whose indexes are also very old — check each entry's index age before deleting anything, so a recently-archived core you may still want to recover is never swept up.

Per-core health check

_check_solr_core_health runs every pass (not throttled), once per running instance (solr7 on 9077, solr9 on 9099, gated on /etc/init.d/solrN). It queries the admin STATUS API read-only via python3 and makes no changes — it is the diagnostic feeding the optimise/orphan decisions. It logs:

• HEALTH-ERROR — no response from the Solr instance on its port (daemon down or unreachable), or the STATUS JSON failed to parse. The most operationally useful marker — it means the health check never got a reading.
• HEALTH-INIT-FAIL — a core reporting initFailures (failed to load).
• HEALTH-WARN — segment count > 50, deleted docs > 20% of maxDoc, or index size > 500 MB.
• HEALTH-INFO — N cores registered (or no cores registered), the per-instance roll-up line.
• HEALTH-SKIP — python3 not installed, check skipped.

Index optimise

_run_optimize_if_due runs last in each pass (throttled 12 h). It decides per core from the deleted-doc ratio whether to run expungeDeletes or a background full optimise; the ratio thresholds and the reason protected cores never get a full merge are covered in Overview and tuning. Every decision is logged one marker per core — grep ^OPTIMIZE- in the newest pass log:

Throttle sentinels and forced maintenance

There is no per-site state cache (older docs referenced /var/xdrago/.solr-state/<sitename>.state; that path does not exist). The real state is the throttle sentinels:

Force an immediate orphan sweep or optimise on the next run by removing its sentinel:

rm -f /var/backups/solr/.orphan_cleanup_last_run.pid   # force orphan cleanup
rm -f /var/backups/solr/.optimize_last_run.pid         # force index optimise
bash /var/xdrago/manage_solr_config.sh

Logs

Each pass writes its full stdout/stderr (the === section markers and all ORPHAN-* / HEALTH-* lines) to a timestamped file:

/var/backups/solr/log/solr-<YYMMDD-HHMMSS>.log

Files older than a day are pruned at the start of each pass. The cron line discards cron-level output to /dev/null, so the per-pass file is the only persisted record — there is no /var/log/boa/solr_management.log or /var/xdrago/monitor/log/solr-*.log (older docs cited those; no code path writes them).

tail -f "$(ls -t /var/backups/solr/log/solr-*.log | head -1)"

Custom-config interaction

With solr_custom_config = YES, the daemon skips re-deploying schema.xml/solrconfig.xml; operator config in sites/<domain>/files/solr/ stays in place; all other lifecycle behaviour (deletion on INI removal, solr.php refresh, optimisation) still runs. To deploy a Drupal-generated config without permanent lock-in, set solr_custom_config = NO for one pass, then restore YES (see Overview and tuning).

A config refresh — a template family moved forward, or a tenant upload applied from sites/<domain>/files/solr/ — leaves a one-shot marker (conf/.just-updated.pid) that makes the same pass rewrite sites/<domain>/solr.php and RELOAD the core once; the pass consumes the marker as it reloads. Until 2026-09-11 the upload path never cleared it (the upload dir is emptied by the copy, so that branch was never entered again), and every later pass rewrote solr.php and reloaded the core, forever.

Failure modes

"Create core" never completes — Solr daemon down. service solr9 status; tail -50 /var/solr9/logs/solr.log; restart, and the next pass retries.

Core not deleted after INI removal — deletion fires within 15 minutes once the INI no longer names a recognised module. If still present after that, force a run and read the newest pass log; under load over _O_LOAD_MAX the site may not have been reached yet.

Orphan core not archived yet — cleanup is deliberately patient: a core is archived only once past its tier threshold (14 d unreferenced, 60 d vhost+alias), the sweep runs at most every 6 h, and solr_integration_module-managed cores are age-immune. Read the newest pass log — the ORPHAN-FRESH/ORPHAN-SKIP/ ORPHAN-CANDIDATE lines state exactly why each core was kept or archived. Force a sweep by removing the throttle sentinel. Archived cores are moved to /var/backups/solrN/, never deleted, so a wrongly-archived core can always be restored.

Source location

Deployed: /var/xdrago/manage_solr_config.sh (read-only). Source: aegir/tools/system/manage_solr_config.sh — modify there to upstream a fix.

Related

• Overview and tuning — the INI variables this daemon enforces and the optimisation internals.
• Solr 9 modules — the module set the daemon keeps deployed.
• Reference appendix — Solr site-INI variables.
• Discontinued features — the Solr-4-era manual core-management flow this daemon replaced.