Request guards
The request guards are the real-time half of the Abuse Guard. Where
scan_nginx is a post-hoc log scorer that decides
who to ban, the guards are a wall of map/geo directives in the rendered
vhost config that classify every incoming request and drop or downgrade
hostile ones before they reach PHP-FPM, at effectively zero backend cost.
The classification keys on:
- client IP
- URI shape
- User-Agent
- Referer
- query string
They live in three places:
server.tpl.php(provision-private) — themap/geodefinitions in the sharedhttp {}block (rendered once per box). Each map turns a request attribute into a flag variable.Inc/vhost_include.tpl.php(provision-private) — theif (…) { return … }enforcement rules in eachserver {}block (rendered per vhost). These read the map variables and act. The other vhost shapes — the subdir location, the Grav location set, both Textpattern vhosts and the wild-ssl front — never include that file, so each restates the guards it needs for itself./etc/nginx/conf.d/limit-req-zones-boa.conf(written by BOA, not by Provision) — thelimit_req/limit_connzones plus the maps declared here rather than in the master render, so that a consumer can gate on this file:$is_amp_chain, a static inline map with no include and nothing feeding it, and the$boa_fleet_*chain, whose threeincludefragments are written by the per-minutenginx_fleet.sh. Every consumer of those checks that this file declares the map before rendering itsif, which is what makes the two halves order-independent.
Because the maps are evaluated lazily and the if checks are cheap string tests,
this whole layer runs before any try_files, any @drupal fallback, and any
FastCGI round-trip.
The 444-vs-404-vs-429 convention
BOA uses three distinct refusal codes deliberately, and the choice is load-bearing:
return 444— Nginx's "close the connection, send no response". Used for abuse denials: a banned IP, a malformed asset-chain flood, a no-referer search probe, a forged/training AI crawler, a scanner pattern, a foreign-CMS admin probe, a TLS handshake on the plain port. It gives the attacker no signal (no status line, body, or timing leak) and is the cheapest refusal. It also feedsscan_nginx's per-IP444-weight, so a 444'd request both costs the attacker a connection and accrues score towards a ban. The base444"close without response" semantics are on Rewrites & locations.return 404— a normal, cacheable "not found". Reserved for the cheap content-shape misses where a recoverable error keeps the false-positive blast radius small: the node-chain / lang-chain / content-chain URL-mutation floods, the no-referer print, flag-toggle and HybridAuth-window gates (their traffic class includes search crawlers, which surface a 444 as a 5xx), and the special.php-probe URLs. A 404 still avoids a PHP bootstrap, so it is nearly as cheap as 444 but safer when the pattern could (rarely) match real content.return 429— "too many requests", used by exactly one guard: the crawler-fleet refusal below. It is chosen for what it is not: noscan_nginxscorer counts a429, so a fleet refusal can never feed the per-IP ban score, the UA-burst CSF ban or the i18n shedding signal — a guard aimed at a class that spans hundreds of addresses must not convert its own volume into bans. It is equally php-fpm-free, and unlike444it does not reach a Cloudflare-proxied visitor as a520.
A note on
return 403.vhost_include.tpl.phpalso emitsreturn 403in several places — but these are not abuse denials. Each is anif ($cache_uid = '')unauthenticated-session gate on a Hostmaster/admin*or/hosting/c/server_*location: an anonymous (no session cookie) request to an admin URL gets403, while a bot in the same block gets444. In short: 403 is reserved for admin/Hostmaster session gates; abuse denials use 444.Search engines and the login paths. Verified search-engine crawlers routinely walk the slashless
/userand/user/loginforms into the bot guard: stock Drupal 7 ships a robots.txt that disallows only the trailing-slash forms (/user/login/), so the bare forms stay crawlable, and a busy site can log hundreds of bot-guard444s a day to genuine crawler ranges. That is the guard working as designed — the refusals cost nothing on the box and do not affect ranking; they only surface as crawl-error noise in the engines' consoles. The remedy belongs on the site, not in the guard: a site-side robots.txt override (sites/<site>/files/robots.txt, served ahead of the platform file) carryingDisallow: /userremoves those fetches at the source.
Keying on the real client (realip)
Every guard that tests the client IP — and scan_nginx itself — keys on the
true client address, not on a spoofable X-Forwarded-For. On
Cloudflare-fronted vhosts BOA plumbs the realip module in the shared http {}
block:
real_ip_header CF-Connecting-IP;
real_ip_recursive on;
include /data/conf/nginx_cloudflare_real_ip.c*;
The trusted CF source ranges are supplied by the BOA-managed wildcard include —
written and refreshed by cloudflare_realip.sh — so a missing file never breaks
nginx -t; with no trusted ranges the CF-Connecting-IP header is ignored and
$remote_addr is left unchanged (no spoofing risk).
After realip runs, $remote_addr is the real visitor, which is what the
$is_banned geo and the IP-counting in scan_nginx both score.
One subtlety on the FastCGI side: BOA pins
fastcgi_param REMOTE_ADDR $realip_remote_addr;
so the PHP global sees the original TCP peer (the CF edge), while Nginx's own
$remote_addr stays realip-rewritten to the real client for rate-limit keys, logs
and the deny geo. This keeps Provision's own PHP-side real-client resolution
correct and the Nginx-side guards correct at the same time.
This is the
request-path counterpart of
scan_nginx's real-client resolution;
the full CF range refresh and the per-vendor realip plumbing are on
Edge policy.
$is_banned — the closing link of the pipeline
The closing link of the ban pipeline is a geo keyed on the
realip'd client:
geo $remote_addr $is_banned {
default 0;
include /data/conf/nginx_banned_ips.c*;
}
enforced near the top of every vhost:
if ($is_banned) {
return 444;
}
This closes the loop: nginx_deny.sh regenerates
/data/conf/nginx_banned_ips.conf from the current CSF state, the wildcard .c*
include picks it up on the next reload, and the next request from a banned
client is 444'd at zero backend cost. The same geo also includes
nginx_banned_ips.conf6, written by nginx_deny6.sh from the nginx-native IPv6
ban store — csf is IPv4-only, so IPv6 offenders (only reachable via the trusted
realip proxy) are banned here at Nginx and 444'd by this identical guard.
Two safety properties matter here:
- Absent/empty file is safe. With no entries
$is_bannedstays0, so a fresh box or a cleared ban list never errors. - The
.c*glob is leading-dot-safe.nginx_deny.shwrites its in-flight and rollback copies as dot-prefixed names —.nginx_banned_ips.tmp.$$and.nginx_banned_ips.last_good.conf— precisely so the.c*include never picks up a half-written temp or a backup. Only the finalnginx_banned_ips.confis matched.
Because the deny is keyed on the realip'd $remote_addr, it bites a
Cloudflare-proxied attacker at the origin's Nginx — where an origin
CSF/iptables ban on a CF-fronted IP would only ever see the CF edge and miss.
Every vhost shape carries the guard. The full-domain vhost include is not
the only place it has to appear: the standalone subdir server and both
Textpattern vhosts (plain and SSL) never pull that include in, so each restates
if ($is_banned) { return 444; } for itself, and the Grav location set and the
wild-ssl front carry their own copies. On the wild-ssl proxy path the copy is
the one that bites at all — the port-80 vhost behind it sees peer 127.0.0.1
and can never match, while at the front $remote_addr is the real visitor.
$boa_fleet_block — crawler-fleet fingerprint → 429
Every guard above refuses a request by its shape, and the ban pipeline
refuses by address. A distributed crawler fleet defeats both: it rotates
hundreds of addresses, so no per-address scorer ever sees more than a few dozen
requests from any one of them, and its requests are ordinary GETs for real
pages, so no shape map matches them.
What such a fleet cannot rotate is the thing it shares. One exact user agent,
walking one route class of one vhost, from many addresses at once, without a
Referer, pulling a different URL on almost every request. That combination is
the fingerprint this guard refuses — and the shared string alone can never be
the key, because real browsers send stock strings too.
/var/xdrago/nginx_fleet.sh reads the last few minutes of access.log every
minute, declares a (vhost, user agent) fingerprint when one route class passes
every gate, records the addresses seen acting as members, and renders three
exact-match include fragments. The generator, its store and its reload discipline
are on
the ban pipeline;
the gates and thresholds are on
Configuration reference.
This section is the consuming side.
The map chain
The maps are declared in the BOA-written http-scope file
/etc/nginx/conf.d/limit-req-zones-boa.conf, not in the master render:
map $http_user_agent $boa_fleet_uaid {
default "";
include /data/conf/nginx_fleet_ua.c*;
}
map $remote_addr $boa_fleet_net {
default $remote_addr;
"~^([0-9]{1,3}\.[0-9]{1,3})\." $1;
}
map "$host|$remote_addr|$boa_fleet_uaid" $boa_fleet_addr {
default 0;
include /data/conf/nginx_fleet_addr.c*;
}
map "$host|$boa_fleet_net|$boa_fleet_uaid" $boa_fleet_nt {
default 0;
include /data/conf/nginx_fleet_net.c*;
}
map $http_referer $boa_fleet_noref {
default 0;
"" 1;
}
map $http_cookie $boa_fleet_anon {
default 1;
~SESS[[:alnum:]]+=[[:graph:]] 0;
"~*(?:^|;\s*)grav-[-_a-z0-9]{0,32}-(?:[0-9a-f]{7}|[0-9a-f]{32})-admin=" 0;
"~(?:^|;\s*)txp_login(?:_public)?=[[:graph:]]" 0;
}
map "$boa_fleet_nt$boa_fleet_addr$boa_fleet_noref$boa_fleet_anon" $boa_fleet_hit {
default 0;
"~^1" 1;
"0111" 1;
}
map $boa_fleet_uaid $boa_fleet_block {
"" 0;
default $boa_fleet_hit;
}
The four bits composed into $boa_fleet_hit are, in order: member network,
member address, no Referer, anonymous. A network member is refused outright
(~^1); an address member needs the last two as well (0111).
The first lookup is the cheap one: an agent that is not in the rendered
fragment leaves $boa_fleet_uaid empty, $boa_fleet_block is 0 by the last
map's "" key, and the rest of the chain is never evaluated.
Two scopes, deliberately asymmetric
The generator classifies each declared agent and the maps enforce that classification:
| Scope | Which agents | Refused per | Conditions |
|---|---|---|---|
| Network | a self-declared crawler name that is not browser-shaped (bot, crawl, spider, slurp, scrap, headless or a bare URL in the string) |
member /16 (IPv6: per address) |
any Referer, any cookie — no human browses with such an agent |
| Address | browser-shaped agents, and every agent that does not name itself a crawler | member address | only without a Referer and only when no session cookie is present |
The asymmetry is the whole false-positive design. A real visitor who happens to
share a member address (carrier NAT, an office egress) or the exact browser
string passes by doing anything a browser does: following any link sends a
Referer, and logging in sets a session cookie. The cookie test recognises a
Drupal/Backdrop SESS-family cookie with a value, a Grav admin cookie and the
Textpattern login cookies.
Keys are exact strings, never a regex built from a user agent — nginx
compares exact map keys case-insensitively, which is why the generator stores
the lower-cased agent and refuses to emit any agent outside a strict printable
ASCII grammar. Because a generated map can hold thousands of exact keys, the
master render sets map_hash_max_size 32768; without it every configtest and
reload warns that it could not build an optimal map hash.
Where it fires
The guard renders only where the zones file declares the chain — every
consumer tests for the exact line map $boa_fleet_uaid $boa_fleet_block {
before emitting anything, so no delivery order can produce a config that
references an undeclared variable:
- the full-domain vhost include, immediately after the
$is_bannedguard (so a banned address still gets its444); - the subdir location;
- the Grav location set;
- both Textpattern vhosts, plain and SSL;
- the wild-ssl front, inside a marker-delimited block that a Barracuda pass
strips again if the zones file ever stops declaring the maps. It has to be
there for the same reason the ban guard is: the fleet maps key on
$remote_addr, which the port-80 vhosts only ever see as127.0.0.1on that proxied path.
Absent fragments are safe in both directions: the .c* globs then match
nothing, every lookup takes its default, and $boa_fleet_block is 0.
What it catches, measured
On one hosted site's anonymised two-day log (439,597 lines) with both scopes refusing:
| Cohort | Refused |
|---|---|
| a fleet sending a self-declared crawler name | 76,776 of 76,972 requests |
| a fleet rotating two stock desktop-browser strings over cloud addresses | 37,474 of 40,001 on the one string, 36,688 of 39,295 on the other |
| every other cohort on that vhost | 0 |
| the 114,296 requests on the box's other vhosts | 0 |
A 766,782-line synthetic battery — a busy general-audience site, a campus audience behind one egress, a mobile app backend, newsletter bursts and an AI assistant fan-out — produced zero non-fleet refusals.
What it does not catch
The refusal is narrow by design, and four shapes walk past it.
- A fleet that fakes a same-site
Refereron every request. The address scope requires the absence of one. - A fleet that sends one request per address on a current browser string — there is nothing to declare it from and no member to record.
- A fleet that uses a new agent per address. The fingerprint is the shared string; without sharing there is no fingerprint.
- Anything below the scale bar — the request, address, share and uniqueness gates are set so that ordinary traffic cannot reach them.
The collateral of the steering is bounded rather than absent: an address that
acted as a member stays a member for _NGINX_FLEET_MEMBER_TTL (six hours by
default) even if the fleet stops, and during that window a Referer-less,
cookie-less request from it carrying that exact agent is still refused.
Chain-mutation flood maps
A distributed botnet that exploits broken relative-URL resolution appends Drupal asset references onto deep content URLs, producing self-mutating chains. BOA classifies the family with purpose-built maps, split by whether the mutated URL ends in a static asset (444) or a content segment (404).
$is_static_chain → 444
if ($is_static_chain) {
return 444;
}
It matches any of:
- a Drupal asset-dir marker (
sites/all/modules,ui/external, …) buried under a content path, - a canonical Drupal core asset file (
system.base.css,drupal.js, …) buried the same way, or - the same asset-dir token repeated.
Legitimate Drupal asset URLs are root-anchored, so these can only be the
broken-relative-URL flood. The map is validated against 44k real flood
requests with zero false positives on root-anchored assets, aggregated files,
image styles and /system/files private files.
The 444 fires before the /(?:external|system)/ asset router would route the
absent file to @drupal → /index.php → php-fpm.
$is_content_chain → 404
The content-path twin: the same mutation, but the URL ends in a content segment (no static asset), so without a guard it falls through to Drupal and renders a full themed page (200).
if ($is_content_chain) {
return 404;
}
It matches only when both signals hold:
- a Drupal code-dir marker (
sites/all/modules,modules/system,ui/external…) appears as a path segment, and - some path segment repeats 3+ times (the relative-URL accumulation signature).
It is deliberately conservative — it covers the clear majority of the variant,
not the 2x-repeat tail — and uses a cheap 404 rather than 444 because these
are content URLs where a recoverable error keeps the false-positive blast radius
small. The complete cure is a source-side <base href>/theme fix that stops the
site emitting root-relative-without-leading-slash links.
$is_amp_chain → 404
The query-side cousin: a crawler that HTML-escapes every link it re-follows
turns each & into &, then &amp;, one layer per hop, so its query
keys grow into amp;amp;page (or the percent-encoded amp%3Bamp%3Bpage) and
every hop renders another uncached page.
map $args $is_amp_chain {
default 0;
"~*(?:^|[&;])(?:amp(?:;|%3b)){2}" 1;
}
It matches two or more consecutive amp; layers, in either spelling and
any case, at a query-key boundary. A single & left by a badly escaped
newsletter or CMS link is still served. Blocking at two layers also stops the
recursion: the deeper URLs are only ever discovered from pages served at the
shallower one.
Unlike the other chain maps, this one is declared in the BOA-written http-scope
file /etc/nginx/conf.d/limit-req-zones-boa.conf, and the vhost consumer
renders only when that file declares the map, so no Barracuda/Octopus delivery
order can reference an undefined variable.
Deliberate omission on subdir vhosts
The static and content chain guards apply on full-domain vhosts only. They
are intentionally not present in subdir.tpl.php: a subdir site
legitimately serves /<subdir>/sites/all/... assets, which $is_static_chain
would match as buried-under-content. The node-chain, lang-chain and amp-chain
guards (which match on node/<id> repetition, language-prefix runs and the
query, not asset paths) do still apply on subdir vhosts.
Print no-referer gate → 404
A printer-friendly or email-this-page request is always a click from a page, so
it carries a Referer; a Referer-less hit to a /print* path is the distributed
botnet (100% of the observed flood had no Referer). The gate composes
$is_print_path (a /print… URI shape, anchored on a numeric node id or an
export-format segment) with $has_no_referrer:
map $is_print_path$has_no_referrer $block_print_no_referer {
default 0;
"11" 1;
}
enforced as if ($block_print_no_referer) { return 404; } — a static 404, not
444, because the no-Referer class also contains search crawlers following linked
print pages: a 444 reached them as Cloudflare 520 / proxy 502 and filled Search
Console with 5xx noise, while a static 404 is equally php-fpm-free and the right
crawl outcome for duplicate print content. It is referrer- and path-shape
only — no module or version detection — so it is FP-safe and
version-agnostic: it covers D7 print / print_mail / print_pdf / printer_and_pdf,
D10+ entity_print + printable, and Backdrop, while content slugs
(/printing-services, /print/about-us, /printable-maps) never match.
Flag-toggle no-referer gate → 404
The Flag module exposes GET action links (/flag/flag/<name>/<id>,
/flag/unflag/<name>/<id>) on every rendered page, and each hit is an
uncacheable full Drupal bootstrap answered 302 — a status the log-scoring IDS
never counts. One observed flood drove ~11k such bootstraps a day into a shared
FPM pool from ~7k IPs at a median of one request per IP under rotated
browser-family UAs, a large share carrying HTML-entity-mangled tokens
(?destination=&%3Btoken=…) — links scraped from raw HTML that no real
browser produces. A real flag click always carries a same-origin Referer, and a
Referer-less toggle would fail the anonymous CSRF token check anyway,
bootstrapping only to redirect — so blocking it removes cost, not function. The
gate composes the toggle path shape with $has_no_referrer, keyed on the request
method so only GET matches:
map $uri $is_flag_toggle {
default 0;
"~*^/(?:[a-z]{2}(?:-[a-z]+)?/)?flag/(?:flag|unflag)/[a-z0-9_]+/[0-9]" 1;
}
map $request_method$is_flag_toggle$has_no_referrer $block_flag_no_referer {
default 0;
"GET11" 1;
}
enforced as if ($block_flag_no_referer) { return 404; } — static 404 for the
same crawler-safety reasons as the print gate above. The tail is anchored to a
machine name plus a numeric entity id, so content slugs (/flag-day,
/flag/flag-history) never match, and POST/AJAX-form flagging flows are
untouched. It works on D7 Flag 2/3 and D8+ Flag 4, whether or not the module is
enabled. To see the gate working, count 404s on /flag/ paths in the vhost
access log — each one is a bootstrap that never reached php-fpm.
The same $has_no_referrer map also feeds the search-amplification family below.
HybridAuth-window cold-fetch gate → 404
The HybridAuth module renders social-login links
(/hybridauth/window/<Provider>) on every page that shows its login block or
comment form, and each hit is an uncacheable full Drupal bootstrap that
starts a session and answers 302/200. One observed flood followed these
links from thousands of rotating residential-proxy IPs at a median of one
request per IP — the same scraped-link class as the flag flood, with the twist
that the bots ran headless browsers and partially executed the OAuth popup
flow, burning up to three bootstraps per sequence.
Referer alone cannot gate this path, and that is why the gate also tests the
session. The window path is not only the entry point: the module passes it as
hauth_return_to, so the provider's callback at /hybridauth/endpoint
redirects the browser back to /hybridauth/window/<Provider>, and only
that final hop runs the account match/create, the login and the popup-close
page. A 302 carries the original request's referrer forward rather than
substituting the redirecting URL, so whenever the provider strips the Referer
— a policy the operator can neither see nor control — the login-completing hop
arrives Referer-less. Measured on a hosted box, the completion-shaped window
200s were overwhelmingly Referer-less, so a Referer-only gate would 404 real
logins, silently, with no PHP-side trace.
What every real hop does carry is the Drupal session cookie: the outbound leg starts the session holding the hauth state, so the return hop cannot work without it. A cold scraper following a scraped link carries neither a Referer nor a session. The gate fires on that intersection only:
map $cache_uid $has_no_session {
default 0;
"" 1;
}
map $uri $is_hybridauth_window {
default 0;
"~*^/(?:[a-z]{2}(?:-[a-z]+)?/)?hybridauth/window/[a-z0-9_.-]+/?$" 1;
}
map $request_method$is_hybridauth_window$has_no_referrer$has_no_session $block_hybridauth_no_referer {
default 0;
"GET111" 1;
"HEAD111" 1;
}
enforced as if ($block_hybridauth_no_referer) { return 404; }. $cache_uid
is the same map the cache-bypass gates use; any SESS/SSESS cookie sets it,
including an anonymous session — exactly the mid-flow case. The tail is
anchored to a single path segment (the provider name) with an optional trailing
slash, so deeper paths and content aliases never match; HEAD is included
because a HEAD costs the same bootstrap and no login hop is ever a HEAD.
/hybridauth/endpoint stays ungated — it is the provider's own callback
target. It works whether or not the module is enabled. Together with the flag
and print gates, the 404s this gate emits are the tell
Detector 6
counts.
TLS-on-plain → 444
map $request $tls_on_plain {
default '';
~*^\x16\x03 tls_on_plain;
}
matches a TLS ClientHello frame (record type 0x16, TLS version 0x03…) arriving
on the plain HTTP port, enforced as if ($tls_on_plain) { return 444; }. It
silently drops a TLS handshake mistakenly or maliciously sent to port 80 instead
of returning an error that would feed scanner automation. Shipped in
BOA-5.9.3.
$is_cms_probe — foreign-CMS admin probes → 444
if ($is_cms_probe) {
return 444;
}
map $uri $is_cms_probe matches WordPress / Joomla / phpMyAdmin path tokens
that can never exist on a Drupal / Backdrop / Ægir-Hostmaster docroot, on
any UA:
wp-(admin|login|content|includes|json|config|cron|signup|mail|register|links-opml|trackback|comments-post),
administrator and phpmyadmin.
Each token matches only as a whole path segment: the wp-* and phpmyadmin
tokens must be followed by /, ., ? or end-of-path, and administrator by
/ or end only — so a legitimate alias like /wp-content-strategy or
/site-administrator does not match (nor does /administrator.php).
The guard exists because of the FPM sink these probes used to hit: the
extensionless variant (/cms/wp-admin, /ru/administrator) misses every
static location, falls through try_files → @drupal → /index.php → php-fpm,
and pays a full Drupal bootstrap just to render a 404 — the exact sink that
let a distributed auth-probe flood saturate a small VM's FPM pool.
The 444 drops the probe pre-bootstrap and is scored by scan_nginx's per-IP
444-weight (the 301/extensionless-404 routing these requests previously hit was
not scored), so offenders now accrue IDS score with every probe.
Two deliberate omissions:
admineris excluded — BOA ships Adminer.- Generic auth words (
login,signin,admin,user,account) are not matched — they collide with real customer URL namespaces and with Drupal's own/adminand/user. The distributed tail that probes them is handled in aggregate by thescan_nginxUA-burst detector instead (scan_nginx scoring).
There is no operator knob and no opt-out. The guard takes effect on a vhost once its templates are re-rendered after the Provision update.
Bot, crawler and botnet maps
Several UA-keyed maps hard-block known-bad agents:
| Map | Variable | Enforcement |
|---|---|---|
$is_crawler |
named scraper/SEO/abusive bots (Ahrefs, MJ12, Semrush, PetalBot, serpstatbot, HTTrack…) | if ($is_crawler) return 444 |
$is_botnet |
semalt/kambasoft referrer-spam family | if ($is_botnet) return 444 |
$is_bot |
generic crawler tokens (crawl, bot, spider, google, bing, …) |
return 444 on private, robots-disallowed and callback locations only (search, /user/login, admin, AJAX/batch callbacks, private files, /bgp-start/, …), never on content, asset, feed or sitemap locations, which crawlers must reach; also part of the Speed Booster cache key |
Legitimate search and preview crawlers (Sogou, Pinterest, TikTok) and the
generic Go-http-client library token are deliberately not in
$is_crawler: a hard 444 on them blocks real user-facing services, not
scrapers.
AI-vendor traffic is classified separately by the $is_ai_* maps and the
per-class AI policy — those tokens are deliberately kept out of $is_crawler so
they don't bypass that policy. See
Edge policy.
A separate $deny_on_high_load UA map (the same roster as $is_bot:
crawl/bot/spider/tracker/click/parser/google/yahoo/yandex/baidu/bing) is the
load-shedding variant: it answers those agents 503 only while Spider
Protection is armed, i.e. while load per CPU is above _CPU_SPIDER_RATIO (2.1
by default). Below that load a declared crawler, or anything borrowing a
crawler's name, is served like any visitor.
Stale-Chrome botnet detection
Chrome auto-updates aggressively, so a genuine consumer install more than ~12
months stale is extremely rare. Search-amplification bots fake a "moderately
outdated but not obviously fake" Chrome UA to dodge $is_bot while still being
detectably stale:
map $http_user_agent $is_stale_chrome {
default 0;
~*Chrome/1(?:[01][0-9]|2[0-79]|3[0-79])\. 1; # Chrome/100–139 minus 128 and 138
}
$block_stale_chrome_search combines a stale Chrome UA with fulltext/facet search
params and fires only in search location blocks (so no impact on non-search
requests from the same UA class). The standalone $is_catalina_stale_chrome
matches the Mac UA shape (Mac OS X 10_15_7) at a stale Chrome version — the shape
every confirmed Solr search-amplification bot has presented — and is applied
directly in the /search blocks, so it needs no $has_fulltext_search dependency.
Chrome and Safari freeze that platform token on every macOS release, so it does
not identify Catalina itself: the stale version is what makes the match safe.
Chrome/128 and Chrome/138 are carved out of both maps: they are the last releases
for macOS 10.15 and macOS 11, so a Mac pinned there for life is the one genuine
browser that still presents a stale major, and it keeps site search. Both maps
shipped in BOA-5.9.3.
Maintenance caveat (carry verbatim). These dated regexes are self-flagging. The in-source note instructs: move the upper bound by release date, never by counting versions — Chrome shipped a major every ~4 weeks until Chrome/153 (2026-09-08) and every ~2 weeks since. Widen to the newest major whose stable release is more than 12 months old (Chrome/139 reached stable on 2025-08-05), carve out the last major any macOS is pinned at, and keep both maps on the same pattern. The ceiling must move forward as Chrome versions age, or the maps stop catching this botnet class; it must never pass a version released within the last 12 months, or they start matching current browsers (false positives).
Scanner-pattern maps: $is_denied / $ua_denied
Two maps scan the request for attack payloads and 444 it. $is_denied (keyed on
$args) is value-scoped — each pattern is anchored to a single query-string
parameter value ((?:^|&)[^=&]+=…) to avoid base64/aggregate false positives —
and covers:
- SQLi:
union…select,select…from/where,insert…into,delete…from(with whitespace /%20/%2B//**/variants); - blind/timing:
waitfor delay,declare @,benchmark/sleep/pg_sleep(; - hex-literal (
0x…after=/char/cast/convert) and comment-obfuscated SQLi (/**/after a SQL keyword); - XSS:
<script,%3Cscript,javascript:,vbscript:,data:text/html,onload=,document.cookie(raw and percent-encoded); - PHP-source probes (
.php?…src/source/highlight); - shell injection (
system(); - path traversal raw and single/double percent-encoded (
../,%2e%2e/,%252e%252e/).
$ua_denied (keyed on $http_user_agent) catches the same WAITFOR/declare/
benchmark injection payloads when smuggled inside the User-Agent header
itself. Both shipped/expanded in BOA-5.9.3.
Search-amplification family
Solr / Search-API full-text search is expensive, so a botnet that hammers it (even
one request per IP) can amplify load far beyond its request rate. BOA defends the
/search and /user/login location blocks with a layered map family, all keyed
off $has_fulltext_search (matches search_api_views_fulltext,
search_api_fulltext, im_taxonomy_vid in the query string):
| Tier | Composed map | Signal |
|---|---|---|
| Tier 1 | $block_search_no_referrer |
fulltext params and no Referer |
| Tier 2 | $has_excessive_facets |
6+ facets (f[5]+), encoded or literal |
| Tier 2 | $block_search_root_referer |
fulltext and bare-root Referer and a facet present |
| login | $block_login_search_destination |
search payload in /user/login?destination= and no Referer |
These apply as return 444 inside the /search block, the language-prefixed
/xx/search block and the /user/login block, alongside limit_req
search-rate zones.
5.9.5 facet-required refinement. Tier 2's
$block_search_root_refereroriginally fired onfulltext + bare-root Refereralone — which falsely blocked a homepage plain-search submission (a real user submitting the search form from the front page sendsReferer: https://example.com/, a bare root, with no facets). The fix adds$has_any_facetas a required third signal, so the block now needsfulltext + root Referer + at least one facet param. The plain homepage submission has no facet and is no longer a false positive.
$block_login_search_destination closes a bypass: bots send
/user/login?destination=search%2F... so the request path is /user/login and
the /search guards never run. The map detects the URL-encoded search components
(apachesolr_search, search_api, im_taxonomy_vid) inside the destination=
value, combined with $has_no_referrer. The search-amplification family landed in
BOA-5.9.3.
Tier-A cap on anonymous localised concurrency (boa_i18n_anon)
Every guard above refuses requests by shape. This one bounds a request class by concurrency: a distributed scraper crawling localised pages drives each uncached page through expensive synchronous backend work, holding a PHP-FPM worker per request — and FPM pools are shared per account, so enough concurrent localised requests collapse every site on the pool.
The source spreads across thousands of IPs at one or two requests each, so per-IP limits never trip. The Tier-A cap therefore bounds the aggregate in-flight count of the class per vhost instead of chasing rotating IPs.
The shared http {} block declares
limit_conn_zone $boa_i18n_anon_key zone=boa_i18n_anon:10m plus three maps that
build the key:
| Map | Keyed on | 1 / ON when |
|---|---|---|
$boa_i18n_guard |
$host |
always, default 1 (ON) — per-host opt-out via the wildcard-included /data/conf/boa_i18n_guard.map* |
$boa_i18n_path |
$request_uri |
the URI starts with a two-letter language prefix — ~*^/[a-z][a-z](-[a-z]+)?/ covers /pt-br/, /zh-hans/ — or carries the D7 form ~*[?&]q=/?[a-z][a-z](-[a-z]+)?/ |
$boa_is_anon |
$cache_uid |
the session map is empty — no Drupal session cookie |
Three design points make the maps safe:
- Default-on is safe fleet-wide. An absent or empty
/data/conf/boa_i18n_guard.mapleaves every host guarded, because a leading two-letter path prefix is Drupal's URL language-negotiation convention, never a content subdirectory — the existing/[a-z][a-z]/searchand/[a-z][a-z]/civicrmlocations rely on the same convention. $request_uri, not$uri. Clean URLs are internally rewritten to/index.phpbefore the map is evaluated, so only the original request URI still carries the language prefix. The?q=pattern cannot match ordinaryq=node/orq=user/values — those are never exactly two letters followed by/.- Logged-in users are never capped.
$boa_is_anonreuses the authoritative$cache_uidsession map, so an editor working in/de/admin/…is invisible to the zone.
The composite key $boa_i18n_anon_key is $host only when all three flags read
111, otherwise empty — and empty keys are not counted. The cap is thus
per-vhost and constant-keyed: English, static, authenticated and opted-out
traffic never touches it.
Enforcement sits at location = /index.php — the single chokepoint every
dynamic request funnels through:
limit_conn boa_i18n_anon 24;
limit_conn_status 444;
The 24 comes from the Provision-side drush option nginx_i18n_anon_conn
(default 24; values below 1 are clamped back to 24) — a provision/drush option,
not a .barracuda.cnf _VAR. The in-source sizing note pegs 24 at roughly
1/8 of a 192-worker FPM pool. Static files under /xx/ are served by their own
locations and never reach this chokepoint, so they are correctly excluded.
The cap itself bans nobody: a shed request is answered 444 by Nginx and the
shed creates no CSF entry — nothing appears in csf -t / csf -g for the shed
as such (a heavy single IP can still accrue per-IP scan_nginx score from its
logged 444s via the normal 444-weight, a separate mechanism).
The windowed count of these 444s is also the earliest trip signal (the C444
threshold) for the log-side Tier-B i18n-flood detector in
scan_nginx.
Opt a vhost out by adding a "host" 0; line to
/data/conf/boa_i18n_guard.map and reloading Nginx.
One capacity cross-note: when this request class saturates a pool, raising
pm.max_children is not the cure — see
FPM capacity sizing.
Per-vhost cap on background-batch launches (bgp_flood)
The Drupal 7 background_process + background_batch module pair turns every
batch into a chain of HTTP POSTs the site sends to itself, at
/bgp-start/<handle>/<token>. Under a saturated pool each POST times out on
the client side (logged 499) while still holding a worker for the full request
wall, and the module re-dispatches processes it considers stale — so each pass
re-sends everything and the loop feeds itself. Because the source is the box's
own address, no edge ban can touch it: the firewall refuses to ban the host
itself, correctly. One observed storm ran roughly a hundred stale batches at
about a hundred POSTs each and drove a 48-core box to load 87, with the
database tier healthy throughout. It is a pure PHP-tier amplification.
bgp_flood bounds that class per vhost at 5 r/s with a burst of 50,
answering 444. The sizing sits well above the worst legitimate cadence: a
running batch re-launches itself roughly every ten seconds (sooner when its
memory guards end a pass early, so up to about 1 r/s in the heaviest case), an
open progress page re-dispatches at most once a second, and a cron fan-out is
absorbed by the burst. A storm demands an order of magnitude more, so it is
throttled to a trickle within seconds — measured on a test box at 386 r/s
attempted, 5.6 r/s admitted, the rest shed at the edge for no PHP cost at all.
Two properties are worth knowing before tuning it:
- A shed launch does not retry. The module dispatches fire-and-forget: it
opens the socket, writes the request and never reads the response, so it
cannot tell a
444from a200. That is exactly why the cap collapses a storm — every rejected re-launch permanently ends that chain. The flip side is the accepted trade-off: if a rejection lands on a legitimate batch that has no progress page open, that batch stops where it is, silently. The rate is set so this cannot happen at any modelled legitimate load; inside a real storm, ending the runaway chains is the point. update.php, drush and programmatic batches are unaffected. The module converts only progressive batches whose URL isbatch, so those paths never reach/bgp-start/at all.
Coverage details: only the exact two-segment shape /bgp-start/<handle>/<token>
is passed through — anything else under the prefix is answered 444 before
Drupal bootstraps, which is cheaper than the 404 it used to cost. A sibling
location covers the two-letter language prefix that multilingual sites prepend
(/pl/bgp-start/...), sharing the same per-vhost budget; longer prefixes such
as pt-br fall through unthrottled, the same limitation the /xx/search
guards have. Known bots are answered 444 in the rewrite phase, before the
counter is touched, so crawler noise cannot spend a site's budget.
The zone lives in the BOA-written /etc/nginx/conf.d/limit-req-zones-boa.conf
rather than the master's generated http config, because the per-instance vhost
include checks that file before rendering its limit_req lines. That check is
what makes the two halves order-independent: an instance that renders before
its master has the zone simply renders nothing and keeps the old behaviour,
instead of producing a config that references an undeclared zone — which would
fail nginx -t for every site on the box. Never delete that file while
any rendered vhost include references the zone.
This is the prevention half of the batch-storm work; the healing half is the
batch_guard monitor (Monitoring), and the two are
complementary rather than redundant: the cap sheds only re-dispatch walls
ABOVE its rate, while a sub-cap simmer is a real storm shape — an observed
recurrence re-launched ~27 looping bids at ~2.3 r/s aggregate for over 40
minutes, entirely under the cap, and only the guard can end that one. The cap
keeps a runaway wall from saturating PHP-FPM; the guard deletes the stale rows
any admitted flywheel feeds on.
Per-vhost cap on anonymous page renders (boa_perhost_anon)
The general sibling of the Tier-A i18n cap: a per-vhost limit_conn bounding
in-flight anonymous page renders at location = /index.php, default
100, shedding the excess with 444 (limit_conn_status). Like the
bgp_flood zone it is declared in the BOA-written
/etc/nginx/conf.d/limit-req-zones-boa.conf, not the master http config.
Anonymous-only by construction — a Drupal session cookie empties the key, so
logged-in editors are never shed (though the anonymous login POST itself is
counted). It counts requests in the location, not FPM occupancy.
Tune it per instance via the provision option nginx_perhost_anon_conn,
aiming at roughly 1.5× that instance's pool pm.max_children
(FPM capacity sizing). Known
limitation of the shipped default: on a small pool (16–28 children) FPM
saturates long before 100 in-flight renders, so the cap is effectively inert
there until tuned down toward the pool size.
Two composition facts for reading alerts correctly:
- Its 444s feed the i18n detector's early trip.
limit_conn_statusis one-per-context, so the 444s this general cap sheds on a multilingual vhost are indistinguishable in the logs from the Tier-A guardrail's — they count toward_NGINX_I18N_FLOOD_C444_THRESHOLD(configuration), which means a general, non-localised flood can trip the i18n detector's early path. - The two caps compose. A localised anonymous request consumes a slot in both zones and is bounded by the lower one (24 by default on the i18n side).
The edge-policy layer (defined here, documented separately)
Three further request-path defences are defined in the same server.tpl.php /
vhost_include.tpl.php pair but belong to BOA's edge-policy layer, documented on
Edge policy rather than redefined here:
- AI-class policy maps —
$is_ai_training,$is_ai_search,$is_ai_evasive,$is_ai_forged. Training and evasive AI fetchers are blocked by default (444) with a per-site opt-in; forged AI UAs (robots.txt-only tokens a real client never sends) are universally 444'd. - Secret-path deny —
map $uri $is_secret_path444's probes for.env/.git/.aws/.ssh,secrets.json,config.json,application.yml,settings.pyand similar, on any UA. - Cloudflare realip ranges — the trusted-range include refreshed by
cloudflare_realip.sh(see "Keying on the real client" above); per-site IP access control lives on the same Edge policy page.
These share this map/geo layer but are policy-configurable per site, so they
are documented with their own control files on
Edge policy rather than as fixed guards
here.
Where each guard fires (ordering)
Two guards run before the vhost is even reached on the proxied HTTPS path.
The wild-ssl front tests $is_banned (→ 444) and then $boa_fleet_block
(→ 429) at the door, because both key on $remote_addr and the vhost behind
the proxy sees only 127.0.0.1. Its own crawler deny ($is_crawler → 444)
follows them.
Within a vhost, the guards then run roughly in this order — earliest = cheapest / most universal:
$is_node_chain → 404
$is_lang_chain → 404
$is_static_chain → 444
$is_content_chain → 404
$is_amp_chain → 404 rendered only when the BOA zones file declares it
$block_print_no_referer → 404
SA-CORE-2018-002 RCE → 444
$is_banned → 444 ← ban-pipeline closing guard
$boa_fleet_block → 429 rendered only when the BOA zones file declares it
=PHP… version probe → 404
$is_secret_path → 444 edge-policy
$is_cms_probe → 444
$is_ai_forged → 444 edge-policy
AI training / evasive → 444 edge-policy
$is_crawler → 444
$is_botnet → 444
bad request method → 444
$is_denied → 444
$ua_denied → 444
$tls_on_plain → 444
… then per-location: /search, /xx/search, /user/login families,
and the Tier-A boa_i18n_anon cap at location = /index.php
Config-template tunables (5.9.3)
Two shared http {}-block tunables were adjusted alongside these maps and are
worth noting at the request-guard layer, though they are config rather than
guards:
variables_hash_max_size 2048— raised to accommodate the growing set ofmapvariables.fastcgi_cache_use_staleno longer includeshttp_503(it now readserror http_500 invalid_header timeout updating) — a503is no longer served from stale cache.
Related
- scan_nginx scoring engine — the post-hoc scorer that produces the bans these guards enforce.
- The ban pipeline — how
web.log→ guest-fire / guest-water → CSF →nginx_deny.sh→nginx_banned_ips.conffeeds the$is_bannedgeo, and hownginx_fleet.shrenders the$boa_fleet_*fragments. - Rewrites & locations — the base
return 444"close without response" semantics and the location-matching model. - Edge policy — the per-class AI bot
policy, Cloudflare realip range refresh, and secret-path deny that share this
map/geolayer but are policy-configurable per site. - Security & isolation — CSF + LFD firewall — the firewall lifecycle that consumes the scorer's output.
- FPM capacity sizing — why
raising
pm.max_childrenis not the answer to the abusive saturation the Tier-A cap absorbs.