ISP NOC · runbook v1 · 2026-10-30

Kahf DNS forwarder: troubleshooting runbook

For the ISP network operations center. Covers the customer-facing side of the Kahf DNS service: how to tell what is wrong, what you can fix yourself, and what to send us when you cannot.

Section 1

Quick facts

Print this section and keep it at the desk.

Item	Value
Kahf forwarders for ISP	198.51.100.10 and 198.51.100.11 (examples)
Protocols	DNS port 53, UDP and TCP
Who may query it	Customer ranges that Kahf has added to its allow-list, and the router's own address. Everyone else gets no reply at all (not an error).
Blocked-site answer	A record `203.190.10.123`, TTL 300 s. Other record types: empty, no error.
Allowed-site answer	Normal address, TTL as the site sets it (up to 300 s at the forwarder)
Redirect test address	`192.0.2.53` (documentation range, not a real DNS server)
Router script	Retrofit module `forwarder-router-dns-healthcheck.rsc` (the router already redirects port 53) or the full `forwarder-router-dns-v2.rsc` for a fresh router.
Traffic split	Router alternates between the two forwarders (nth round-robin). One down: all to the survivor. Both down: fallback list.
Router tags	`KAHF-DNS` `KAHF-VPN` `KAHF-TOR` `KAHF-HC`
Failover time (estimate from script settings)	about 3 to 4 s to mark a forwarder down, about 5 s to trust it again (retrofit module, 1 s probes, 3 misses / 5 hits); about 9 to 10 s each way with the full script (3 s probes)
Kahf contact (fill in)	name / phone / email / chat
ISP on-call (fill in)	name / phone

Two forwarders: run every dig and mtr for both. Commands in this runbook show one line per forwarder. A result that differs between the two is itself the finding.

Three rules. (1) A blocked site is a successful answer with the block-page address. (2) A timeout is always a fault. (3) When the router fails over, DNS keeps working but filtering is off; that is by design, not a reason to panic.

Section 2

Five-minute triage

Do the steps in order. Stop at the first step that explains the symptom. Run each dig three times before believing it.

Step 1 · Does the forwarder answer?

dig @198.51.100.10 google.com +time=2 +tries=1
dig @198.51.100.11 google.com +time=2 +tries=1

Result	Go to
Answer, status NOERROR	Step 2
Timeout, all 3 runs	Step 4
Timeout some runs	Faults F3 (one prefix) or F6 (slow)
One forwarder answers, the other does not	F1 (one dead) or F8 (prefix on one allow-list only)
status SERVFAIL / REFUSED	Escalate with the dig output

Pass: answer within 2 s

Step 2 · Is filtering on?

dig @198.51.100.10 pornhub.com +short
dig @198.51.100.11 pornhub.com +short
# expect: 203.190.10.123

Any other address means the site is not blocked at the forwarder. If it is a site that should be blocked, send us the domain (F7).

Pass: block-page address returned

Step 3 · Is the router redirecting?

dig @192.0.2.53 pornhub.com +short +time=2 +tries=1
# expect: 203.190.10.123  (an answer from a non-DNS address proves the redirect)
# timeout or a real IP: redirect rules missing or bypassed -> F4

Pass: block-page address returned

Step 4 · Forwarder silent: where does it break?

mtr -rwc 20 198.51.100.10
mtr -rwc 20 198.51.100.11
dig @198.51.100.10 google.com +tcp +time=2 +tries=1
dig @198.51.100.11 google.com +tcp +time=2 +tries=1
# from a host in a DIFFERENT part of your network too

Observation	Meaning
Loss starts inside your network	Your path. Fix routing/uplink first (F2).
Clean path, last hop dead, fails from everywhere	Forwarder problem. Check the router failed over (Section 5), then call Kahf (F1).
Works from other hosts, fails from one prefix	Prefix not on the allow-list (F3).

Section 3

Fault catalogue

Seven faults, in the order you are likely to meet them. Each says what customers see, why, how to confirm, and what to do.

F1 · A forwarder is down

Filtering stays on

Customers see: For about ~3–4 s a share of lookups fail or retry (the ones still sent to the dead forwarder). Then normal browsing. Filtering stays on.
Why: The router's probes to that forwarder failed 3 times; everything now goes to the other forwarder.
Confirm: /tool netwatch print where name~"kahf-hc" shows one forwarder down. /log print where message~"kahf-hc" shows "DOWN" and a state change. dig against each forwarder shows which one is silent.
Do: Nothing on the router. The forwarder returns by itself about ~5 s after it is healthy. Tell Kahf. If both forwarders are down: the router uses the fallback list: DNS works, filtering is off. Call Kahf.

F2 · Forwarders and fallbacks all unreachable

No DNS

Customers see: Nothing loads by name. Sites by IP work.
Why: Usually your uplink or upstream, not DNS.
Confirm: Ping each forwarder and each fallback. Check the default route.
Do: Fix the uplink. If you must restore DNS now and accept no filtering: /system script run kahf-hc-uninstall (Section 5).

F3 · One prefix gets timeouts, the router looks healthy

Silent

Customers see: One pool or area has no DNS. Others are fine.
Why: The forwarder ignores sources that are not on its allow-list. The router probe uses the router's own address, so the health check stays green and no failover happens.
Confirm: dig from a host inside the affected prefix times out; dig from a host in an allowed prefix works. Get the customer's public source IP (after CGNAT).
Do: Send Kahf the prefix (CIDR) and the time. We add it and restart the forwarder; allow about a minute. The router cannot fix this. Prevention: tell us before you bring a new pool or CGNAT range into service.

F4 · Redirect rules missing

Filtering silently off

Customers see: Nothing broken. Blocked sites open.
Why: After a router upgrade, restore or manual edit, the NAT rules are gone or disabled.
Confirm: Step 3 fails. /ip firewall nat print where comment~"KAHF-HC" returns nothing or shows disabled rules. Rule counters (print stats) do not move.
Do: Re-enable the rules, or re-import the script (it removes and rebuilds its own rules and address lists). Re-run Step 3.

F5 · Encrypted DNS or VPN refused

By design

Customers see: One app or a phone reports "can't connect to DNS" or "Private DNS server cannot be accessed". A VPN app will not connect.
Why: DoT (853), DoQ, DoH to known providers, common VPN ports and Tor ports are rejected on purpose.
Confirm: /ip firewall filter print stats where comment~"KAHF-DNS" counters rise when the customer retries.
Do: No change on the router. Android: Settings, Private DNS = Automatic or Off.

F6 · Slow answers

Degraded

Customers see: Pages start slowly; first load of a site is slow, repeats are fast.
Why: Loss or latency on your path to the forwarder, or trouble behind it. The probe allows 1 s; sustained slowness can flip the router between forwarder and fallback.
Confirm: mtr -rwc 20 198.51.100.10 and five timed digs (Section 4). Look at the log for repeated UP/DOWN lines.
Do: If loss is on your side, fix the path. If the path is clean, send Kahf the timed digs; we can see answer-time percentiles for ISP on our dashboard.

F7 · A site is blocked or open when it should not be

Policy

Customers see: Block page on a site they expect, or an open site they expect blocked.
Confirm: dig @198.51.100.10 <domain> +short. 203.190.10.123 = blocked. Anything else = allowed.
Do: Send Kahf the exact domain (not the full URL) and the time. After our confirmation wait 5 minutes, flush the device or CPE cache, and retest before re-reporting.

F8 · Prefix listed on one forwarder only

Intermittent

Customers see: One prefix has patchy DNS: about every other lookup is slow or fails, and a retry often works.
Why: The router alternates between forwarders. If the prefix is on the allow-list of only one of them, lookups landing on the other get no answer. Both probes stay green.
Confirm: From a host in that prefix, ten digs to each forwarder: one answers every time, the other never does.
Do: Send Kahf both dig results. We keep the two allow-lists identical, so this means a missed step on our side; we fix it and restart one forwarder at a time.

Section 4

Command reference

From a host in your network

# basic
dig @198.51.100.10 google.com +time=2 +tries=1
dig @198.51.100.11 google.com +time=2 +tries=1
dig @198.51.100.10 google.com +tcp +short
dig @198.51.100.11 google.com +tcp +short
# blocked?
dig @198.51.100.10 pornhub.com +short
dig @198.51.100.11 pornhub.com +short
# redirect works?
dig @192.0.2.53 pornhub.com +short +time=2 +tries=1
# timing, five runs
for i in 1 2 3 4 5; do dig @198.51.100.10 example.com +stats | grep "Query time"; done
dig @198.51.100.11 example.com +stats | grep "Query time"; done
# path
mtr -rwc 20 198.51.100.10
mtr -rwc 20 198.51.100.11

On the MikroTik

/tool netwatch print where name~"kahf-hc"
/log print where message~"kahf-hc"
/ip firewall nat print where comment~"KAHF-HC"
/ip firewall nat print stats where comment~"KAHF-HC"
/ip firewall filter print stats where comment~"KAHF-DNS"
/ip firewall connection print where reply-src-address~"198.51.100.10:53"
/ip firewall connection print where reply-src-address~"198.51.100.11:53"
/system scheduler print where comment="KAHF-HC"
/system script print where name~"kahf-hc"
/ping 198.51.100.10 count=5
/ping 198.51.100.11 count=5

Test every protocol

One row per protocol or behaviour, with the ideal result. Windows commands use nslookup and PowerShell; use curl.exe (with the .exe) in PowerShell.

Check	Windows	Linux / macOS	MikroTik	Ideal result
Reachability (ICMP)	ping -n 10 198.51.100.10 ping -n 10 198.51.100.11	ping -c 10 198.51.100.10 ping -c 10 198.51.100.11	/ping 198.51.100.10 count=10 /ping 198.51.100.11 count=10	Replies, 0 % loss, low and steady latency.
Path and where it breaks	tracert -d 198.51.100.10 pathping -n -q 20 198.51.100.10	mtr -rwc 20 198.51.100.10 traceroute -n 198.51.100.10	/tool traceroute 198.51.100.10	Clean path, last hop answers, no loss before it.
DNS over UDP 53	nslookup example.com 198.51.100.10 Resolve-DnsName example.com -Server 198.51.100.10 -Type A -DnsOnly	dig @198.51.100.10 example.com +time=2 +tries=1 dig @198.51.100.11 example.com +time=2 +tries=1	/tool netwatch print where name~"kahf-hc"	NOERROR with an address, well under 100 ms.
DNS over TCP 53	nslookup -vc example.com 198.51.100.10 Resolve-DnsName example.com -Server 198.51.100.10 -TcpOnly Test-NetConnection 198.51.100.10 -Port 53	dig @198.51.100.10 example.com +tcp +short nc -vz -w3 198.51.100.10 53	-	Same answer as UDP.
A blocked name	Resolve-DnsName pornhub.com -Server 198.51.100.10 -Type A -DnsOnly	dig @198.51.100.10 pornhub.com +short dig @198.51.100.11 pornhub.com +short	-	The Kahf block-page address, with status NOERROR (a block is a successful answer).
The redirect (ask a non-DNS address)	nslookup pornhub.com 192.0.2.53 Resolve-DnsName pornhub.com -Server 192.0.2.53 -DnsOnly	dig @192.0.2.53 pornhub.com +short +time=2 +tries=1	/ip firewall nat print stats where comment~"KAHF-HC"	The block-page address comes back from 192.0.2.53. A timeout means the redirect is not working.
IPv6 DNS	Resolve-DnsName example.com -Server 2001:db8:0:1::10 -Type AAAA -DnsOnly ping -6 -n 5 2001:db8:0:1::10	dig -6 @2001:db8:0:1::10 example.com AAAA +time=2 +tries=1 ping -6 -c 5 2001:db8:0:1::10	/ipv6 firewall nat print stats	AAAA answer over IPv6, or a clean fallback to IPv4.
Large answers (EDNS, TCP retry)	Resolve-DnsName isc.org -Type DNSKEY -Server 198.51.100.10 -DnsOnly	dig @198.51.100.10 isc.org DNSKEY +bufsize=4096 +time=3	-	DNSKEY answer arrives, possibly after a TCP retry. No timeout.
Path MTU	ping -f -l 1472 -n 3 198.51.100.10	ping -M do -s 1472 -c 3 198.51.100.10	/ping 198.51.100.10 size=1500 do-not-fragment count=3	1472-byte payload gets replies (1500 MTU end to end). "Needs fragmentation" means a smaller MTU on the path.
DNS over TLS must be refused (853/tcp)	Test-NetConnection 1.1.1.1 -Port 853	nc -vz -w3 1.1.1.1 853 openssl s_client -connect 1.1.1.1:853 -brief </dev/null	/ip firewall filter print stats where comment~"Reject DoT"	Refused or blocked from a customer-side host. That means the router is closing the side door.
DNS over HTTPS must be refused (443 to resolver IPs)	curl.exe -m 8 --doh-url https://1.1.1.1/dns-query https://example.com Test-NetConnection 1.1.1.1 -Port 443	curl -m 8 --doh-url https://1.1.1.1/dns-query https://example.com dig @1.1.1.1 +https example.com +time=3 +tries=1	/ip firewall filter print stats where comment~"Reject DoH"	Connection fails. Normal web (next row) keeps working.
DNS over QUIC / DoH3 must be refused (UDP)	-	kdig @1.1.1.1 +quic +timeout=3 +retry=0 example.com	/ip firewall filter print stats where comment~"Reject DoQ"	Times out or is refused. Windows has no built-in client, so test from Linux or macOS.
Normal web and QUIC still work	curl.exe -I -m 8 https://www.google.com	curl -I -m 8 https://www.google.com curl -I -m 8 --http3 https://www.google.com	-	HTTP 200 or 301/302. Filtering must never slow ordinary browsing.
Each fallback rung answers	Resolve-DnsName example.com -Server 142.132.207.231 -DnsOnly Resolve-DnsName example.com -Server 1.1.1.3 -DnsOnly	for s in 203.190.10.120 157.90.95.113 142.132.207.231 46.4.74.205 1.1.1.3 1.1.1.1; do echo $s; dig @$s example.com +short +time=2 +tries=1; done	/tool netwatch print where name~"kahf-hc"	All rungs answer when asked directly. The router probes them all the time.
What the customer device uses (Windows)	ipconfig /all \| findstr /i "DNS Servers" Get-DnsClientServerAddress -AddressFamily IPv4 Get-DnsClientCache \| Select -First 10 ipconfig /flushdns	resolvectl status sudo resolvectl flush-caches	-	Whatever it shows, the router redirects port 53, so answers are still filtered.
Packet capture	pktmon filter add -p 53 pktmon start --capture pktmon stop pktmon etl2txt PktMon.etl	sudo tcpdump -ni any 'port 53 and host 198.51.100.10'	/tool sniffer quick port=53 ip-address=198.51.100.10	One query out, one answer back per lookup. Retries mean loss.

Verify before first use. Command spellings above follow the Kahf script and RouterOS 7.x. If your RouterOS version prints an error, tell us the version; do not guess a different command during an incident.

Section 5

Operating the router script

How the failover decides

Every probe interval (1 s) the router asks each forwarder and each fallback a real DNS question (google.com, A record, 1 s timeout).
3 missed probes in a row: the target is marked DOWN.
5 good probes in a row: marked UP again.
Forwarders healthy: round-robin across them. One down: all traffic to the survivor. All down: first healthy fallback, masqueraded out of the uplink.
Kahf's own monitoring addresses are exempt from the rescue rules, so our probes see the true state.

Modes (retrofit module)

Mode	Does
dry	Probes and logs only. NAT untouched.
test	Builds the rules disabled, so you can inspect them.
live	Builds the rules enabled and disables the old static rules.

/system script set kahf-hc-mode source=dry
/system script set kahf-hc-mode source=test
/system script set kahf-hc-mode source=live

The fallback ladder

The fallback list is ordered. The router probes every rung and uses the first healthy one, then moves back up by itself when a higher rung recovers.

Rung	Who	Addresses	Filtering
1	Your Kahf forwarders	198.51.100.10 198.51.100.11	Kahf filtering
2	Kahf resolver, Bangladesh (BDIX)	203.190.10.120	Kahf filtering
3	Kahf global resolvers	157.90.95.113 142.132.207.231 46.4.74.205	Kahf filtering
4	Filtered public resolvers	1.1.1.3 1.0.0.3	public family filter
5	Open public resolvers	1.1.1.1 8.8.8.8 9.9.9.9	no filtering

# in the script header, order = priority
:global kahfHcFallbacks "203.190.10.120,157.90.95.113,142.132.207.231,46.4.74.205,1.1.1.3,1.0.0.3,1.1.1.1,8.8.8.8,9.9.9.9"

Tell Kahf your router address. Fallback queries are masqueraded, so Kahf sees every customer as one source. Rungs 2 and 3 need that address allowed and free of per-source rate limits. Remove the last rung if you prefer no DNS to unfiltered DNS.

Rollout checklist (new router or retrofit)

Edit the variables at the top: forwarder IP(s), fallbacks, uplink interface (kahfHcOutIf). Confirm the router's own address is on the forwarder allow-list (ask Kahf). Otherwise probes fail forever. Import in dry. Watch /log print where message~"kahf-hc" for at least 10 minutes. Expect "installed" and no DOWN lines. Switch to test; inspect /ip firewall nat print where comment~"KAHF-HC". For the first live run set kahfHcFailsafeMin=10 so the router rolls itself back after 10 minutes unless you stay. Switch to live in a low-traffic window. Run Triage Steps 1 to 3. Remove the failsafe scheduler: /system scheduler remove kahf-hc-failsafe.

Rollback and bail-out

Need	Command	Effect
Stop failover changes, keep watching	/system script set kahf-hc-mode source=dry	Probes and logs only. Existing rules stay as they are.
Remove the health-check layer	/system script run kahf-hc-uninstall	Removes `KAHF-HC` items and re-enables the original static `Do53 UDP` / `Do53 TCP` redirect rules, which point at fixed addresses. Check they still point at a healthy forwarder.
Remove everything the scripts added	forwarder-router-cleanup.rsc	Full removal of Kahf rules and lists.
Check an existing router, change nothing	forwarder-router-audit.rsc	Read-only report.

The forwarders must be exempt from the redirect. If a forwarder sits behind the router, its own upstream lookups must never be redirected back to a forwarder. Keep an accept rule for the forwarder addresses at the top of the dstnat chain (and in the Kahf_HC_Fwds list). Symptom when it is missing: forwarders answer from cache but new names time out.

Do not re-import the full script during an incident unless that is the plan. Import is destructive by design: it removes and rebuilds its own rules and address lists. After any import, run Triage Steps 1 to 3.

Known script habits (RouterOS). No trailing # comment after a value. Settings changed in the file take effect only after you set the mode to dry and re-import.

Section 6

Escalating to Kahf

Severity	Example	How	Response target
1	No DNS for most customers for more than 5 minutes; forwarder dead from everywhere and failover not working	Phone, then message	to be agreed
2	One pool or prefix without DNS; repeated flapping; unfiltered for more than an hour with no known reason	Message + ticket	to be agreed
3	Domain block/unblock requests; single customer	Ticket	to be agreed

What to send

Subject: ISP DNS - <one-line symptom>
Severity: 1 / 2 / 3
1. When it started (UTC and BDT):
2. Who is affected (prefix / public IP / everyone):
3. Domain tested:
4. dig @198.51.100.10 <domain> result (paste):
dig @198.51.100.11 <domain> result (paste):
5. Same dig from a second host in a different part of the network (paste):
6. dig @192.0.2.53 <blocked domain> result (redirect test):
7. Router: /tool netwatch print where name~"kahf-hc"  (paste)
8. Router: /log print where message~"kahf-hc"  (last 20 lines)
9. Anything changed in the last 24 h (router, ranges, uplink):

Tell us ahead of time when you add a customer range or CGNAT pool, change a router's address, replace a router, or plan an uplink maintenance. Most real incidents so far came from a range that was in use before we knew about it.

Section 7

After an incident

Triage Steps 1 to 3 all pass. /tool netwatch print shows the forwarder UP and the log shows no new DOWN lines for 15 minutes. Any failsafe scheduler removed; mode is live (not left in dry or test). Timeline written down: start, detection, failover, fix, end (UTC and BDT). Kahf informed of the cause; any allow-list change confirmed by us.

Change record

Date	Router / host	Change	By	Rollback tested

Limits of this runbook. Failover timings are computed from the script settings, not measured on a ISP router. Router model, RouterOS version, interface names and addresses in this material are examples. Check them against your own router before you copy anything. Please correct this document where your network differs.