How the Kahf DNS
forwarder works
on your network
What happens to a DNS question from a customer's phone to our forwarder and back. How your router script behaves when things break. How to find the fault in five minutes.
Two halves. You run the left, we run the right.
The story for the session: the router is the darwan, the gatekeeper of an apartment building in Dhaka, who stops every visitor and sends them to the right office whichever door they asked for. The forwarders are the service counters at a bank or a Union Parishad office: it takes the question, checks it, and gives the answer. The Kahf service is the head office that sets the rules. Nobody at the gate or the counter has to visit it.
You own
Customer access (ONT, OLT, POPs), your core, the router or routers, their firewall and NAT, the script we gave you, the uplink to our forwarders.
We own, and you can see
The two forwarder VMs, 198.51.100.10 and 198.51.100.11 (example addresses; yours are different). It answers DNS on port 53, UDP and TCP. You can dig it from anywhere in your network.
We own and run for you
How the Kahf service decides what to block, and keeping it up to date. Kahf manages it, so you do not operate it and you do not need it to troubleshoot.
Life of one DNS question
Watch the blue dots go out and the green dots come back. Step with → to see each hop explained.
- Customer asks. "What is the address of example.com?" It goes to whatever DNS server the customer's device knows. That may be yours, 8.8.8.8, anything.
- Router redirects. The script rewrites the destination of every port-53 packet to our forwarder. The customer never notices.
- Forwarder decides. It sends the question to the Kahf service and gets back a real address, or the address of the block page.
- Answer returns the same way. Customer opens the site or sees the block page.
The redirect trick (destination NAT)
A customer who types 8.8.8.8 into their phone still gets filtered DNS. The router swaps the address on the envelope.
Packet leaves the customer
| Field | Value |
|---|---|
| source | customer IP |
| destination | 8.8.8.8 : 53 |
Packet leaves the router
| Field | Value |
|---|---|
| source | customer IP (unchanged) |
| destination | 198.51.100.10 or .11 (alternating) : 53 |
192.0.2.53. If an answer comes back, the redirect is working. If it times out, it is not.Every query ends in one of three ways
Real output from the ISP forwarder, taken today with dig.
Real address
$ dig @198.51.100.10 example.com status: NOERROR example.com. 173 A 172.66.147.243
The site opens normally.
Block-page address
$ dig @198.51.100.10 pornhub.com status: NOERROR pornhub.com. 300 A 203.190.10.123 ; MX/TXT etc: NOERROR, empty
Answer is valid, but the address is Kahf's block page. The customer sees the block page, not an error.
Timeout
$ dig @198.51.100.10 google.com
;; communications error: timed out; no servers could be reached
This is the one that is always a fault: forwarder down, path broken, or the source IP is not on our allow-list.
A site is blocked, a site is unblocked
Why a change on our side does not reach every customer in the same second.
Customer opens a blocked site
For HTTPS sites some browsers show a certificate warning before the page. That is normal for any DNS-level block and not a ISP fault.
The cache timeline after an unblock
ipconfig /flushdns). Do not re-report before that.Closing the side doors
Redirecting port 53 is not enough. Apps can ask DNS over an encrypted tunnel, or hide in a VPN. The script rejects those, so the app falls back to normal port-53 DNS, which we do filter.
| Attempt | Port | Router does |
|---|---|---|
| Plain DNS | 53 | redirect to forwarder |
| DNS-over-TLS | 853/tcp | reject |
| DNS-over-QUIC | 853/udp | reject |
| DoH (known providers) | 443 | reject |
| VPNs: OpenVPN, WireGuard, IKEv2, L2TP, PPTP | 1194 51820 500 4500 1701 1723 | reject |
| Tor | 9001 9030 9050 9051 9150 | reject |
| Normal web and QUIC | 443 | untouched |
What is inside the health-check module
ISP's router already redirects port 53, so you run the retrofit module: forwarder-router-dns-healthcheck.rsc. It changes nothing else. Every object it creates is tagged KAHF-HC, so it can be found and removed.
1 · Probes
One netwatch DNS probe per forwarder and per fallback: a real question (google.com, type A) every 1 s, 1 s timeout.
2 · Controller
Scheduler kahf-hc runs a script every 1 s. It counts misses and hits and decides who is alive.
3 · NAT rules
It rebuilds the port-53 redirect rules for whoever is alive: alternate across both, survivor only, or fallback. It replaces the two static Do53 UDP/TCP rules.
4 · Roll back
kahf-hc-uninstall removes all KAHF-HC items and re-enables the original static rules. Modes dry, test, live let you rehearse first.
accept rule for the forwarder addresses at the top of the dstnat chain.Other files in the kit
| forwarder-router-dns-v2.rsc | full enforcement for a fresh router (destructive import) |
| forwarder-router-audit.rsc | read-only check of an existing router |
| forwarder-router-cleanup.rsc | removes what the scripts add |
Two forwarders: share the load, survive a failure
Your network runs two Kahf forwarders behind one router. The health check decides who gets traffic. Pick a state.
Both healthy
The router alternates new lookups between the two (nth round-robin). Each forwarder carries about half.
One down
All lookups go to the survivor. Filtering stays on. Each forwarder must be able to carry the full load; today's peak is a small fraction of what one can do.
Both down
First healthy fallback. DNS works, filtering is off, until a forwarder is healthy again.
If both forwarders fail, the router climbs down a ladder
The fallback list is ordered. The router probes every rung all the time and sends customers to the first healthy rung. When a higher rung recovers, traffic moves back up on its own. Take rungs down one at a time.
Try it: kill the forwarder
The router asks every forwarder a real DNS question every 1 s. 3 misses in a row and it is marked down; 5 hits in a row and it is trusted again. This is a live simulation of that logic.
Counters (per forwarder)
| Forwarder | Probe | Missed | Hit | Verdict | Share |
|---|
One tick = 1 s: the simulation runs in real time, like your router.
How long is the outage window?
Customers feel the gap between a forwarder dying and the router noticing. These numbers come from the script settings, not from a stopwatch on your routers. Treat them as estimates.
What we watch about your forwarder
You do not need the dashboard to troubleshoot. It is how we notice a problem before you call. This is what the ISP view answers.
Is it alive?
Forwarders up and an external probe from outside ISP that asks a real question every few seconds. Probe fails = we know within a minute, before your customers do.
Is it healthy?
SERVFAIL %, answer-time percentiles, and cache hit rate. A rising SERVFAIL or slow tail means trouble behind the forwarder.
Is it doing its job?
Blocked %, queries per second, and whether filtering is switched on. Blocked at 0 % with normal traffic means filtering is off.
Two views of the same fault
| You see | We see |
|---|---|
| Netwatch says forwarder DOWN | External probe failing |
| Some prefixes get timeouts | Rising "dropped by allow-list" counter |
| Slow browsing | p95 answer time up |
Prove the ideal state, protocol by protocol
Pick your tool. Each row says what a healthy network shows. Rows marked "must be refused" prove the side doors are closed, so a failure to connect is the good result.
| Check | Command | Ideal result |
|---|
nslookup talks to the server you name and ignores the hosts file. Resolve-DnsName -DnsOnly skips LLMNR and mDNS. Run PowerShell as administrator for pktmon. curl.exe ships with Windows 10 1803 and later; type the .exe so PowerShell does not use its own alias.Router-side cheat sheet
Run the first block from a Linux host inside ISP. Run the second on the MikroTik.
# 1. does the forwarder answer at all? dig @198.51.100.10 google.com +time=2 +tries=1 dig @198.51.100.11 google.com +time=2 +tries=1 # 2. is filtering on? (expect 203.190.10.123) dig @198.51.100.10 pornhub.com +short dig @198.51.100.11 pornhub.com +short # 3. TCP works too? dig @198.51.100.10 google.com +tcp +short dig @198.51.100.11 google.com +tcp +short # 4. is the router redirecting? any answer = yes dig @192.0.2.53 pornhub.com +short +time=2 +tries=1 # 5. path to the forwarder mtr -rwc 20 198.51.100.10 mtr -rwc 20 198.51.100.11
# probe state, per forwarder /tool netwatch print where name~"kahf-hc" # what the controller decided and when /log print where message~"kahf-hc" # which redirect rules are live now /ip firewall nat print where comment~"KAHF-HC" # are packets hitting them? /ip firewall nat print stats where comment~"KAHF-HC" # connections currently pinned to the forwarder /ip firewall connection print where reply-src-address~"198.51.100.10:53" /ip firewall connection print where reply-src-address~"198.51.100.11:53"
The ideal rollout: dry, test, live
Three modes let you rehearse the failover before it can touch a customer. A safety timer undoes the first live run by itself.
Before dry
Set forwarders, fallback list and uplink interface. Make sure the forwarders are exempt from the redirect. Give Kahf your router address.
During live
Run the tests. Stop one forwarder on purpose, watch the survivor take over, start it again. Then, in a quiet window, stop both.
After live
Remove the failsafe timer. Keep kahf-hc-uninstall in your notes. Write the date and the result in your change log.
Faults, one map
Same map every time. Only the state changes. Pick a fault.
Check
Fix
The failure the health check cannot see
The router probes from its own address. So the probe can be perfectly healthy while some customers get nothing.
Prefix A
Works. Gets answers.
Prefix B
Times out. The forwarder never replies; there is no error for the customer to see.
Router probe
Reports UP, so no failover happens. The router believes everything is fine.
Five-minute triage, answer by answer
Click through it as if a ticket just arrived.
Result
Answer the questions on the left.
Quick remedies, in order of preference
The first option fixes the cause. The last gives up filtering to protect DNS. Every one can be undone.
| # | Situation | Action | Effect | Undo |
|---|---|---|---|---|
| 1 | One forwarder dead | Nothing. Failover is automatic. Tell Kahf. | Survivor serves everything, filtering stays on, after ~4 s | auto |
| 2 | Prefix gets timeouts | Send Kahf the source IP / prefix | Prefix works after our restart | we revert |
| 3 | Redirect rules missing | Re-import the script (it rebuilds its rules) or re-enable disabled rules | Filtering back | cleanup script |
| 4 | Health check misbehaving (flapping) | /system script set kahf-hc-mode source=dry | Probes + logs only, NAT untouched | set source=live |
| 5 | Emergency: DNS broken and no time | /system script run kahf-hc-uninstall | Health-check layer gone, original rules re-enabled | re-import healthcheck |
kahfHcFailsafeMin=10 for the first live run. After 10 minutes the router rolls itself back unless you cancel the timer. Stay in dry mode first and read the log. Then test (rules built, disabled), then live.When to call us, and what to bring
Eight facts make the first reply take minutes instead of hours.
Call right away if
- Forwarder times out from several places for over 2 minutes.
- A whole customer pool has no DNS.
- A site must be blocked or unblocked urgently.
- Netwatch flaps up/down more than a few times an hour.
Can wait for a ticket
- One customer, one device.
- Blocked/unblocked request, no deadline.
Subject: ISP DNS - <one-line symptom> 1. When it started (UTC + BDT): 2. Who is affected (prefix / public IP / all): 3. Domain tested: 4. dig @198.51.100.10 result (paste): dig @198.51.100.11 result (paste): 5. Same dig from a second host (paste): 6. dig @192.0.2.53 result (redirect test): 7. Router: /tool netwatch print (paste) 8. Router: /log print where message~"kahf-hc" (last 20 lines)
What is solid, what is not
Solid
- Redirect, reject rules and failover are in the script and read in this session.
- The numbers on the impact slide come from the live dashboard, today.
- The dig outputs are real answers from both forwarders.
Estimates and gaps
- Failover timings are computed from script settings, not measured on a ISP router.
- Router model, RouterOS version, interface names and addresses in this material are examples. Check them against your own router before you copy anything.
- The impact window is hours, not a month.
- Timings come from the retrofit module's default settings. A full-script install probes every 3 s, so detection takes about 9 to 10 s instead of 3 to 4 s.
runbook.html (printable). Questions?