Windows server 2019: DNS problem

What you’re seeing (the same FQDN sometimes resolving to a “private”/non-public IP, and the issue disappearing when you tunnel via VPN) strongly suggests that the answer your AD DNS servers receive depends on which WAN link (ISP A vs ISP B) the DNS query exits from. With SD‑WAN/load-balancing, that can flip from one query to the next, and Windows DNS will then cache whichever answer it got first (until TTL expires).

1) Why would your local DNS servers ever get the web server’s “private” IP?​

Your Windows DNS servers are not generating that IP. They learn it from upstream in one of these common ways:

A. The authoritative DNS (or CDN/DNS provider) returns different answers “on-net”​

Many DNS providers/CDNs use location/ISP-based steering (sometimes EDNS Client Subnet, sometimes just “where the recursive resolver is”). If your recursive path appears to come from ISP A, they may return an “on-net” address that is not globally routable (could be RFC1918 like 10/8, or CGNAT 100.64/10, or some private peering range). When it exits via ISP B, you get the normal public answer.

B. ISP A is intercepting DNS (DNS hijack/transparent redirect)​

Even if you configured forwarders as 9.9.9.9 / 9.9.9.11, an ISP can still transparently intercept plain UDP/TCP 53 and answer from their recursive resolvers instead. Your VPN would bypass that interception, which matches your symptom.

C. Cache + dual-WAN randomness​

Once one of your DNS servers gets the “wrong” answer, it will cache it. So you can observe “sometimes” behavior even if you later force traffic out a different WAN, until cache/TTL expires.

2) Who is providing the private IP?​

Either:

• the authoritative DNS for that hostname (or the CDN/DNS provider in front of it), or
• a recursive resolver you didn’t intend to use (due to ISP A interception), or
• less commonly, an internal device acting as DNS proxy (FortiGate DNS proxy / DNS filter) rewriting answers.

3) The quickest way to prove where it’s coming from​

Run these from each Windows DNS server when the issue happens:

Test 1 — compare answers from multiple public resolvers​

If only the queries exiting via ISP A return the private IP, that points to steering or ISP interception.

Test 2 — query the authoritative nameserver directly​

If the authoritative server returns the private/non-public IP to you, then it’s a DNS provider/CDN policy problem (not Quad9).

Test 3 — check what your DNS server cached​

Get-DnsServerCache -Name "yourhost.yourdomain.com"

4) Fixes (pick the cleanest that fits your environment)​

Fix A (best): Ensure public DNS never returns private/non-public addresses​

If your public DNS provider/CDN is returning a private/CGNAT address based on ISP, ask them to stop doing that for that record. Public DNS for an Internet-facing name should return publicly routable answers unless you’re intentionally doing split-horizon.

Fix B: Make your DNS forwarding deterministic (stop “sometimes ISP A / sometimes ISP B”)​

On the FortiGate/SD‑WAN, create a rule so that DNS from your AD DNS servers to the forwarders always exits the same WAN (e.g., ISP B).
Also strongly consider enforcing that only your DNS servers can do outbound DNS (block all other clients from UDP/TCP 53 to the Internet). This is a well-established best practice to prevent bypassing controls and weird resolver behavior.

Fix C: Hard-code the internal answer in AD DNS (yes, you can)​

Yes—you can override the FQDN internally so clients always get the public IP (or whatever you choose).
Best practice for only ONE hostname: create a zone named exactly the hostname (not the whole domain), e.g.:

• Zone: portal.example.com
• Record: (same name / zone apex) A → x.x.x.x (public IP)

That avoids creating a full example.com zone internally (which would force you to manage all example.com records).
PowerShell example:
Then:

• On DNS servers: Clear-DnsServerCache
• On clients: ipconfig /flushdns

5) A key detail I need from you​

When you say “private IP only known to ISP A”, is it:

• RFC1918 (10.x / 172.16-31.x / 192.168.x), or
• 100.64.x.x (CGNAT), or
• something else?

Also, where exactly do you see “DNS rebind attack” reported—FortiGate logs, browser, or the web app?
If you paste:
1) one “good” and one “bad” nslookup/Resolve-DnsName result (redact domain if needed), and
2) the IP range of the “private” answer,
I can tell you which of the three root causes (authoritative steering vs ISP interception vs caching/SD‑WAN) is most likely and which fix is the most direct.