Ansuz

Mastodon WTF timeline

In the last few days I've been fortunate to witness an interesting chapter in the Internet's history, and I'm trying to compile a timeline of what has happened while the memories are still reasonably fresh. This is incomplete and a work in progress; I'll be updating it, and not necessarily in chronological order, as I dig up other things worth including. Some of my TODO markers may remain. But here goes.

Fixing "unexpected RCODE (SERVFAIL)" and "unexpected RCODE (REFUSED)"

This is another one where I searched the net, the answers I found were very unhelpful, and so I'm posting what worked for me for the benefit of anyone making similar searches.

The problem: new ADSL connection from MTS Allstream, which is the deregulated ghost of the Manitoba telecom monopoly. Works pretty well, except they do that damn misguided "helpful" redirection of failed DNS requests to a search engine, thereby screwing up all non-Web activities that depend on the DNS actually working according to the protocol. They offer opt-out but that doesn't work. So I set up my own caching DNS server and everything seemed fine... except just a few Web sites wouldn't work. Always the same sites; little or no rhyme or reason to which ones they were. Penny Arcade, Weather Underground, the Canada Revenue Agency, and the CBC, were the most annoying examples. The browser would hang, trying to connect, forever.

Digging through the system logs revealed lines like these:

Jan 25 11:34:38 tetsu named[1613]: unexpected RCODE (SERVFAIL) resolving 'art.penny-arcade.com/A/IN': 72.52.2.1#53
Jan 25 11:37:55 tetsu named[1613]: unexpected RCODE (REFUSED) resolving 'cbc.radio-canada.ca/A/IN': 207.164.234.37#53

Searching on the Web produced many people complaining about error messages like these, and the following answers on how to resolve it:

• "You must be the authoritative server for these domains, and you haven't given BIND the correct path to the zone files." No, I am not the authoritative server for these domains.
• "You must be the authoritative server for these domains, and someone on the Net is trying to break into your nameserver." No, I really am not the authoritative server for these domains, the failing requests are coming from an authorized user on localhost (namely me), and incoming unauthorized DNS requests would be stopped at the firewall anyway.
• "The remote authoritative servers for these domains are misconfigured and you must contact the admins and tell them to fix the problem." Yes, I REALLY HAVE THE TIME AND ABILITY TO CONVINCE EVERY ADMINISTRATOR OF A MISCONFIGURED NAMESERVER ON THE ENTIRE INTERNET TO FIX THEIR CONFIGURATIONS BECAUSE THEY WILL ALL LISTEN TO ME! Also, of course, I can contact these administrators by pure mental telepathy, since my computer cannot connect to theirs to send them email.

Clearly, none of these answers was helpful. Here's the actual answer: The MTU on my Ethernet connection was set to the default of 1500. Packets that size cannot pass through the ADSL connection; and to make matters worse, MTS apparently drops ICMP traffic (this could be my fault because it may be happening at the firewall box, which is theirs but was reconfigured by me), so that Path MTU Discovery (which would automatically adjust the setting) doesn't work. It wasn't really anything specific to DNS, but would cause subtle effects in a lot of places; DNS was just the most visible thing failing. Solved by changing my MTU to 1400; there may be some slightly larger number that will work (I'll experiment), but 1500 evidently is too big. It appeared only, but consistently, on a few domains, because those were the ones where the DNS query or its answer (which would generally be consistent per domain) happened to both exceed the ADSL connection's real MTU and not be fragmented anywhere else in the network.