aa.net.uk Broadband - Broadband you can work with

Skip to Navigation / Skip to Content

News Change to aa.net.uk

13th Feb 2012 We made some changes which caused more problems than expected. The page details what we changed, why and what went wrong.

Obviously we always try to learn from mistakes.

What did we do?

Our main domain has been aaisp.net.uk for a long time. We have a lot of DNS records under aaisp.net.uk and a lot of sub domains for all sorts of things. Most of this is handled as normal zone files in our authoritative DNS servers, however a few sub domains are special (e.g. mail.aaisp.net.uk and a few others) which are handled by an algorithmic/database name server.

For a year now we have also had aa.net.uk which we managed to get following a change in Nominet policy. We like this new domain and have been using it for over a year. What we did was to create a DNAME record mapping aa.net.uk to aaisp.net.uk. A DNAME record is a bit special as it maps everything, all sub domains, e.g. something.aa.net.uk maps to something.aaisp.net.uk. DNAMEs are a bit new and not handled by all resolvers, but that is OK as the name servers automatically send a CNAME for any lookup as well, and that is handled by pretty much everything.

We have been testing aa.net.uk a lot - using it internally and publishing URLs and servers and all sorts using aa.net.uk. It has worked well. And a year and 5 days of testing seemed more than enough.

So what did we do exactly?

Well, it was simple, swap aaisp.net.uk and aa.net.uk. How hard could it be?

To be honest, we expected some minor issues. By far the biggest would be the change of reverse DNS. All machines that were previously something.aaisp.net.uk would now be something.aa.net.uk. So, we were careful. Apart from changing TTLs in advance, we set up one machine that would retain an ...aaisp.net.uk reverse. This would allow us in to various servers to change access lists over. Basically, we expected issues affecting our office and staff.

We tested that all, and renamed the zones, swapping aaisp.net.uk and aa.net.uk over.

BANG!

What went wrong

This is where it gets more complex. Lots of things did not go to plan...

Authoritative servers not working!

First off, when we reloaded the authoratative servers, they did not run. WTF!

There were thousands of log lines, and buried in them one of the customer zone files had a line that was wrong. Somehow it had got through the checkers on the web site, and instead of just killing that zone, it caused the whole name server not to run. That was not expected and took a while to track down (thanks Mike!).

Bad timing

Somehow we managed to get a reverse for the back door machine cached which meant we could not get in to any servers for a while. Arrrg!

Unexpected magic

Then we realised that aaisp.net.uk was delegated to our special algorithmic/database name servers. That was fine, as they would serve normal records, but they did not know how to handle DNAMEs. So they would not map aaisp.net.uk domains to aa.net.uk as expected. We changed the delegation, but the timeout on the top level records is 48 hours. Arrrg! The code was changed to handle DNAMEs, but this took several hours.

It then took a while to work out that aa.net.uk was not fully working for the special sub domains like mail.aa.net.uk. This was because bind was being brain dead. Now the penny dropped! This was why aaisp.net.uk had been delegated to the special servers in the first place as bind was refusing to do simple delegation properly!!! The fix was to change delegation of aa.net.uk to the special servers. Again, a 48 hour TTL on that. Arrrg!

Mopping up

Finally, by the end of the day we were mopping up things we expected. Changing access lists on servers. Updating some zone files. The things we expected to be doing and generally not customer effecting.

Just after 5pm we had the special name server handling the DNAMEs OK as well, which fixed the last of the problems, as far as we know.

Why now?

It is a change we have wanted to do for nearly a year. It is something which we expected to have minor inconvenience and mainly effect staff. It is a change that needed a lot of staff working on the repercussions, so doing at lunch time made most sense.

One reason though is the technicolor routers doing TR069 won't follow a CNAME, D'Oh! So if we start hard coding them to talk to an aaisp.net.uk server we will never be able to change it over. As we are working on the TR-069 server then we finally had an excuse to actually make the change.

Do I need to make changes?

No, aaisp.net.uk and aa.net.uk should work the same as each other. If you are making changes or entering new things, we prefer aa.net.uk.

Why not back out the change?

Basically, one of the domains needed to be on the special servers and one not, so it seems. The TTL on the top level delegation from Nominet is 48 hours, and we cannot control that. Having started this, backing out the change would leave people with the changed entries in cache and a 48 hour timeout, so not help. We had to find a way to move forward and fix the special DNS servers. That done, we are OK.

Apology?

This was inconvenient for a lot of people. Some more than others, and some for longer than others. I would like to apologise personally for any inconvenience caused. I felt it was something we had to do, and the issues we encountered we would never have expected. Sorry. Adrian, Director.


Knowledge Base

  • All the technical information your geek heart could desire.
Find out more