Red Hat CDK DNS Issues

While not my usual wheelhouse, I've been struggling with a DNS issue with the Red Hat CDK on my home network. Not that I'm totally ignorant in the ways of configuring and running my own BIND daemons for fun and profit, this issue caught me off guard.

I run an internal BIND server on a Raspberry Pi running an Fedora 25 for ARMv7. While it resolved internal network IP addresses perfectly, I didn't have it configured as a caching DNS server for external addresses. Anything "outside the building" (my house) would go through my ISP's DNS servers because that's the list of DNS servers given to any machine on the network, whether by DHCP or manual configuration.

My CDK hosts' /etc/resolv.conf files looked like this:

# Generated by NetworkManager
search <My local domain name>
nameserver <My local BIND server's IP>
nameserver <My ISP router's IP>


Therein lay the failure.

Here's the behavior I saw. When starting the vagrant image, I'd see the following:

Bringing machine 'default' up with 'libvirt' provider...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
==> default: Registering box with vagrant-registration...
    default: Would you like to register the system now (default: yes)? [y|n]y
    default: username: XXXXXXXXXXX
    default: password:
==> default: Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.
==> default: Registering to: subscription.rhn.redhat.com:443/subscription


At that point, the CDK was running and was available for SSH. However, all DNS resolution attempts failed with "temporary failure in name resolution," including that required for Red Hat Subscription Manager registration and to pull new Docker images for the CDK.

Vagrant and libvirt use dnsmasq to provide a small DHCP service and DNS proxy to hosted containers. However, dnsmasq by default apparently only considers the first nameserver configured for the host. Since my host's first nameserver only resolved internal hostnames, redhat.com names were not resolvable.

The solution was nice and simple, thankfully. Just add a set of forwarders--in my case, my local ISP router and a couple of well-known DNS server farms.

forwarders {
  192.168.1.1;
  8.8.8.8;
  8.8.4.4;
};


Be careful with the BIND configuration options though. With recent DNS attacks causing all kinds of widespread havoc, there's plenty of opportunity for perpetuating someone else's bad behavior.