Login
Every few months, an important ceremony takes place. It’s not splashed all over the news, and it’s not attended by global dignitaries. It goes unnoticed by many, but its effects are felt across the globe. This ceremony helps make the internet more secure for billions of people.
This unique ceremony began in 2010 when Verisign, the Internet Corporation for Assigned Names and Numbers (ICANN), and the U.S. Department of Commerce’s National Telecommunications and Information Administration collaborated – with input from the global internet community – to deploy a technology called Domain Name System Security Extensions (DNSSEC) to the Domain Name System (DNS) root zone in a special ceremony. This wasn’t a one-off occurrence in the history of the DNS, though. Instead, these organizations developed a set of processes, procedures, and schedules that would be repeated for years to come. Today, these recurring ceremonies help ensure that the root zone is properly signed, and as a result, the DNS remains secure, stable, and resilient.
In this blog, we take the opportunity to explain these ceremonies in greater detail and describe the critical role that Verisign is honored to perform.
DNSSEC is a series of technical specifications that allow operators to build greater security into the DNS. Because the DNS was not initially designed as a secure system, DNSSEC represented an essential leap forward in securing DNS communications. Deploying DNSSEC allows operators to better protect their users, and it helps to prevent common threats such as “man-in-the-middle” attacks. DNSSEC works by using public key cryptography, which allows zone operators to cryptographically sign their zones. This allows anyone communicating with and validating a signed zone to know that their exchanges are genuine.
The root zone, like most signed zones, uses separate keys for zone signing and for key signing. The Key Signing Key (KSK) is separate from the Zone Signing Key (ZSK). However, unlike most zones, the root zone’s KSK and ZSK are operated by different organizations; ICANN serves as the KSK operator and Verisign as the ZSK operator. These separate roles for DNSSEC align naturally with ICANN as the Root Zone Manager and Verisign as the Root Zone Maintainer.
In practice, the KSK/ZSK split means that the KSK only signs the DNSSEC keys, and the ZSK signs all the other records in the zone. Signing with the KSK happens infrequently – only when the keys change. However, signing with the ZSK happens much more frequently – whenever any of the zone’s other data changes.
Something to keep in mind before we go further: remember that DNSSEC utilizes public key cryptography, in which keys have both a private and public component. The private component is used to generate signatures and must be guarded closely. The public component is used to verify signatures and can be shared openly. Good cryptographic hygiene says that these keys should be changed (or “rolled”) periodically.
In DNSSEC, changing a KSK is generally difficult, whereas changing a ZSK is relatively easy. This is especially true for the root zone where a KSK rollover requires all validating recursive name servers to update their copy of the trust anchor. Whereas the first and only KSK rollover to date happened after a period of eight years, ZSK rollovers take place every three months. Not coincidentally, this is also how often root zone key signing ceremonies take place.
The notion of holding a “ceremony” for such an esoteric technical function may seem strange, but this ceremony is very different from what most people are used to. Our common understanding of the word “ceremony” brings to mind an event with speeches and formal attire. But in this case, the meaning refers simply to the formality and ritual aspects of the event.
There are two main reasons for holding key signing ceremonies. One is to bring participants together so that everyone may transparently witness the process. Ceremony participants include ICANN staff, Verisign staff, Trusted Community Representatives (TCRs), and external auditors, plus guests on occasion.
The other important reason, of course, is to generate DNSSEC signatures. Occasionally other activities take place as well, such as generating new keys, retiring equipment, and changing TCRs. In this post, we’ll focus only on the signature generation procedures.
A month or two before each ceremony, Verisign generates a file called the Key Signing Request (KSR). This is an XML document which includes the set of public key records (both KSK and ZSK) to be signed and then used during the next calendar quarter. The KSR is securely transmitted from Verisign to the Internet Assigned Numbers Authority (IANA), which is a function of ICANN that performs root zone management. IANA securely stores the KSR until it is needed for the upcoming key signing ceremony.
Each quarter is divided into nine 10-day “slots” (for some quarters, the last slot is extended by a day or two) and the XML file contains nine key “bundles” to be signed. Each bundle, or slot, has a signature inception and expiration timestamp, such that they overlap by at least five days. The first and last slots in each quarter are used to perform ZSK rollovers. During these slots we publish two ZSKs and one KSK in the root zone.
The root zone KSK private component is held inside secure Hardware Security Modules (HSMs). These HSMs are stored inside locked safes, which in turn are kept inside locked rooms. At a key signing ceremony, the HSMs are taken out of their safes and activated for use. This all occurs according to a pre-defined script with many detailed steps, as shown in the figure below.
Also stored inside the safe is a laptop computer, its operating system on non-writable media (i.e., DVD), and a set of credentials for the TCRs, stored on smart cards and locked inside individual safe deposit boxes. Once all the necessary items are removed from the safes, the equipment can be turned on and activated.
The laptop computer is booted from its operating system DVD and the HSM is connected via Ethernet for data transfer and serial port for console logging. The TCR credentials are used to activate the HSM. Once activated, a USB thumb drive containing the KSR file is connected to the laptop and the signing program is started.
The signing program reads the KSR, validates it, and then displays information about the keys about to be signed. This includes the signature inception and expiration timestamps, and the ZSK key tag values.
FreshRSS Validate and Process KSR /media/KSR/KSK46/ksr-root-2022-q4-0.xml...
# Inception Expiration ZSK Tags KSK Tag(CKA_LABEL)
1 2022-10-01T00:00:00 2022-10-22T00:00:00 18733,20826
2 2022-10-11T00:00:00 2022-11-01T00:00:00 18733
3 2022-10-21T00:00:00 2022-11-11T00:00:00 18733
4 2022-10-31T00:00:00 2022-11-21T00:00:00 18733
5 2022-11-10T00:00:00 2022-12-01T00:00:00 18733
6 2022-11-20T00:00:00 2022-12-11T00:00:00 18733
7 2022-11-30T00:00:00 2022-12-21T00:00:00 18733
8 2022-12-10T00:00:00 2022-12-31T00:00:00 18733
9 2022-12-20T00:00:00 2023-01-10T00:00:00 00951,18733
...PASSED.
It also displays an SHA256 hash of the KSR file and a corresponding “PGP (Pretty Good Privacy) Word List.” The PGP Word List is a convenient and efficient way of verbally expressing hexadecimal values:
SHA256 hash of KSR:
ADCE9749F3DE4057AB680F2719B24A32B077DACA0F213AD2FB8223D5E8E7CDEC
>> ringbolt sardonic preshrunk dinosaur upset telephone crackdown Eskimo rhythm gravity artist celebrate bedlamp pioneer dogsled component ruffled inception surmount revenue artist Camelot cleanup sensation watchword Istanbul blowtorch specialist trauma truncated spindle unicorn <<
At this point, a Verisign representative comes forward to verify the KSR. The following actions then take place:
The signing program outputs a new XML document, called the Signed Key Response (SKR). This document contains signatures over the DNSKEY resource record sets in each of the nine slots. The SKR is saved to a USB thumb drive and given to a member of the Root Zone KSK Operations Security team. Usually sometime the next day, IANA securely transmits the SKR back to Verisign. Following several automatic and manual verification steps, the signature data is imported into Verisign’s root zone management system for use at the appropriate times in the next calendar quarter.
Keeping the internet’s DNS secure, stable, and resilient is a crucial aspect of Verisign’s role as the Root Zone Maintainer. We are honored to participate in the key signing ceremonies with ICANN and the TCRs and do our part to help the DNS operate as it should.
For more information on root key signing ceremonies, visit the IANA website. Visitors can watch video recordings of previous ceremonies and even sign up to witness the next ceremony live. It’s a great resource, and a unique opportunity to take part in a process that helps keep the internet safe for all.
The post Verisign’s Role in Securing the DNS Through Key Signing Ceremonies appeared first on Verisign Blog.
This blog was also published by APNIC.
With so much traffic on the global internet day after day, it’s not always easy to spot the occasional irregularity. After all, there are numerous layers of complexity that go into the serving of webpages, with multiple companies, agencies and organizations each playing a role.
That’s why when something does catch our attention, it’s important that the various entities work together to explore the cause and, more importantly, try to identify whether it’s a malicious actor at work, a glitch in the process or maybe even something entirely intentional.
That’s what occurred last year when Internet Corporation for Assigned Names and Numbers staff and contractors were analyzing names in Domain Name System queries seen at the ICANN Managed Root Server, and the analysis program ran out of memory for one of their data files. After some investigating, they found the cause to be a very large number of mysterious queries for unique names such as f863zvv1xy2qf.surgery, bp639i-3nirf.hiphop, qo35jjk419gfm.net and yyif0aijr21gn.com.
While these were queries for names in existing top-level domains, the first label consisted of 12 or 13 random-looking characters. After ICANN shared their discovery with the other root server operators, Verisign took a closer look to help understand the situation.
One of the first things we noticed was that all of these mysterious queries were of type NS and came from one autonomous system network, AS 15169, assigned to Google LLC. Additionally, we confirmed that it was occurring consistently for numerous TLDs. (See Fig. 1)
Although this phenomenon was newly uncovered, analysis of historical data showed these traffic patterns actually began in late 2019. (See Fig. 2)
Perhaps the most interesting discovery, however, was that these specific query names were not also seen at the .com and .net name servers operated by Verisign. The data in Figure 3 shows the fraction of queried names that appear at A-root and J-root and also appear on the .com and .net name servers. For second-level labels of 12 and 13 characters, this fraction is essentially zero. The graphs also show that there appears to be queries for names with second-level label lengths of 10 and 11 characters, which are also absent from the TLD data.
The final mysterious aspect to this traffic is that it deviated from our normal expectation of caching. Remember that these are queries to a root name server, which returns a referral to the delegated name servers for a TLD. For example, when a root name server receives a query for yyif0aijr21gn.com, the response is a list of the name servers that are authoritative for the .com zone. The records in this response have a time to live of two days, meaning that the recursive name server can cache and reuse this data for that amount of time.
However, in this traffic we see queries for .com domain names from AS 15169 at the rate of about 30 million per day. (See Fig. 4) It is well known that Google Public DNS has thousands of backend servers and limits TTLs to a maximum of six hours. Assuming 4,000 backend servers each cached a .com referral for six hours, we might expect about 16,000 queries over a 24-hour period. The observed count is about 2,000 times higher by this back-of-the-envelope calculation.
From our initial analysis, it was unclear if these queries represented legitimate end-user activity, though we were confident that source IP address spoofing was not involved. However, since the query names shared some similarities to those used by botnets, we could not rule out malicious activity.
These findings were presented last year at the DNS-OARC 35a virtual meeting. In the conference chat room after the talk, the missing piece of this puzzle was mentioned by a conference participant. There is a Google webpage describing its public DNS service that talks about prepending nonce (i.e., random) labels for cache misses to increase entropy. In what came to be known as “the Kaminsky Attack,” an attacker can cause a recursive name server to emit queries for names chosen by the attacker. Prepending a nonce label adds unpredictability to the queries, making it very difficult to spoof a response. Note, however, that nonce prepending only works for queries where the reply is a referral.
In addition, Google DNS has implemented a form of query name minimization (see RFC 7816 and RFC 9156). As such, if a user requests the IP address of www.example.com and Google DNS decides this warrants a query to a root name server, it takes the name, strips all labels except for the TLD and then prepends a nonce string, resulting in something like u5vmt7xanb6rf.com. A root server’s response to that query is identical to one using the original query name.
Now, we are able to explain nearly all of the mysterious aspects of this query traffic from Google. We see random second-level labels because of the nonce strings that are designed to prevent spoofing. The 12- and 13-character-long labels are most likely the result of converting a 64-bit random value into an unpadded ASCII label with encoding similar to Base32. We don’t observe the same queries at TLD name servers because of both the nonce prepending and query name minimization. The query type is always NS because of query name minimization.
With that said, there’s still one aspect that eludes explanation: the high query rate (2000x for .com) and apparent lack of caching. And so, this aspect of the mystery continues.
Even though we haven’t fully closed the books on this case, one thing is certain: without the community’s teamwork to put the pieces of the puzzle together, explanations for this strange traffic may have remained unknown today. The case of the mysterious DNS root query traffic is a perfect example of the collaboration that’s required to navigate today’s ever-changing cyber environment. We’re grateful and humbled to be part of such a dedicated community that is intent on ensuring the security, stability and resiliency of the internet, and we look forward to more productive teamwork in the future.
The post More Mysterious DNS Root Query Traffic from a Large Cloud/DNS Operator appeared first on Verisign Blog.
