Email or username:

Password:

Forgot your password?
Jan Schaumann

Remember the X.509 PKI? You know, the one that gave us

- "Oh wait, certificate revocation is basically all broken"
- The One Where That Dutch CA Issued A Fraudulent *.google.com Cert

and my all-time favorite:

- Honest Ahmed's Used Cars and Certificates
bugzilla.mozilla.org/show_bug.

31 comments
Jan Schaumann

It's great, because it secures virtually all web traffic, and all you have to do is get a certificate from a certificate authority -- any one at all!

Don't be picky: there are literally hundreds in your trust bundle:

Terminal screenshot showing 'security find-certificate' commands.
Screenshot of macOS Keychain Access showing many, many trusted CA certs.
Jan Schaumann

But you probably would want to allow only a very small number of CAs to do that.

What can you do?

For a while, we tried dynamic HTTP Public Key Pinning (HPKP, via the Public-Key-Pins header).

But, TOFU issues aside, that was a big footgun, so we deprecated that swiftly.

Jan Schaumann

Except, we didn't: _static_ HPKP is still very much alive, even if your company long forgot they had submitted pins several years ago:

source.chromium.org/chromium/c

Jan Schaumann

Certificate Transparency (CT) was supposed to make up for (dynamic) HPKP being deprecated, but of course that shifts the defense mechanism from _prevention_ to _detection_.

Monitoring CT logs for thousands of domains is a bitch, so ~nobody does that.

Jan Schaumann

And DANE... well. For starters it requires DNSSEC, which is like asking for Linux on the Desktop, or generally available IPv6, so... yeah.

But sure, let's use the DNS, where we have "Certificate Authority Authorization" or CAA records, specified in RFC8659, and which CAs are required to honor via CA/B Forum Ballot 187 since 2017.

Jan Schaumann

CAA records let you specify which CAs you want to authorize to issue certificates for the given domain.

This is not perfect: CNAMEs get messy quickly, and you actually have to have your act together and know which domains are used where and how, e.g., with respect to third-parties you CNAME or delegate to.

Jan Schaumann

But alright, as so often, it is what it is. Still better than allowing Honest Ahmed and his uncles and cousins to issue certs for your domains.

Let's take a look at how widely used CAA records actually are.

Jan Schaumann

I looked up CAA records for ~214 million domain names in almost 1,200 TLDs as well as across the Tranco Top 1M Domains list.

In total, fewer than 3 million domains have CAA records; fewer than 50K of the Top 1M domains. That's barely 1.4% of all TLDs (4.8% of Top 1M domains) -- not that great, adoption wise.

Pie chart showing the use of CAA records across all TLDs: with (1.4%), without (98.6%)
Pie chart showing use of CAA records in the Top 1M Domains: with (4.8%), without (95.2%)
Jan Schaumann

For the domains that _do_ have CAA records set, the most common number of records is small, e.g., four or five CAs authorized to 'issue' and 'issuewild':

But over 900 domains have more than 20 CAA records, and a few even more than 50!

A table showing # of CAA records | # of domains:
10 	1,060,973
8 	841,310
1 	420,128
2 	313,908
3 	65,862
Screenshot of a terminal showing output of 'host -t caa lifelessandcalm.com'.
Jan Schaumann

The 'flags' field should be exactly either 0 or 128: the "Issuer Critical Flag" is bit 0 of the flags field, and not the _value_ of this field.

Unsurprisingly, many people get that wrong (with values encountered ranging from 0 to 250, with no clear indication what people might think those values mean):

Table showing 'flag | # of records | comment':
0 	20,064,100 	  	valid, critical flag unset
1 	219,928 	  	invalid, critical flag unset
128 	49,775 	  	valid, critical flag set
10 	735 	  	invalid, critical flag unset
250 	18 	  	invalid, critical flag set
Jan Schaumann replied to Jan

RFC8659 defines three different properties: 'issue', 'issuewild', and 'iodef'. That's it.

(CA/B Forum Ballots SC13 and SC14 added 'contactemail' and 'contactphone', but those aren't part of the RFC, and I only encountered 741 and 23 uses of these respectively, so let's ignore them.)

Jan Schaumann replied to Jan

Ignoring misspelled records ("issiue", "issuewld", "iodev", ...), here are the CAA records by tag:

Pie chart showing CAA records by tag type for all TLDs: issue (52.2%), issuewild (46.9%), iodef (0.9%)
Pie chart showing CAA records by tag type for Top1M Domains: issue (55.9%), issuewild (40.9%), iodef (3.2%)
Jan Schaumann replied to Jan

And here's the frequency of combination of these records:

Table: # of domains with | all TLDs | Top 1M domains:
issue 	2,851,746 	151,046
issuewild 	2,173,641 	110,532
iodef 	182,461 	8,622
issue and issuewild 	2,139,747 	28,910
issue, issuewild and iodef 	86,098 	4,041
issue and issuewild 	2,139,747 	28,910
iodef and issue 	178,556 	8,265
iodef and issuewild 	87,840 	4,195
either issue or issuewild and not iodef 	2,705,342 	24,869
only issue 	711,999 	17,555
only issuewild 	33,894 	1,856
only iodef 	2,163 	99
Jan Schaumann replied to Jan

The 'iodef' property can have one of three methods: mailto, http, and https.

Extra l33t points go to kyhwana.org for putting a log4j canary into their CAA record.

Table: iodef method | all TLDs | Top 1M domains
mailto 	174,230 	8,248
raw email (invalid) 	7,294 	248
https 	166 	24
http 	18 	0
Screenshot of terminal showing 'host -t caa kyhwana.org'
Jan Schaumann replied to Jan

The most frequently 'iodef' records are shown here.

Pretty pleased to see Yahoo's records dominate, since setting the right CAA policy and adding default CAA records for all of Yahoo's (many) parked domains was something I pushed for at my time there. Yay! \o/

Pie chart showing CAA iodef records for all TLDs: mailto:support@youcan.shop (20.4%), mailto:security@yahooinc.com (16.7%)
Pie chart showing CAA iodef records (Top 1M Domains): mailto:security@yahooinc.com 25.6%
Jan Schaumann replied to Jan

But let's see what CAs everybody is authorizing via their 'issue' and 'issuewild' records.

In total, I found almost 2,200 distinct 'issue' records (for domains in all TLDs, 456 distinct for the Top 1M domains) and 878 'issuewild' records (all TLDs, 227 Top 1M).

Here are the top 20 CAs:

Table: issue records (in all TLDs) | count | issue records (in Top 1M Domains) | count

1. 	letsencrypt.org 	2,769,264 	  	1. 	letsencrypt.org 	38,218
2. 	digicert.com 	2,059,878 	  	2. 	digicert.com 	29,777
3. 	comodoca.com 	2,010,652 	  	3. 	comodoca.com 	24,098
4. 	globalsign.com 	1,901,486 	  	4. 	pki.goog 	19,078
5. 	sectigo.com 	1,300,807 	  	5. 	globalsign.com 	9,522
6. 	pki.goog 	384,434 	  	6. 	sectigo.com 	8,632
7. 	trust-provider.com 	157,727 	  	7. 	amazon.com 	5,382
8. 	; 	79,788 	  	8. 	amazonaws.com 	2,545
9. 	amazon.com 	70,065 	  	9. 	amazontrust.com 	2,139
10. 	certum.pl 	32,870 	  	10. 	godaddy.com 	2,020
11. 	entrust.net 	23,103 	  	11. 	awstrust.com 	1,998
12. 	godaddy.com 	22,537 	  	12. 	entrust.net 	949
13. 	geotrust.com 	14,587 	  	13. 	certum.pl 	620
14. 	starfieldtech.com 	13,776 	  	14. 	; 	417
15. 	ssl.com 	13,484 	  	15. 	quovadisglobal.com 	407
16. 	amazonaws.com 	13,051 	  	16. 	geotrust.com 	395
17. 	amazontrust.com 	10,922 	  	17. 	symantec.com 	354
18. 	awstrust.com 	10,500 	  	18. 	trust-provider.com 	338
19. 	rapidssl.com 	9,549 	  	19. 	thawte.com 	318
20. 	comodo.com 	7,968 	  	20. 	comodo.com 	268
Jan Schaumann replied to Jan

But... let's carefully read RFC8659 again.

The 'issue' tag grants

"authorization to issue certificates containing that FQDN to _the holder of the issuer-domain-name or a party acting under the explicit authority of the holder of the issuer-domain-name_."

Jan Schaumann replied to Jan

Who is the "holder of the issuer-domain-name" for, say, geotrust.com, rapidssl.com, or thawte.com?

That's right: DigiCert.

So by specifying 'geotrust.com' in your CAA record, you are implicitly also granting the various DigiCert subsidiaries authorization, which really isn't obvious at all.

Jan Schaumann replied to Jan

If we then add up the various related CA domains, our breakdown looks more like this:

Pie chart Top CAA issue records (all TLDs): Comodo (29.9%), DigiCert (24%), letsencrypt.org (23.8%), globalsign.com (16.3%)
Pie chart CAA issue records in Top1M Domains: letsencrypt.org (25.6%), Comodo (22.5%), DigiCert (21.1%), pki.goog (12.8%)
Pie chart CAA issuewild records for all TLDs: Comodo (33%), letsencrypt.org (22.2%), DigiCert (20.5%), globalsign.com (19.2%)
CAA issuewild records for Top 1M Domains: Comodo (25.1%), letsencrypt.org (23%), DigiCert (22%), pki.goog (15.5%)
Jan Schaumann replied to Jan

Or, if you prefer Pareto charts to better illustrate the cumulative concentration:

Pareto Chart Top CAA issue records (all TLDs)
Pareto Chart Top CAA issue records (Top 1M Domains)
Pareto Chart Top CAA issuewild records (all TLDs)
Pareto Chart Top CAA issuewild records (Top 1M Domains)
Jan Schaumann replied to Jan

Authorizing a given CA can still be too broad for your taste, which is why RFC8657 specifies the 'accounturi' and 'validationmethods' parameter extensions.

There's also a draft extension for Signed HTTP Exchanges ('cansignhttpexchanges') that appears to only be supported by DigiCert and pki.goog.

The usage of these parameters is quite limited:

Table: extension parameter | count (all TLDs) | count (Top1M Domains)

cansignhttpexchanges 	259,245 	17,108
validationmethods 	559 	43
accounturi 	243 	29
account 	163 	29
root 	11 	0
policy 	9 	4
Jan Schaumann replied to Jan

In conclusion, after analyzing around 214 million domain names for CAA records, the following are worth noting:

1) CAA records are still not widely used.

Across all TLDs, only 1.4% of domains use CAA records; out of the Top 1M Domains, only 4.8%.

Considering that CAA records have been around since 2010 and honoring them has been mandatory for CAs since 2017, this seems like a pretty poor adoption rate.

Jan Schaumann replied to Jan

2) Most people don't set 'iodef'.

Those domains that do use CAA records tend to use the 'issue' and 'issuewild' records, but only minuscule fraction (0.9% of all TLDs' domains; 3.2% of the Top 1M Domains) set 'iodef'.

Jan Schaumann replied to Jan

3) Extensions are not widely used.

The dominance of the 'cansignhttpexchanges' parameter here surprised me, but could be explained by being pushed without industry agreement by Google as part of their "Accelerated Mobile Pages" (AMP) framework?

Jan Schaumann replied to Jan

And finally, and most importantly:

4) A small number of CAs dominate.

Only 7 Certificate Authorities account for over 99% of all CAA 'issue'/'issuewild' records (10 CAs for 99% of the Top 1M Domains).

3 alone account for over 75%: Comodo, DigiCert, and Let's Encrypt.

Jan Schaumann replied to Jan

Even though this only covers the small percentage of domains that do set CAA records, I would not be surprised if the overall use of CAs across all domains followed a similar -- and similarly centralized -- distribution.

(In some markets, regional players will play a bigger role; once again the inability to get access to all ccTLD zones makes this difficult to assess.)

Jan Schaumann replied to Jan

So no, you probably could replace your giant trust bundle with fewer than... 20 or so root CA certs and not notice a difference, I'd guess.

But whether that's a good thing, whether it's wise for the entire internet to place all -- well, >99% -- of its certificates/eggs into fewer than 10 CAs/baskets seems more than questionable.

Jan Schaumann replied to Jan

And that's it for today - thanks for playing "Whose Cert Is It Anyway?" โœŒ๏ธ

This thread is available as a blog post here:

netmeister.org/blog/caa-divers

Jan Schaumann replied to Jan

P.S.: This was the third blog post in a series on the centralization of the internet.

Part 1, covering NS records, can be found here:
netmeister.org/blog/nsauth-div

Or, as a Twitter thread:
twitter.com/jschauma/status/15

Go Up