Login
Hey, it's double-Troy! I'm playing with the Insta360 Link cam, a gimbal-based model that can follow you around the room. It's tiny and pretty awesome for what it is, I'm doing some back-to-back with that and my usual Sony a6400 this week. A little note on that: during the live stream someone suggested there was some lag from that camera (very minor, they suggested), but others couldn't see it. I've just been watching a bit of the video while writing up this post and I reckon they're right. Try the 3:02 mark, for example, where on Insta360 Link I have my finger up but on the Sony a6400, I don't:
It's very minor, but it's just enough to notice. Anyway, see what you think, all that a much more in weekly update 336:
Guns! You know, the things you kinda want to keep pretty well protected and out of the hands of nefarious parties, like the kinds of folks that following their data breach could match firearms to an individual at an address on a phone number of a gender and specific age. But don't worry, no financial information was compromised! 🤦♂️
All that and more in the 337th addition of my weekly update, enjoy!
What if I told you... that you could run a website from behind Cloudflare and only have 385 daily requests miss their cache and go through to the origin service?
No biggy, unless... that was out of a total of more than 166M requests in the same period:
Yep, we just hit "five nines" of cache hit ratio on Pwned Passwords being 99.999%. Actually, it was 99.9998% but we're at the point now where that's just splitting hairs, let's talk about how we've managed to only have two requests in a million hit the origin, beginning with a bit of history:
Optimising Caching on Pwned Passwords (with Workers)- @troyhunt - https://t.co/KjBtCwmhmT pic.twitter.com/BSfJbWyxMy
— Cloudflare (@Cloudflare) August 9, 2018
Ah, memories 😊 Back then, Pwned Passwords was serving way fewer requests in a month than what we do in a day now and the cache hit ratio was somewhere around 92%. Put another way, instead of 2 in every million requests hitting the origin it was 85k. And we were happy with that! As the years progressed, the traffic grew and the caching model was optimised so our stats improved:
There it is - Pwned Passwords is now doing north of 2 *billion* requests a month, peaking at 91.59M in a day with a cache-hit ratio of 99.52%. All free, open source and out there for the community to do good with 😊 pic.twitter.com/DSJOjb2CxZ
— Troy Hunt (@troyhunt) May 24, 2022
And that's pretty much where we levelled out, at about the 99-and-a-bit percent mark. We were really happy with that as it was now only 5k requests per million hitting the origin. There was bound to be a number somewhere around that mark due to the transient nature of cache and eviction criteria inevitably meaning a Cloudflare edge node somewhere would need to reach back to the origin website and pull a new copy of the data. But what if Cloudflare never had to do that unless explicitly instructed to do so? I mean, what if it just stayed in their cache unless we actually changed the source file and told them to update their version? Welcome to Cloudflare Cache Reserve:
Ok, so I may have annotated the important bit but that's what it feels like - magic - because you just turn it on and... that's it. You still serve your content the same way, you still need the appropriate cache headers and you still have the same tiered caching as before, but now there's a "cache reserve" sitting between that and your origin. It's backed by R2 which is their persistent data store and you can keep your cached things there for as long as you want. However, per the earlier link, it's not free:
You pay based on how much you store for how long, how much you write and how much you read. Let's put that in real terms and just as a brief refresher (longer version here), remember that Pwned Passwords is essentially just 16^5 (just over 1 million) text files of about 30kb each for the SHA-1 hashes and a similar number for the NTLM ones (albeit slight smaller file sizes). Here are the Cache Reserve usage stats for the last 9 days:
We can now do some pretty simple maths with that and working on the assumption of 9 days, here's what we get:
2 bucks a day 😲 But this has taken nearly 16M requests off my origin service over this period of time so I haven't paid for the Azure Function execution (which is cheap) nor the egress bandwidth (which is not cheap). But why are there only 16M read operations over 9 days when earlier we saw 167M requests to the API in a single day? Because if you scroll back up to the "insert magic here" diagram, Cache Reserve is only a fallback position and most requests (i.e. 99.52% of them) are still served from the edge caches.
Note also that there are nearly 1M write operations and there are 2 reasons for this:
An untold number of businesses rely on Pwned Passwords as an integral part of their registration, login and password reset flows. Seriously, the number is "untold" because we have no idea who's actually using it, we just know the service got hit three and a quarter billion times in the last 30 days:
Giving consumers of the service confidence that not only is it highly resilient, but also massively fast is essential to adoption. In turn, more adoption helps drive better password practices, less account takeovers and more smiles all round 😊
As those remaining hash prefixes populate Cache Reserve, keep an eye on the "cf-cache-status" response header. If you ever see a value of "MISS" then congratulations, you're literally one in a million!
Full disclosure: Cloudflare provides services to HIBP for free and they helped in getting Cache Reserve up and running. However, they had no idea I was writing this blog post and reading it live in its entirety is the first anyone there has seen it. Surprise! 👋
I'm going lead this post with where I finished the video because it brought the biggest smile to Charlotte's and my faces this week:
This. Is. Amazing 😍 pic.twitter.com/wOl4kpK841
— Troy Hunt (@troyhunt) March 3, 2023
When I talked about the McLaren in this week's video, Frits made the comment "the smile on your face says it all", which absolutely nailed it. But more than that, it brings a smile to the face of everyone who sees it (I suspect the colour helps), we're just loving seeing the excitement expressed by kids and adults alike. It's so much fun 😊
Less fun is dealing with Eye4Fraud. 24 hours on from recording this video, there's still zero visible progress and I lament that this one is just going to slip beneath the radar. If you're in the breach, do push for answers, it really shouldn't be this hard. All that and more in this week's video, enjoy!
Why can't I audio right? It's my 339th video and I still make mistakes 🙂 But it came good and we got a decent show out of it with lots of interesting engagement even though doing this a lot later in the day than usual. I found the discussion around IoT door locks especially interesting as it's a real nexus of security, usability and a bit of critical thinking about real world risks. That term "security absolutism" that came up in the comments is gold, I hope you enjoy watching this episode.
I'm excited about coming to Prague. One more country to check off the list, apparently a beautiful city and perhaps what I'm most stoked about, it's the home of Prusa 3D. Writing this as I wrangle prints out of my trusty MK3S+, I'm going to do my best to catch up with folks there and see some of the super cool stuff they're doing. Other than that, this week is full of the usual; data breaches, IoT and a cold 🍺
Most of this week's video went on talking about the UniFi Dream Wall. What a unit! I mean it's big, but then it wraps a lot of stuff up in the one device too. If you watch this and have thoughts on how I can integrate it into the new garage such that it doesn't clash with the dark theme, I'd love to hear about it. I'll share more once I set it up in the coming weeks but for now, enjoy this week's video 🙂
A quick summary first before the details: This week, the FBI in cooperation with international law enforcement partners took down a notorious marketplace trading in stolen identity data in an effort they've named "Operation Cookie Monster". They've provided millions of impacted email addresses and passwords to Have I Been Pwned (HIBP) so that victims of the incident can discover if they have been exposed. This breach has been flagged as "sensitive" which means it is not publicly searchable, rather you must demonstrate you control the email address being searched before the results are shown. This can be done via the free notification service on HIBP and involves you entering the email address then clicking on the link sent to your inbox. Specific guidance prepared by the FBI in conjunction with the Dutch police on further steps you can take to protect yourself are detailed at the end of this blog post on the gold background. That's the short version, here's the whole story:
Ever heard that saying about how "data is the new oil"? Or that "data is the currency of the digital economy"? You've probably seen stories and infographics about how much your personal information is worth, both to legitimate organisations and criminal networks. Like any valuable commodity, marketplaces selling data inevitably emerge, some operating as legal businesses and others, well, not so much. In its simplest form, the illegal data marketplace has long involved the exchange of currency for personal records containing attributes such as email addresses, passwords, names, etc. Cybercriminals then use this data for purposes ranging from identity theft to phishing attacks to credential stuffing. So, we (the good guys) adapt and build better defences. We block known breached passwords. We implement two factor authentication. We roll out user behavioural analytics that identifies abnormalities in logins (why is Joe suddenly logging in from the other side of the world with a new machine?) And in turn, the criminals adapt, which brings us to Genesis Market.
Until this week, Genesis had been up and running for 4 years. This is an excellent primer from Catalin Cimpanu, and it describes how in order to circumvent the aforementioned fraud protection measures, cybercriminals are increasingly relying on obtaining more abstract pieces of information from victims in order to gain access to their accounts. Rather than relying on the credentials themselves and then being subject to all the modern fraud detection services mentioned above, criminals instead began to trade in a combination of "fingerprints" and "cookies". The latter will be a familiar term to most people (and was obviously the inspiration for the name behind the FBI's operation), whilst the former refers to observable attributes of the user and their browser. To see a very easy demonstration of what fingerprinting involves, go and check out amiunique.org and hit the "View my browser fingerprint" button. You'll get something similar to this:
Among more than 1.6M sampled clients, nobody has the same fingerprint as me. Somehow, using the current version of Chrome on the current version of Windows, I am a unique snowflake. Why I'm so unique is partly explained by my time zone which is shared by less than half a percent of people, but it's when that's combined with the other observable fingerprint attributes that you realise just how special I really am. For example, less than 0.01% of people have a content language request header of "en-US,en,en-AU". Only 0.12% of people share a screen width of 5,120 pixel (I'm using an ultrawide monitor). And so on and so forth. Because they're so unique, fingerprints are increasingly used as a fraud detection method such that if a malicious party attempts to impersonate a legitimate users with otherwise correct attributes (for example, the correct cookies) but the wrong fingerprint, they're rejected. Which is why we now have IMPaaS.
There's an excellent IMPaaS explanation from the Eindhoven University of Technology in the Netherlands via a paper titled Impersonation-as-a-Service: Characterising the Emerging Criminal Infrastructure for User Impersonation at Scale. Released only a year and a half after the emergence of Genesis, the paper explains the mechanics of IMPaaS:
IMPaaS allows attackers to systematically collect and enforce user profiles (consisting of user credentials, cookies, device and behavioural fingerprints, and other metadata) to circumvent risk-based authentication system and effectively bypass multifactor authentication mechanisms
In other words, if you have all the bits of information a website requires to persist authenticated state after the login process has successfully completed (including after any 2FA requirements), you can perform a modern equivalent of session hijacking. Obtaining this level of information is typically done via malicious software running on the victim's machine which can then grab anything useful and send it off to a C2 server where it can then be sold and used to commit fraud (from the IMPaaS paper):
Catalin's story from the early days of Genesis showed how buyers could browse through a list of compromised victims and pick their target based on the various services they had authenticated too, along with their operating system and location. Pricing was inevitably based on the value of those services with the examples below going for $41.30 each (and just like a legitimate marketplace, these were marked down prices so a real bargain!)
To make things as turn-key as possible for the criminals, buyers would then run a browser extension from Genesis that would reconstruct the required fingerprint based on the information the malware had obtained and grant them access to the victims' accounts (I'm having flashbacks of Firesheep here). It was that simple... until this week. As of now, the following banner greets anyone browsing to the Genesis website:
The aptly named "Operation Cookie Monster" is a joint effort between the FBI and a coalition of law enforcement agencies across the globe who have now put an abrupt end to Genesis. I imagine they'll be having some "discussions" with those involved in running the service, but what about the individuals who are the victims? These are the people whose identities have been put up for sale, purchased by other criminals and then abused to their detriment. The FBI approached me and asked if HIBP could be used as a mechanism to help warn victims of their exposure in the same way as we'd previously done with the Emotet malware a couple of years ago. This is well aligned with the mantra of HIBP - to do good and constructive things with data breaches after they occur - and I was happy to provide support.
There are 2 separate things that have now been loaded into HIBP, each disassociated from the other:
The Pwned Passwords API is presently hit more than 4 billion times each month, and the downloadable data set is hit, well, I don't know because anyone can grab it run it offline. The point is that password corpuses loaded into HIBP have huge reach and are used by thousands of different online services to help people make better password choices. You're probably using it without even knowing it when you signup or login to various services but if you want to check it directly, you can browse to the web interface. (If you're worried about the privacy of your password, there's a full explainer on how the service preserves anonymity but I also suggest testing it after you've changed it as a generally good practice.)
The email address search is what HIBP is so well known for and that's obviously what will help you understand if you've been impacted. Per the opening paragraph, this breach is flagged as "sensitive" so you will not get a result when searching directly from the front page or via the API, rather you'll need to use the free notification service. This approach was chosen to avoid the risk of people being further targeted as a result of their inclusion in Genesis. All existing HIBP subscribers have been sent notification emails and between individuals and those monitoring domains, tens of thousands of emails have now been sent out. Whilst the volume of accounts represented is "8M", please note that this is merely an approximation (hence the perfectly round number on HIBP), intended to be an indicative representation of scale as many of the breached accounts didn't include email addresses. This number only represents the number of unique email addresses which showed up in the data set so consider it a subset of a much larger corpus.
Let me add some final context and this is important if you do find yourself in the Genesis data: due to the nature of how the malware collected personal information and the broad range of different services victims may have been using at the time, the exposed data can differ significantly person by person. What's been provided by the FBI is one set of passwords (incidentally, as SHA-1 and NTLM hash pairs fed into the law enforcement ingestion pipeline), one set of email addresses and a list of meta data. Beyond the data already listed here, the meta data includes names, physical addresses, phone numbers and full credit card details among other personal attributes. This does not mean that all impacted individuals had each of those data classes exposed. The hope is that by listing these fields it will help victims understand, for example, why they may have observed fraudulent transactions on their card, and they can then take informed and appropriate steps to better protect themselves.
Lastly, as flagged in the intro, following is the guidance prepared by the FBI and Dutch police on how people can safeguard themselves if they get a hit in the Genesis data or frankly, just want to better protect themselves in future:
The FBI reached out to Have I Been Pwned (HIBP) to continue sharing efforts to help victims determine if they've been victimized. In this instance, the data shared emanates from the Initial Access Broker Marketplace Genesis Market. The FBI has taken action against Genesis Market, and in the process has been able to extract victim information for the purposes of alerting victims.
In all, millions of passwords and email addresses were provided which span a wide range of countries and domains. These emails and passwords were sold on Genesis Market and were used by Genesis Market users to access the various accounts and platforms that were for sale.
Prepared in conjunction with the FBI, following is the recommended guidance for those that find themselves in this collection of data:
To safeguard yourself against fraud in the future, it is important that you immediately remove the malware from your computer and then change all your passwords. Do this as follows:
How can I prevent my data being stolen (again)?
Just one more thing to end on a lighter note: a quick shoutout to whoever at the bureau slipped a half-eaten cookie into the takedown image, having been munched on by what I can only assume is a very satisfied FBI agent after a successful "Operation Cookie Monster" 😊
Next time I post a poll about something as simple as "when is next Friday", I don't expect I'll get as much interest. Of course "next time" will be whatever poll follows the last one, not the poll that falls after that one! But more seriously, I cannot think of a better example of ambiguous language that's open to interpretation and so easily avoided (hello MM-DD people!)
Also, Genesis Market and Operation Cookie Monster. This is just amazing stuff and a testament to a coalition of law enforcement agencies across the globe that have now made well over 100 arrests. Off the back of the NCA's DDoS market honeypot, the BreachForums admin arrest and the takedown of RaidForums before that, if you're playing in this space you'd have to be looking over your shoulder by now. Interesting times in cyber(crime) space.
A bit late this week as I've prioritised time out with the family doing as many New Zealand adventure things as we can. And we've seriously maxed out the time, as you can see via the FB link below. But that hasn't stopped a couple of new data breaches flowing into HIBP nor me having some pretty direct thoughts on the premise that the vast bulk of IT pros are being told not to report data breaches. I hope you enjoy this impromptu vid from a faraway location at an odd time, I'll be back to normal again next week.
I want to try something new here - bear with me here:
Data breach processing is hard and the hardest part of all is getting in touch with organisations and disclosing the incident before I load anything into Have I Been Pwned (HIBP). It's also something I do almost entirely in isolation, sitting here on my own trying to put the pieces together to work out what happened. I don't want to just chuck data into HIBP and the first an organisation knows about it is angry customers smashing out their inbox, there's got to be a reasonable attempt from my side to get in touch, disclose and then coordinate on communication to impacted parties and the public at large. Very frequently, I end up reaching out publicly and asking for a security contact at the impacted company. I dislike doing this because it's a very public broadcast that regular followers easily read between the lines of and draw precisely the correct conclusion before the organisation has had a chance to respond. And the vast majority of the time, nobody has a contact anyway but a small handful of people trawl through the site and find obscure email addresses or look up employees on LinkedIn or similar. There has to be a better way.
Yesterday, I posted this tweet:
After I shared this, multiple people said "ah, but at least we have GDPR", as though that somehow fixes the problem. No, it doesn't, at least not in any absolute sense. Case in point: I'm now going through the disclosure process after someone sent me data from a company HQ'd well… https://t.co/yMYIlFXkCU
— Troy Hunt (@troyhunt) April 18, 2023
And around the same time I got to thinking about Twitter Subscriptions as a channel for communication with a much more carefully curated subset of the 214k people that follow my public feed. Tweets within a subscription are visible only to subscribers so the public broadcast problem goes away. (Of course, you'd always work on the assumption that a subscriber could take a tweet and share it more broadly, but the intention is to make content visible to a much smaller, more dedicated audience.) Issues around where to find contact details, verification of the breach, what's in it or all sorts of other discussions I'd rather not have with the masses prior to loading into HIBP can be had with a much more curated audience.
I don't know how well this will work and it's something I've come up with on a whim (hey, I'm nothing if not honest about it!) But that's also how HIBP started and sometimes the best ideas just emerge out of gut feel. So, I set up the subscription and of the 3 pricing options Twitter suggested ($3, $5 or $10 per month), I went middle of the road and made it 5 bucks (that's American bucks, YMMV). You can sign up directly from the big "Subscribe" button on my Twitter profile or follow the link behind this text. Just one suggestion from Twitter's "welcome on board" email if you do:
Encourage your followers to Subscribe on the web. Web Subscriptions go through Stripe, which takes a 3% fee from each purchase, compared to the 30% fee that Apple and Google currently take. Meaning web Subscriptions may potentially lead to more money in your pocket.
My hope is that this subscription helps me have much more candid discussions about data breaches with people that are invested in following them than the masses that see my other tweets. I also hope it helps me go through this process feeling a little less isolated from the world and with the support of some of the great people I regularly engage with more publicly. If that's you, then give it a go and if it isn't floating your boat, cancel the subscription. I think there's something in this and I'd appreciate all the support I can get to help make it a worthwhile exercise.
I feel like a significant portion of this week's video went to discussing "the Coinbase breach that wasn't a Coinbase breach". There are various services out there that are used by the likes of password managers to alert their customers to new breaches (including HIBP in 1Password) and whoever Dashlane is using frankly, royally cocked up the attribution. What was a garden variety list of email addresses someone had just chucked the "Coinbase" name on had absolutely nothing to do with a breach of the crypto company. It's frustrating to watch, and I suspect that will come through when you watch the video too. See what you think.
I stand by my expression in the image above. It's a perfectly accurate representation of how I looked after receiving the CityJerks breach, clicking on the link to the website then seeing what it actually was 😳 Fortunately, the published email address on their site did go through to someone at TruckerSucker (😳😳) so they're aware of the breach and that it's circulating broadly via a public hacking website. That segment is last up in this week's video and I do give fair warning just in case you're not in the best environment to be watching that part of the update. Viewer discretion advised!
I wish I'd read this blog post years ago.
I don't have any expertise whatsoever to be guiding others through this process so please don't look at this as a "how to". But what I do have is an audience, and I've found that each time I've opened up about the more personal aspects of my life and where I've struggled (such as my post a few years ago on dealing with stress), I've had a huge amount of feedback from people that have been helped by it.
Just read this. Hugely helpful to me going through the never ending stress of divorce. It had given me hope and focus. Thank you 🙏
— Ruth Cornish (@RuthACornish) March 16, 2022
Perhaps my willingness to talk openly about it has led to others coming to terms with their own similar circumstances, and that's my hope in writing this. Not to guide you through divorce, but to help you understand it.
Here's what I've learned.
This title is deliberately blunt, and I chose to run with it because it's one of the most important things I've learned throughout this process. Let me explain:
Nobody goes into a marriage expecting it to fail. You're marrying for life and when the day comes that you realise it's not going to happen, you feel like a failure. You also feel stigmatised; "I've not been able to deliver on the promise of my marriage, what will people think of me? What will my family think? My kids?" Now, I'm not religious in any way whatsoever but I'm conscious of the social expectation of marriage. I found it extremely difficult early on to talk to people about it outside my closest circle of friends, partly because I had difficulty simply finding the words to explain it and partly because to be honest, I was worried about being judged for having a failed marriage.
It took time to realise that people don't care or more specifically, they don't care about stigmatisation or judgement or most of the other emotional baggage you get caught up with. Friends care about you, of course, but the fact you've decided to dissolve a marriage really isn't their concern. Your wellbeing is their concern. Your happiness. Your children's happiness. The person caring most about the mechanics of divorce and leading separate lives was me, and that's something I had complete control over.
The exception to "nobody cares", in my experience, is family and others in your inner circle who hold onto that same stigma that dogged me early on. There may be traditional reasons for this, cultural reasons, religious reasons or just the simple sadness for a relationship that is no more. Family in particular can be complex, especially when there may be existing resentment, jealousy or as we've all experienced at one time or another, individuals who just revel in drama. They can be your greatest supporters or, if they prefer, antagonists. But that's their own emotional struggle to deal with, not yours, and a truly supportive inner circle will prioritise your wellbeing.
Where it really started to normalise for me was over the course of time as I learned how many other people had, themselves, gone through divorce. Sometimes it was much simpler happening earlier in life and without kids, but often it was much, much more complex, especially where there was financial distress or older children. Once I came to terms with the fact that the concept of a marriage falling apart is not the thing I should be worried about and I should instead focus on the logistics of the various practical challenges that presents, I became a lot more comfortable with the situation and frankly, a lot happier.
Someone I spoke with recently was married to an abusive drunk. They knew the relationship was over when they found themselves thinking how easy it would be to push their inebriated spouse down the stairs.
A friend confided in me about how their partner had physically assaulted them since the birth of their child. The kid was about to enter adulthood.
Another friend explained recently how it wasn't until their wedding night they acknowledged they were gay.
Not all stories are as dramatic; one friend is happily married with two children of their own and two their partner had in a previous marriage. Another divorced young and now lives happily with their new partner, the child they had together, their partner's ex and the child they'd had previously.
I've deliberately used gender-neutral pronouns here; it surprised me how often personal stories didn't align to the stereotypical norms of male and female behaviours. Especially in cases where there has been mental illness, alcohol dependency or drug addiction, you realise just how unique everyone's own journey is and how even though your own may feel exceptional, it's probably not.
I mention this here because as my life started to settle down, that headline kept coming back up - "everyone has their own story". As time went by and I met new people and heard new stories, it would come up over and over again in my mind. It helped me normalise my own circumstances and overcome the stigma I'd felt so much in the early days.
It's tricky when there's mutual friends, common contacts in social circles, other parents at school and all sorts of scenarios where you're going to be spending time with the same people your ex is. Whose side do they take? Who are they sympathetic to? Angry towards?
There's a temptation to inject your own views into discussions with these people but frankly, the chances of doing that in a balanced fashion in the midst of the most emotional period of your life are zilch. I know when I think back to conversations with friends who've gone through similar trauma in their own lives, I'm acutely aware that as much as I want to be there to support them, I'm only hearing half the story. One friend in particular I've spent a lot of time with is convinced their ex is actively turning their kids against them and they may very well be right, but I only hear one side of the story. Another was concerned their child had been abused whilst in their ex's care and again, I've only heard one story. But to my earlier headline, I don't care because I'm not listening to their stories so that I can play judge, I'm listening to help them get it off their chest and deal with their emotions.
What I found over the course of years was that when it comes to mutual friends, it was preferable to simply not discuss the ex. It might come up organically (which parent will the kids be with when a friend wants a play date, for example), but that's a discussion that can be had in a pure mechanical fashion without emotion. It's harder when more pointed questions are asked - "How's it going with the divorce?" - and candid, honest responses aren't always compatible with the goal of remaining neutral. Interestingly, I found that people judged bitterness towards the other party quite harshly, especially where they viewed behaviour as derogatory. "Why can't they just get over it and move on", I'd keep hearing. It feels almost trite to put it this way, but people respond well to positivity and just getting on with life, but judge negativity and bitterness quite harshly.
Giving people time and space to observe without feeling like someone is trying to influence their views is invaluable and, in my experience, led to much more support.
During the good days of my marriage, I knew my wife wanted the best for me as I did for her. After all, that's the bedrock of a relationship: that you're there to support each other and wish for nothing other than their happiness. Divorce changes that and in many cases, inverts that bedrock yet somehow your brain is still wired to want the best for them and in turn, to expect that they want the best for you.
During the divorce process, there was constant strategising about how to move matters forward and drive the formal things to a conclusion and time and time again, I'd talk to my lawyer and say "she's telling me she'd like [blah]". In this context, [blah] was normally something that had the optics of good intentions, often motherhood statements such as "desirous of an amicable outcome" or words like "fair", "kind" and "considerate". Who wouldn't want these things?! These things are all great! At one stage I relayed this messaging to him after which he paused, and then asked a very simple question:
What do her actions tell you?
Uh... something different. Opposite.
It was the "be kind" of misdirection where someone says words you naturally support (of course we all should be kind!) yet demonstrate actions to the contrary. Do you judge them on the words? Or the actions? Of course it should be the latter, but that realisation only comes once you recognise that the two don't always align.
The problem is the aforementioned brain wiring where you're conditioned to expect the other party to want the best for you and to take them at their word. It's hard to let go of the fact that your wellbeing is no longer their first priority and frankly, the inverse is also true. But that doesn't change the intention being represented so we need to move beyond judging on words and start judging on behaviour. I later heard this same sentiment expressed in a more eloquent way:
Characterise people by their actions and you will never be fooled by their words
This was another epiphany for me, and it fundamentally changed the way I viewed the situation. If I'm honest, it gave me a lot more clarity of mind; it forced me to let go of many of the emotions surrounding the divorce and instead just focus on the facts. The motherhood statements and platitudes no longer mattered, all that mattered was actions.
I read a lot to try and help me understand what was going on, particularly in the earlier days of separation. One piece I read really resonated as it helped explain how two people who were once so close can now be on totally different wavelengths and have different versions of the same events. The piece I read was about the Rashomon Effect:
The effect is named after Akira Kurosawa's 1950 film Rashomon, in which a murder is described in four contradictory ways by four witnesses. The term addresses the motives, mechanism, and occurrences of the reporting on the circumstance and addresses contested interpretations of events, the existence of disagreements regarding the evidence of events, and subjectivity versus objectivity in human perception, memory, and reporting.
Same event, different perceptions of what happened. The Rashomon Effect doesn't help explain what actually happened, rather it describes how people in a highly emotional, life-changing time of their lives can have fundamentally different views of the same circumstances. This might sound kind of clinical and detached, but it helped explain behaviour that I simply couldn't rationalise before. Recognising that people can have different perceptions of the same events that led to the separation helped me deal with the grief. But probably about a year after separating, I had an epiphany that really helped me move forward: the root cause didn't matter anyway.
It's only natural to seek out answers and, indeed, to apportion blame. I've done it, others in similar situations I've spoken to have done it and should you ever find yourself in the same place as me, you'll do it too. It's not always just blame with the other party either and I suspect there's rarely a divorce where all the fault lies purely on one side.
I found all the reasons in the world to explain why this had happened. Recent incidents, things related to money, to work, to kids and even signs that I should have picked up on right from day one of the relationship. I'm sure she did the same. But ultimately, it's inconsequential and my pinning the blame on particular things was making no difference whatsoever.
Legally, it doesn't matter either. In Australia (and in many other parts of the world), we've had the concept of No Fault Divorce since 1975. The court doesn't care who did this or who didn't do that, all they care about is that there's an irretrievable breakdown of the relationship and that either one or both of you wants the marriage annulled. That is all.
In playing back all the events over all the years, I was just reliving bad memories. It was making me angry, regretful, emotional. I wasn't longing for reconciliation and as I moved forward in my own life, I wasn't even wishing things were back where they were years ago. I was seeking answers where I wasn't going to find them, and they wouldn't change a thing today even if I did.
Stop dwelling on it and move on. I can't say I always do that, but the more I've just put my head down, looked to the future and powered forward, the better things got.
Telling the kids was the worst. In the hours beforehand, I was a mess. Inconsolable. I felt that stigma I mentioned earlier coming over me in waves as we prepared to tell our children we were breaking up the family.
Their mother was the one who told them as we all sat down together. It was pretty short and to the point, effectively boiling down to us having mutually decided to lead separate lives. The bomb was dropped, then she finished by prompting the kids for any questions they'd like to ask us. They paused, then our 9-year old son spoke up:
Can we have pizza for dinner tonight?
I smile thinking about that even now 🙂 It was a relief valve at an enormously stressful time not just because it was kinda funny given the gravitas of the news they'd just heard, but because it demonstrated that just as with the observations above about friends not caring, the kids didn't care either. They cared about being loved, supported, having their parents' attention and really, just fundamental Maslow's hierarchy of needs sort of stuff. They didn't understand the social concept of marriage, they weren't aware of the stigma I felt and frankly, if it didn't have any actual impact on their lives in any meaningful sort of way, they didn't care.
In later reading I'd learn that as far as divorces and kids go, this is the ideal time to do it. Were they to be much older (our daughter was 6 at the time), things would be harder as they became more independently minded and more aware of the social issues surrounding a marriage breakdown. But at this age and in an environment that was still civil at the time, both the news on that day and everything else I've observed in the years since has been entirely unnoteworthy.
But I also don't want to trivialise the situation with kids as I've seen things work very differently for other people, especially when teenagers are involved. I can only relay my own experiences here and acknowledge that I've been extraordinarily fortunate. Part of that good fortune has been luck due to the timing of our separation, their age and their personalities, and part of it has been good management on our behalf as parents.
What I've found most difficult to navigate is loyalty binds:
A loyalty bind in divorce is where the child does not feel allowed to love both parents. He has to side with one or the other about any number of issues, big and small. His anger, sadness, and anxiety increases as he feels pushed to choose and either choice results in the loss, or fear of loss, of the other parent. He can’t win.
When you've got two people who've decided to wind up a relationship, there's going to be flashpoints. Disagreements. Possibly legal battles. You're both angry, both convinced you're right and sometimes, certain that the other party is the devil. Now, imagine amidst the heights of that frustration a parent gives the kids some pretty unfiltered opinions about the other party - how do the kids react? Angry towards the parent being spoken harshly of due to the things they've allegedly done? Or defensive of that parent as they watch the other one unleashing on them? He can't win!
I've found this to be an extremely delicate area to navigate for two reasons:
Firstly, I've had to make sure that no matter how I've felt about the situation, I avoid negativity towards the ex in front of the kids to the fullest extent possible. Sometimes that's easy insofar as there are many discussions that simply don't need to be had with the kids (it's much better to vent to close friends and family), but other times it can be extremely difficult if it's a topic that directly impacts them (e.g., their movements over school holidays). But that burden is on us - the adults - and it's one the kids shouldn't have to bear.
Secondly, there's dealing with times where the other parent puts the kids in the very position you're trying so hard to avoid. Particularly when derogatory messaging comes home in ways that could only have come from the other party, you're left feeling defensive and wanting to set them straight with your version of the record, but now you're back at the loyalty bind problem. It's not always explicitly derogatory behaviour that creates that loyalty bind either, it can be something as minor as being emotional when the kids mention the other party or particular activities they've been involved in; "every time I talk about [thing], it makes [mum|dad] upset".
In dealing with the latter situation, I sought support from a family counsellor who gave me an example from another client that epitomises everything that is wrong with creating loyalty binds. A lady had attended with her 6-year old daughter and during the session, received a call from her lawyer related to matrimonial matters. After hanging up, she burst into tears and in an attempt to calm her, the daughter put her arms around the mother and said, "that's ok mum, I hate dad too". That story is just heartbreaking and even though it may not have been the mother's intention, her reaction drove a wedge between a child and their parent. That's a hard one but again, we're the adults, it's our responsibility to manage our emotions around these situations.
I keep coming back to what is ultimately a very simple premise: putting the kids in a situation where it creates a loyalty bind is a selfish act that prioritises your emotions over the kids' wellbeing. Whether it's deliberate or accidental, it must be avoided to the fullest extent possible.
I originally started seeing a psychologist to help me deal with stress and sustain my performance when I felt everything was getting too much for me. It quickly became clear that the bulk of my stress wasn't due to my workload, it was due to my relationship. This is where psychologists can make a big difference - cutting through the emotion and getting to the core of what's eating you.
So, I started seeing Clive. He wasn't the first psych I'd seen, but he was the first one that really resonated with me, so I made appointments to see him every couple of weeks. We'd spend an hour each time going through recent events, how they made me feel and how I'd deal with them moving forward. He made an enormously positive difference not just in terms of understanding my own emotions, but reframing situations to reduce the unnecessary stress I was feeling.
Here's a perfect example: I'd often worry about things that were really of very little consequence but would bug the hell out of me. They'd come through an email, via the kids or in a lawyer's letter. On one occasion, I unloaded the whole lot onto Clive after which he sat thoughtfully, then suggested the following:
Think of her as a drunk person in a pub throwing punches at you. It's demanding your attention, but nothing is connecting and eventually she'll tire out or sober up.
I loved that and ever since that session, I've become much more adept at separating the things that actually require my emotional input from the drunk punches.
The other exceptionally helpful guidance he gave came during a protracted legal stoush that felt like it had no end in sight:
Me: "This feels like it will never end"
Clive: "Do you know what to do next?"
Me: "Yes, I'm going to do [legal thing] then [other legal thing] then if that doesn't work, [alternating legal thing]."
Clive: "Then just follow the process."
Follow the process. Time and time again, I'd sit on the couch, pour out my heart and we'd come back again to simply following the process. Divorce paperwork - follow the process. Parenting orders - follow the process. Financial settlement - follow the process. At their essence, they were merely business deals and negotiations, they just happened to be wrapped up in multiple layers of emotions.
In a later session, there was one addition to this guidance; follow the process and sustain performance. You can't let the process sap you of energy such that you're unable to perform. You can't let it mentally or emotionally drain you, distract you from life's essentials or keep you from reaching the goal. This was the high-performance coaching I was seeking out in the first place, and it was more relevant at this juncture than ever before.
That's not to say that following the process is simple, it certainly wasn't for me, and those layers of emotions would regularly impede progress. Clive would often break it down into psychological behaviours he'd plot out on the whiteboard:
I'd rarely take notes, but I'd take photos. I'd go back through them later on in an attempt to make sense of it all. Professional help made an enormously positive difference; it helped me process everything going on in my life, understand it more objectively and ultimately, lead a happier life. Speaking of which...
Every person I spoke to who'd been through divorce and "emerged on the other side" told me the same thing - it gets better. Clive told me that from day 1, pointing out that there's a very predictable cycle we all go through:
This maps pretty closely to your classic Kübler-Ross 5 stages of grief and we recognised that I was somewhere around the righthand side of the whiteboard. It's not always movement in the one direction, indeed I was oscillating back and forth around "understanding of new normal", sometimes a couple of steps backwards towards "anger", but increasingly towards "engaging and embracing new normal". And my new normal was starting to look pretty damn good:
Let’s face it. Everyone who’s as slick as Troy and done as much for the sec community deserves a girl that fine!😂 #yesG
— .b (@dot_b) June 4, 2021
I love this comment, not because I alone somehow deserve romantic happiness, but because we all do. To inject further optimism into the end of this post, upon reflection, every single story I relayed above about friends who have gone through their own divorce struggles has resulted in new partners, new lives and new happiness. Every. Single. One. Charlotte and I got engaged on New Year's Day two years ago and married in September. Life has never been better 😊
Sometimes, life feels like a fairytale. This is now my favourite photo ever ❤️ pic.twitter.com/lspKwVVSly
— Troy Hunt (@troyhunt) December 9, 2022
One final note on this, a quote from Lao Tzu:
If you are depressed you are living in the past.
If you are anxious you are living in the future.
If you are at peace you are living in the present.
There are still "drunk punches" and occasional anxious moments, but they're increasingly fleeting and I'm at peace. I hope this post has been helpful and if you recognise yourself in this, that you reach this stage of the process quickly and peacefully.
It's a bit of a mixed bag this week with a very light-hearted look at the death of the browser padlock icon (which has been replaced by an icon that looks like a sex act), and a much more serious discussion about divorce. It took a long time to write and be ready to publish that blog post, many years in fact, but I'm so glad I did. You don't have to scroll far through the responses to the launch tweet or the comments on the blog itself to get a sense of how it's impacted people, and as I said in the very opening of the post, this sort of openness tends to be really well received. Wherever you are in your own stage of life, I hope you enjoying reading that post and share it generously with those for whom it might just make a real difference.
A late one this week as I cover from the non-stop conferencing that was the Azure user group in Perth, followed by the Cyber West keynote, then the social drinks that night, the flight back home straight into the AusCERT gala dinner, the panel on data governance that morning then wrapping up with the speed debate Friday arvo. I think that's all... Anyway, better later than never and nothing too serious in this week's update. Personally, I'm finding the house works the most fun to talk about so I'm going to hit the publish button on this post now then go back to drafting the blog series on everything we've done 😊
I feel like the .zip TLD debate is one of those cases where it's very easy for the purest security view to overwhelm the practical human reality. I'm yet to see a single good argument that is likely to have real world consequences as far as phishing goes and whilst I understand the sentiment surrounding the confusion new TLDs with common file types, all "the sky is falling" commentary I've seen is speculative at best. But hey, there's no rolling it back now, we can start judging by what actually happens with the TLD rather than sitting around creating misuse hypotheses.
This week's update is dominated by my experience with "Lena", the scammer from Gumtree who tried to fleece my wife of $800. There's a blow-by-blow rundown of how it all happened in this video and it's fascinating to think that these things can actually be successful given all the red flags. But they are, and in Australia alone innocent victims are stung to the tune of more than 3 billion dollars every year by fraudsters which is a staggering number. Understanding how these scams work and sharing that knowledge broadly with the less technical of those around us is part of how to combat this, so please share the tweet thread generously... and enjoy the entertainment 😊
And so ends a long period of back-to-back weeks of conferences and talks. It's funny how these things seem to cluster together at times and whilst the last 6 or 8 weeks (I honestly lose track!) have been chaotic, I've now got a few weeks of much less pressure which will give me time to finally push out some HIBP stuff that's been in the wings for ages. I've just got to get through this weekend first, stay tuned for pics on social for that, it's going to be pretty epic 😎
I spent most of this week's update on the tweaking I went through with Azure's API Management service and then using Cloudflare to stop a whole bunch of requests that really didn't need to go all the way to the origin (or at least all the way to the API gateway sitting in front of the origin Azure Function instance). I'm still blown away by how cool this is - tweak the firewall via a web UI to inspect traffic and respond differently based on a combination of headers and response codes and bam! A massive reduction in unnecessary traffic follows. That's so cool, I love cloud 😊
There are presently 201k people monitoring domains in Have I Been Pwned (HIBP). That's massive! That's 201k people that have searched for a domain, left their email address for future notifications when the domain appears in a new breach and successfully verified that they control the domain. But that's only a subset of all the domains searched, which totals 231k. In many instances, multiple people have searched for the same domain (most likely from the same company given they've successfully verified control), and also in many instances, people are obviously searching for and monitoring multiple domains. Companies have different brands, mergers and acquisitions happen and so on and so forth. Larger numbers of domains also means larger numbers of notifications; HIBP has now sent out 2.7M emails to those monitoring domains after a breach has occurred. And the largest number of the lot: all those domains being monitored encompass an eye watering 273M breached email addresses 😲
The point is, just as HIBP itself has escalated into something far bigger than I ever expected, so too has the domain search feature. Today, I'm launching an all new domain search experience and 5 announcements about major changes surrounding it. Let's jump into it!
Every time I look at numbers related to domain searches, they stagger me. One of the stats I found particularly interesting was that of those 200k people monitoring domains, 23k of them were monitoring 2 or more domains. 8.5k were monitoring 3 or more. 4.6k were 4 or more and so on and so forth. The point being that there are a very large number of people monitoring multiple domains. In fact, 1k people are monitoring 9 or more and hundreds have gone through the manual verification process at least 2 dozen times.
To make life much, much easier on those folks monitoring multiple domains, they're now all bundled up into a centralised dashboard accessible from the existing Domain search link on the website. Because I already know who is monitoring which domains and the email address they're using for notifications, that same email address can be used to verify your identity and drop you straight into the dashboard. Here's mine:
One of the problems the dashboard approach helps tackle is unsubscribing on an individual domain basis. In the past, the only way to unsubscribe from domain notifications was to wait until one landed in your inbox then unsubscribe from every single monitored domain in one go. It was an all or nothing affair that nuked the lot of them whereas now, it's a domain-by-domain exercise.
Another problem this solves is how I respond to an often-received question: "Hey, can you tell me which domains I'm currently subscribed to". Uh, the ones you verified? Like, possibly almost a decade ag... ah, yeah, that's a poor answer! The dashboard now makes the answer crystal clear.
And finally, another massive problem it helps tackle is verification, and that brings me to the second big announcement:
I originally introduced domain searches to HIBP only 6 weeks after the project first launched. Up until this week, it functioned exactly the same way for almost a decade: plug in a domain name, verify control of it then see the results. Each and every time. What it meant is that if you wanted to search a domain, you successfully demonstrated control then you came back later and tried to search it again, you had to go back through the same process:
You'd be surprised at how many emails I get about the difficulty this poses. We don't have any of those 4 aliases on our domain. We can't add a meta tag. We can't upload a file. We can't touch DNS. It leaves me prone to asking "well do you really have control of the domain?" Thing is, "control" is a bit of a nuanced term; there are many people in roles where they don't have access to any of the above means of verification but they're legitimately responsible for infosec and responding to precisely the sorts of notifications HIBP sends out after a breach. Usually in these cases they can get support to go through the verification process, but it involves formal internal processes, ticketing, documentation and having to explain to some IT ops person why a data breach website with a funny name needs one of the above things to happen. This doesn't fix the pain of doing it once, but it does mean that it's now a one-off pain.
As the popularity of HIBP and domain searches has grown over the years, another challenge has emerged. Let me illustrate by example: in January this year, I loaded a rather large breach into HIBP:
New scraped data: Twitter had over 200M accounts scraped from a vulnerable API in 2021. Email addresses were passed in and Twitter profiles returned. 98% were already in @haveibeenpwned. Read more: https://t.co/FRBDFk3nkp
— Have I Been Pwned (@haveibeenpwned) January 5, 2023
That's a sizeable whack of data, in fact it was the 14th largest in HIBP out of the existing 644 in there at the time. It also had a massive impact on HIBP subscribers; I sent over 1 million emails to individuals using the notification service which made it the single largest corpus of notification emails we'd ever sent by a significant margin. But further, I also sent 60,851 emails to people monitoring domains. And that's when this started happening:
6 minutes later...
And so on and so forth until my inbox looked like this:
This was Azure auto-scale doing its thing and it was one of the early attractions for me building HIBP on Microsoft's PaaS offering way back in 2013. Need more resources? Just add more cloud! Job done, next problem. Except there are 2 major drawbacks with this:
Domain searches were actually one of the last remaining remnants of a resource intensive process still running on PaaS; most of the other important bits (namely email address searches and Pwned Password's k-anonymity searches) had been on Azure Functions for ages. Functions are awesome as they're "serverless" (except for the servers they run on, but don't let me get in the marketing team's way here), in that you're never deploying large logical containers of compute like with auto-scale so that solves problem 1 above.
As of now, all domain searches run on Azure Functions. There's literally no domain search logic remaining in the Azure App Service PaaS model, it's all gone. That moves things over to much more scalable infrastructure and massively reduces the likelihood of a timeout when searching a larger domain.
I didn't just want to ship a model from years ago and reproduce all the assumptions of the day, so I made a bunch of tweaks to further optimise things. These are all things that benefit both those searching domains and me running the platform as they reduce overhead on everyone.
For example, there was no point searching for a domain then listing every alias on it "@domain.com" so now you'll just see "alias@" instead. Doesn't sound like a lot, but imagine a domain with tens of thousands of results and then a heap of orgs running searches on them. More data equals more processing equals more egress bandwidth equals more latency and more cost. (Sidenote: if you're wondering "how costly can a bit of bandwidth really be", read my post from last year on How I Got Pwned by My Cloud Costs.)
The same logic extended to exporting the domain search results in Excel or JSON format - strip out the redundant data. I went even harder on the JSON front as this format is primarily used for ingestion into other apps where there's a large amount of programmatic control. So, rather than returning a heap of redundant breach metadata over and over again, now each alias just lists the name of the breach and you can match that up to the data from the breaches API. To be clear, the domain search JSON format itself was never an "API"; it wasn't designed for programmatic consumption, it required manual verification first and I set no expectation of stability. That's something that will change soon - there'll be a proper API - but I'll come back to that at the end of this post.
Something else I've been working away on in the background is to better leverage Cloudflare's WAF to minimise the impact on the origin services. For example, last week I did a thread on blocking 401 and excessive 428 responses at the edge rather than having to process them (and pay to process them) at the origin. I've been using similar logic to keep some, well, let's just call it "very excessive" domain queries under control. For example, one particular domain was searched 140 times after a breach was loaded in April, followed by another 40 times immediately after a breach the following month:
Clearly, this is just unnecessary. Remember how domain searches are a resource intensive process that hits my bottom line pretty hard? Yeah, well, not any more!
And finally on the performance front, if you were previously monitoring multiple domains and you got a breach alert, you could run a single search that bundled all the results in together. You reckon searching for one domain can be resource intensive? Try throwing a bunch of them into the one search! As the system grew and grew, this model became increasingly hard to sustain and equally, it became increasingly noisy. So now, exactly the same domains can be searched one by one which breaks the processing down into smaller, more manageable units. Hey, wouldn't it be great to have an API around that so you could just automate the entire thing? Read on!
All these tweaks along with the move to Azure Functions has made a massive difference to the performance problem mentioned earlier, but another problem remains: I'm still paying for your domain searches. Azure Functions are charged based on a combination of how long they run for and how many resources they consume. Both those factors are extraordinarily small for individual email address searches, but they're not for domain searches. That's why soon, the largest users of the service are going to see a small fee.
Pick a brand. A big brand. If I was to bet you that either the brand directly or its parent company has used the HIBP domain search feature in the past, I'd win. I wouldn't win every bet, but I'd come out on top over a bunch of them and I know this because I have the data to be confident of my odds 🙂
Knowing which big brands use which domains for their email is actually a hard metric to define:
Anyone know where I can find a list of the Fortune 500’s domains used for email accounts? There may be more than 1 per company and it may be different to their primary website.
— Troy Hunt (@troyhunt) January 15, 2023
But by cobbling enough OSINT data together, I was able to confidently demonstrate that more than half the Fortune 500 have used this service and the vast majority of those continue to do so via ongoing domain monitoring. That's awesome! And that pattern extends all the way down to much more localised brands too; My bank. My telco. My supermarket. All sorts of commercial organisations running businesses and using data sourced from HIBP to help them do so.
I started analysing the metrics back at that tweet in Jan, just the week after all the domain searches following the scraped Twitter data going into HIBP. For the last 5 months, I've been trawling through the usage patterns and watching how organisations are using the service. I also paid a lot of attention to the reactions following the change in rate limits and annual billing for the public API that enables email address searches last Nov. That's now given me a pretty good sense of how to structure a commercial domain search model. It's not final yet, but I do hope to put the finishing touches on it next month and in the interim, welcome feedback on the high-level overview of how it'll work that I'll list here in point form:
That last point in particular is hotly requested and as of a couple of months ago, already under development:
UserVoice suggestion for @haveibeenpwned to add domain search capability to the API now started! Follow along, vote and subscribe to updates here: https://t.co/Z32eC0d9nb
— Troy Hunt (@troyhunt) April 20, 2023
I'm still working through the mechanics of all this, both technically and commercially. One part of that is looking at raw numbers, for example about half of all the domains being monitored have 10 or less breached accounts on them. These aren't commercial entities of any scale and whilst I'm not saying "10 is the free tier number", clearly there are a massive number of domains that are tiny and shouldn't be at all impacted by this.
To be honest, the experience with the public API keys has taught me that it's usually not money that's the barrier to using commercial services, it's corporate procurement bureaucracy. Onboarding documentation. Vendor assessments. Tax forms. All sorts of things that demand hours of our time, often for the sake of only $3.50 per month. So we politely decline 😊 I know that will be an issue, in fact I suspect it will be the issue and a lot of the work we've been doing this year is to try and ease that pain to the fullest extent possible. I'll talk more about that once things finally launch but for now, that's the direction we're heading and the sorts of issues we're tackling in preparation.
As we approach the 10th birthday of HIBP later this year, it's hard not to look back and reflect. So much has changed in that time, yet the service still feels very much like what it was on day 1. The challenge for me over this time has been to work out how to adapt to the changes whilst keeping true to the original intent of service. Nothing has happened quickly in that regard, and the transparent fashion in which I've chosen to run HIBP has made the rationale for any change very clear to everyone. Even this blog post has been 5 months in the making, gradually evolving to reflect my thinking on the issues until I was confident enough in the path forward.
Go and use the new dashboard. Give it a good run and let me know what you think as I'm sure there are many things we can do better. And do provide your feedback on the both the changes announced here and those to come regarding the commercial tiers too, the more input we get on this the better equipped we are to make good decisions.
Domain searches in HIBP - that's the story this week - and I'm grateful for all the feedback I've received. I've had a few messages in particular since this live stream where people gave me some really excellent feedback to the point where I've now got a much clearer plan in head as to what this will look like. I need to keep writing code, revising the draft blog post to announce it then sometime in hopefully about a month, push it all live. What I'm zero'ing in on now is a free tier that covers most domains, a very low entry fee for almost every personal or small business case you can think of and then a few tiers above that to cover the rest. Do keep that feedback coming, it's all read, it's all taken onboard and I'm responding to absolutely everyone that sends it to me. If you're one of those people, thank you 😊
This feels like a week of minor frustrations with little real world consequence but they just bugged the hell out of me. Couldn't record in my office due to a weird ground loop problem, my Home Assistant instance was unexpectedly rebooting, the Yale IoT door locks had near unprecedentedly bad UX... and then I saw Miele's IoT 😭 Other than that, everything is fine 😊
I'm in Thailand! It's spectacular here, and even more so since recording this video and getting out of Bangkok and into the sorts of natural beauty you see in all the videos. Speaking of which, rather than writing more here (whilst metres away from the most amazing scenery), I'm going to push the publish button on this week's video and go enjoy it. Seeya! 😊
Alrighty, "The Social Media". Without adding too much here as I think it's adequately covered in the video, since last week we've had another change at Twitter that has gotten some people cranky (rate limits) and another social media platform to jump onto (Threads). I do wonder how impactful the 1k tweet view limit per day is for most people (I have no idea how many I usually see, I just know I've never hit the limit yet), and as I say in the video, I find it increasingly hard to tell when community outrage is evidence-based versus "because Elon". Strange times, for now I'll just keep a foot in each camp and then who knows how the whole thing will play out in the future.
Each year since 2011, Microsoft has sent me a lovely email around this time:
I've been fortunate enough to find a passion in life that has allowed me to do what I love and make a great living out of it all whilst contributing to the community in a meaningful and impactful way. In last year's MVP announcement blog post, I talked about one of my favourite contributions of all that year being the Pwned Passwords ingestion pipeline for the FBI. This year, they sent me something nice in return:
This is so cool, thanks @FBI 😊 pic.twitter.com/aqMi3as91O
— Troy Hunt (@troyhunt) June 28, 2023
Thank you to everyone that helps me on this journey by consuming the things I create. Reading my posts, watching my videos, turning up to my talks and consuming services like HIBP and Pwned Passwords. The latter is a great example of community uptake: as of today, there were 5.12 billion requests to that service in just the last 30 days 🤯 That's amazing, thank you everyone 😊
Today was a bit back-to-back having just wrapped up the British Airways Magecart attack webinar with Scott. That was actually a great session with loads of engagement and it's been recorded to so look out for that one soon if you missed it. Anyway, I filled this week's update with a bunch of random things from the week. I especially enjoyed discussing the HIBP domain search progress and as I say in the video, talking through it with other people really helps crystalise things so I think I'll keep doing that as the dev work continues. Stay tuned for more on that next week, see you then 😊
Sad news to wake up to today. Kevin was a friend and as I say in this week's video, probably the most well-known identity in infosec ever, and for good reason. He made a difference, and I have fun memories with him 😊
Felt really sad waking up and seeing “RIP Kevin” in my timeline. I doubt there is a more well known name in our industry but if he’s unfamiliar to you (or you haven’t read this book), go and grab “Ghost in the Wires” which is an exceptional read.
— Troy Hunt (@troyhunt) July 20, 2023
Kevin started regularly coming… pic.twitter.com/w1UMm7mGa8
In other news, I share a lot more on the upcoming domain search changes in this week's video and I've gotta say, I'm feeling pretty good about them. I spent most of the day after recording this writing code and drafting the blog post and I'm pretty damn happy with each right now. I'll keep sharing more info via these updates to the extent that by the time everything launches in a couple of weeks, you'll know it all anyway if you're paying attention here 😎
IoT, breaches and largely business as usual so I'll skip that in the intro to this post and jump straight to the end: the impending HIBP domain search changes. As I say in the vid, I really value people's feedback on this so if nothing else, please skip through to 48:15, listen to that section and let me know what you think. By the time I do next week's vid my hope is that all the coding work is done and I'm a couple of days out from shipping it, so now is your time to provide input if you think there's something I'm missing that really should be in there 🙂
Somewhere in the next few hours from publishing this post, I'll finally push the HIBP domain search changes live. I've been speaking about it a lot in these videos over recent weeks so many of you have already know what it entails, but it's the tip of the iceberg you've seen publicly. This is the culmination of 7 months of work to get this model right with a ridiculous amount of background effort having gone into it. Case in point: read my pain from last night about converting thousands of words of lawyer speak T&Cs from Microsoft Word to HTML. As if preparing these wasn't painful enough, trying to make them simply play nice on a web page has been a nightmare! (I settled for dumping stuff in a <pre> tag for now and will invest the time in doing it right later on.)
I hope you enjoy this week's video, I'll talk much more about the domain search bits in the next video, hopefully following a successful launch!
This is a big one. A massive one. It's the culmination of a solid 7 months of work that finally, as of now, is live. The full back story is in my blog post from mid-June about The Big 5 Announcements but to save you trawling through all of that, here are the cliff notes:
I've spent the last 8 weeks since publishing that post crunching numbers, writing code, doing loads of formal things (namely terms of use and privacy policy), and regularly talking about it on my weekly video. I've had loads of enormously useful feedback, much of which has shaped the state of the services we're launching here today. Thank you everyone who contributed, now let me get into it and explain exactly what we've come up with 🙂
We've been thinking about the best way to structure this since January. How do we take something that has been provided for free for almost a decade and put a reasonable price on it? That's a highly subjective word - reasonable - and there'll never be complete consensus, so it's more about passing the pub test where your average person will look at this and go "yeah, that seems fair enough". Let me explain the thinking and how we reached the pricing structure you'll see further down:
Firstly, we wanted most domain searches to remain free. This keeps with the spirit of HIBP's roots being a community service and ensures the data is accessible without barrier to the majority of people. It would also mean that for most people, these changes would have absolutely no impact on the way they've been using the service, not unless they want access to the new bits.
Next, we wanted to divide the commercial offerings into a manageable number of tiers. The public API key has 4 tiers and I reckon that's the sweet spot; it's not too many options, but it's enough to provide a good separation between the scale of each. We then wanted to distribute the number of domains that would fall into the commercial category roughly equally between those 4 tiers, so it was pretty much a matter of taking what was left after the free ones and dividing them into 4 groups and putting a price on them.
Finally, we wanted the first commercial tier to be easily affordable so that most people could access it without thinking twice about it. My measure for that has always been "the cost of a cup of coffee", so I went down to my favourite local and checked what I was blindly paying when I waved my watch in the general direction of the EFTPOS machine:
$6 Aussie, or just under $4 in USD. Which led us to here (all in USD from now on):
| Plan | Breached addresses | Percent of all domains | Price / m |
| Pwned 0 | Up to 10 | 60% | Free! |
| Pwned 1 | Up to 25 | 10% | $3.95 |
| Pwned 2 | Up to 100 | 10% | $16.95 |
| Pwned 3 | Up to 500 | 10% | $28.50 |
| Pwned 4 | Unlimited | 10% | $115.00 |
What you're looking at here is a list of plan names (more on that soon), the size of the domain it covers (expressed in the number of breached email addresses on it), what percentage of all domains presently being monitored in HIBP this represents and, of course, the monthly price. As with the public API, if you subscribe annually then it's "pay for 10, get 12" which means that "Pwned 1" price works out at only $3.25 a month. As I flagged in the earlier post, this is all based around the number of addresses that appear in a breach, with one important caveat I'll expand on later: this number excludes all breaches flagged as a spam list. As a rough rule of thumb, over the years I've found approximately 20% of addresses on a domain have been breached so by that logic, you'll need 55 actual email addresses on a domain before there's a cost. Or up to 130 before it costs more than a coffee a month. (If you're a stickler for detail and are thinking those percentages are too perfect, I've rounded them from their actual values of 59.1%, 9.7%, 11.3%, 10.4% and 9.4%.)
But what if you have multiple domains? Easy - the one plan will cover all your domains within the size of that plan. For example, if you have 3 domains and one has 5 breached addresses, one has 20 and one has 90, you can get a single "Pwned 2" plan and cover them all. Or get a single "Pwned 1" plan and cover just the first 2. It's pretty simple.
So that was our initial thinking - stand this up as a product that sits alongside the existing API key one then you just purchase whichever one you want. Then, Brendan gave me a much better idea - combine them altogether! You can see the gears turning around in my head as I read his suggestion and as the days progressed and I gave it more thought, it became a brilliant idea. It massively simplifies the code base, it removes a lot of confusion that I'm sure would have otherwise ensued and perhaps most importantly, it gives you all something more than you would have had otherwise. The one fly in the ointment was the price disparity; the above prices are 13% to 15% higher than the old corresponding API key ones. So, what we've decided to do is run the old prices until 8 October then revise everything to the new prices above. That gives more than 60 days' notice to everyone with an existing API key (we'll have to email everyone anyway as the terms of use have changed to incorporate the domain bits), and there's clear verbiage everywhere about the change for anyone purchasing a new subscription. Plus, it gives everyone a little incentive to lock in for a year now and delay the increase until later in 2024. Thanks Brendan! 😊
So that's the rationale. There's no change for 60% of domains that have previously been searched, a negligible cost for the next 10% of them with the remainder paying commensurately more based on their scale. But we didn't just want to whack a cost on an existing service and you're down a few bucks a month with nothing more to show for it, let's talk about new stuff!
There are two brand new features we're now offering to all commercial subscribers. Even if your domain is small and has less than 10 breached addresses on it, you can still get access to these features via the entry level plan and they're both pretty self-explanatory: API-level access and formal support.
API first as I think it's the coolest and it's exactly what it sounds like: there's now a public endpoint you can throw a domain at and get a JSON response of breached aliases and the incidents they've appeared in. It looks just like this:
FreshRSS 
GET https://haveibeenpwned.com/api/v3/breacheddomain/{domain}
hibp-api-key: [your key]Which then responds like this:
{
"alias1": [
"Adobe"
],
"alias2": [
"Adobe",
"Gawker",
"Stratfor"
],
"alias3": [
"AshleyMadison"
]
}If you're already paying for an API key, you have immediate access to this! Same key, same logic in terms of resolving the returned breach name to the full thing via the unauthenticated API that returns breach metadata, the only caveats are that is has to be a domain you've previously demonstrated you control and it has to be within your plan size (e.g. you have a Pwned 1 plan and your domains don't exceed 25 breached addresses). Otherwise:
Subscription upgrade required.Just one more thing with the domain search API: it only makes sense to hit it after a new breach is loaded. There's absolutely no point in hammering away at it non-stop as you'll only get the same result so instead, try polling the brand new API we've just added to return only the most recent breach (it's massively cached at Cloudflare anyway) and just hit the domain search API when there's a new one. But because not everybody will do this and domain searches are expensive relative to other queries, the terms and conditions include this clause:
Controls such as rate limiting may be added to the domain search API if excessive API requests are made despite no new breaches appearing since the last request.
There is a rate limit based on a variety of factors and it's possible you may receive an HTTP 429 if you request it more frequently than is necessary. The only reason I'm not going into the details of how that works here is that I expect it will adapt and change pretty frequently in response to how people use the service. What I can confidently say now though, is that if you use the domain search feature in the way it's intended to work - querying each domain after a new breach is added - you won't have a problem with rate limits.
I'm really excited to see how people will integrate this data into their existing tooling, do please let me know if you do something awesome 😊
Then there's the formal support which we offer via Zendesk at support.haveibeenpwned.com. That launched with the API key upgrades last November and since that time, we've answered almost 600 tickets. We've been trying to fine tune things to the extent that the knowledge base there answers the most common questions, but there's certainly a great deal of time that still goes into supporting the questions that pop up. Adding domain searches to the mix will inevitably increase that, possibly by a significant order of magnitude which is why we're only making this available to commercial subscribers.
So, that's the new bits. If you're in that 60% group of people with smaller domains outside of the commercial tiers, you can get access to both the API and support by subscribing to the smallest possible plan for that cup of coffee a month. We feel that's a pretty reasonable balance, and I hope you do too.
Speaking of reasonable, about those spam lists...
I mentioned sharing as much as I could in my weekly update videos, including the intended pricing structure and how it would be based on the number of breached email addresses on a domain. Several people raised a very important point as it related to the calculations: data breaches ain't data breaches or more specifically, there are breaches in HIBP that shouldn't be treated like the other ones as they artificially inflate the pwn count. Could these be excluded?
The Onliner Spambot incident was the worst culprit and in the case of one person that contacted me, it caused his personal domain to read as though hundreds of addresses had been breached when the correct number was... zero. Someone else had their domain pegged at 40 breached addresses whereas once you took this breach out, the number came down to 13. This created somewhat of a rock and hard place situation because whilst those aliases did appear in this incident, they weren't real addresses. But what's a "real" email address anyway? Or more specifically, how can I tell via a string alone whether an address is real or not? A decade ago now I wrote about how hard this is and per the comments on that post, concluded that the only way to tell for sure is to send an email and have the recipient perform some sort of explicit action such as clicking on a link. Clearly, that's not feasible in this situation but equally, putting a price on a service based on a metric that has been artificially inflated just wasn't fair.
Adding spam lists back in 2016 was the right thing to do but equally, excluding them from the number that determines the pricing tier is also the right thing to do. We've tried to make this logic as clear as possible throughout the system and focus on a simple UX that's explicit but can also provide more insight if required,
And if you're interested in which breaches specifically have been classified as a spam list, I've added a filter to the API that lists all breaches. It's an unauthenticated API you can load directly in your browser via GET request and at the time of writing, has 11 breaches on it with nearly 1.4 billion records.
The very last thing from that screen cap is the "Enable debug mode" link and for that, we need to talk about "domain creep".
Data breaches are obviously an ongoing thing. Always have been, always will be so what that means is when you look at a domain today and see, say, 20 breached accounts on it, that might be 30 breached accounts tomorrow. I think everyone who uses HIBP understands that, but it does create a bit of a problem when domain searches are priced on a metric that can "creep". What if you've just paid for a year's worth of Pwned 1 subscription and per the example here, you've suddenly got more than 25 breached accounts on your domain and can no longer search it?
The sentiment of how this should be handled was always obvious: people have to get what they pay for. We didn't want a situation where someone could be left disappointed, and our fear was that the organic increase in breaches could lead to that event. The solution was easy: when you buy a subscription at a certain scale, every domain you're currently monitoring that can be searched on the first day of the subscription can still be searched on the last day of the subscription. If you take out one year of Pwned 1 today and per the example above, the domain creeps beyond 25 breached accounts tomorrow, it'll have zero impact for the next 364 days.
I'm conscious that this concept can get confusing: domain searches are based on the number of breached accounts on the domain but not including spam lists and then locked in at the size of the domain until the next subscription renew... phew! The debug mode link mentioned above aims to show all this logic in its raw detail:
Even though domain1.com in this example has grown to 26 breached addresses, because it was 22 breached addresses when the subscription was taken out then that's the number it's locked at until it renews in August next year. I hope this is clear enough, do please leave a comment if we can do better.
Lastly, let me put some raw numbers around the "domain creep" situation as I foresee this causing concern beyond what might be warranted. Let's start with the number of unique email addresses which is approximately 6 billion. There have been about 723M records added in the last 12 months and a bunch of those will be for the same email address (shout out to everyone who was pwned again in the last year!) Further, of that number, most email addresses were already pwned. That's a link through to the Twitter feed where I broadcast the percentage of previously seen addresses and you'll see that number is regularly around the 60% to 70% range. In other words, it's probably in the order of 250M new addresses we've seen in the last year which is appx 4% of the entire corpus. So, yes, over the course of time we'll see domains slip into higher plans, but only at about the rate of CPI.
Lastly, locking domain counts for the duration of the subscription creates additional incentive to make it an annual one, and that's beyond the existing incentive of "buy 10 months, get 12 months". That's also in addition to massively cutting down on the number of times you may need to deal with corporate bureaucracy. Speaking of which...
Let me start with a story: Many years ago during my lengthy tenure at Pfizer, I pushed hard to drive us away from traditional hosting models and towards modern cloud paradigms, namely the Azure App Service. Here we had a model where you could self-service provision resources that cost about $50 per month and completely replaced a model that was costing us tens of thousands a year. It was an easy win, however... the organisation demanded vendor assessments, compliance paperwork and a billing model which, of course, was favourable to them. But Microsoft's model was "chuck your credit card in and off you go", so that's what one of my colleagues did. And paid for it himself, entirely out of his own pocket in order to save one of the world's largest companies money. My point is that I've done time on the inside and I understand the barriers organisations put in place "because reasons". I touched on this in the June post about the upcoming domain changes:
To be honest, the experience with the public API keys has taught me that it's usually not money that's the barrier to using commercial services, it's corporate procurement bureaucracy. Onboarding documentation. Vendor assessments. Tax forms.
And so too, I have the experience from the outside having regularly received requests to invest hours doing manual labour for the sake of something an organisation is paying a few bucks a month for. That simply doesn't scale and the whole point of providing services like this at volume is that you can go and set everything up yourself with nothing more than a credit card. This one came in while preparing this blog post:
My company is looking to purchase an API key so we can automate user lookups on your site. Our procurement process is wildly complex and I was wondering if we have the option of submitting a Purchase Order instead of using the Stripe credit card payment method?
If this situation resonates, you have my sympathies and my own corporate bureaucracy scars are still raw! If there's more we can do to ease the onboarding path without creating manual labour on a per-customer basis then please let me know. I'm sure there are improvements that can be made, the last thing I want to see is you ending up like my old mate from Pfizer 😞
We've tried to do everything possible to remove barriers. We've made significant investments in legal counsel to get the terms of use and privacy policy right and we've tried to provide answers to all the regular questions in the FAQs. We've even publicly provided a W-8BEN-E US tax form which was often requested by folks in the US. But it won't be enough for some organisations, which is why we do exactly the same thing as Pfizer often found themselves doing which is to provide an enterprise-orientated process where we deal with all this rigmarole... and charge accordingly. If that's you, then get in touch with me.
There will be lots of "but what about...?" edge cases. Let me give you some examples and our views on them:
But what about addresses that don't actually exist?
For most data breaches, email addresses are extracted using a regular expression run over the entire corpus of data. You can see what this looks like in the open source email address extractor used to process breaches. So, what is an email address? Per my earlier explanation, it's anything that matches the regex when run across the breach. That could mean strings that aren't actually an address on a domain get caught up and reported incorrectly. It happens, but there's no way to practically stop it and it's extraordinarily rare.
But what about email addresses from years ago that still appear as breached on a domain?
The argument here is that whilst these are genuine addresses that did indeed exist at one point, they aren't really relevant anymore either due to their age or the address no longer existing (e.g. ex staff). I have both a philosophical and a technical view on this, with the former being that data breaches are immutable. At a point in time, addresses were exposed, and that fact can never be reversed. As for the latter point, those addresses remain in a storage construct we need to continue to support, and every single domain query needs to pick those addresses up and return them to the code processing the search (the design of HIBP means that Azure's Table Storage returns the entire partition on each domain query). Further, in most cases, that doesn't change the total number of breached accounts being a reasonable metric for organisation size and subsequently, the pricing tier they should fit into.
But what about old breaches I don't care about any more causing me to require a higher plan?
It's a similar answer to the previous point insofar as the immutability of history and the need to store the data. It also remains the most reliable metric we have to determine the size of the domain and in many cases, the organisation that owns it. Think of this measurement primarily as a means of slicing up the corpus of data within HIBP and distributing the cost as equitably as possible across the organisations using the domain search feature.
But what about people who don't want to use a credit card?
I'll give you a two-part answer on this, beginning with the recognition that cards can pose legitimate challenges for some people. Just as I was drafting this blog post, someone trying to sign up to the public API reached out after failing to subscribe multiple times with different cards:
For a variety of reasons, I believe the guy is legit, but Stripe reports two payments declined by his bank and another due to an invalid CVC. But using Stripe doesn't just mean credit cards, it also means Apple Pay and Google Pay, WeChat Pay in China, EPS in Belgium, Afterpay in Australia and a raft of other payment mechanisms in different parts of the world. It's hard to imagine a legitimate case where someone does not have access to any of the available payment mechanisms, which brings me to the second part:
The reason we don't support the likes of anonymous cryptocurrency and rely solely on fiat money payments is that it very quickly weeds out the bad actors. That was the whole rationale for putting a payment gateway on the public API back in 2019 - to cut out the abuse. It turns out that once you have to pass the sort of KYC barriers financial institutions put in place, people don't misbehave under their own identity. And yes, there's always fraudulent use of cards, but Stripe has gotten so good at handling that (we pay for their Radar service as well), our dispute rate is only one in many thousands of transactions.
But what about [other reasons related to calculations and costs]?
Amongst the corpus of 12.6 billion records, there will be anomalies. It'll almost certainly be sub-1% and the anomalies won't be evenly distributed across domains; they'll affect some more than others. It's infeasible to ever get that down to zero and it's also infeasible to respond to every single request I know will come through asking for an anomaly to be rectified. The most practical way we could find to deal with this is to keep the pricing structure such that anomalies will be unlikely to have much impact of consequence.
We're also conscious that some people will challenge the cost and it happens all the time with the existing public API key either because of the individual's position in life or the nature of the organisation they work in. But this is why we've structured it as we have, with the majority of domains being within that free tier and the entry level cost being the cup of coffee that gets you access to things like API level access and formal support. This was the most reasonable, equitable model we could come up with and I hope that shines through in the explanations above.
I know there'll be individuals with catch all domains that have ended up in a couple of dozen data breaches and they think paying $3.95 to see them is unreasonable. I know there'll be organisations with much larger numbers who feel it's unreasonable because similarly sized orgs are more profitable. But I also know that I've been running domain searches totally out of my own pocket for almost a decade so whilst I'm sympathetic to anyone who now needs to pay for a service that was previously free, I'm also comfortable that a reasonable and well thought out model has been arrived at.
I'm excited to see what people do with the new API. The email address search one is presently requested millions of times a day and people have built all sorts of amazing things with it, everything from corporate awareness campaigns to tooling to help protect customers from account takeover attacks to integration within the corporate SOC. It's cases like that last one where I think the domain search API will really shine and if you do something awesome with it, please get in touch and let me know.
I know this was a long read, I hope it adequately explains the rationale for the subscription service and that you use it to do amazing things 😊
You can get started right now from the domain search page on HIBP.
Update: Following feedback and consultation with a range of existing users of the service, we now provide a model for the education and non-profit sectors. See the KB titled Do you provide discounts based on the nature of the organisation? for more information.
So about those domain searches... 😊 The new subscription model launched this week and as many of you know from your own past experiences, pushing major new code live is always a bit of a nail-biting exercise. It went out silently on Sunday morning, nothing major broke so I published the blog post Monday afternoon then emailed all the existing API key subscribers Tuesday morning and now here we are!
One thing I talk a bit about in the video today are the 2 new APIs someone reached out and requested. This was an awesome idea and I can't wait to show you what they've built with them. I expect I'll blog that this coming week and probably quietly slip out the documentation on the 2 new endpoints in advance. Stay tuned for that one, what he's done with this looks so cool 😎
I've been teaching my 13-year old son Ari how to code since I first got him started on Scratch many years ago, and gradually progressed through to the current day where he's getting into Python in Visual Studio Code. As I was writing the new domain search API for Have I Been Pwned (HIBP) over the course of this year, I was trying to explain to him how powerful APIs are:
Think of HIBP as one website that does pretty much one thing; you load it in your browser and search through data breaches which then display on the screen. But when you have an API, it's no longer just locked into your browser, it's in all sorts of other systems. Mobile apps, other websites, dashboards and if you really want, you can even integrate the lights in your room with HIBP! Why? How? Well, there's a Home Assistant integration for HIBP and being pwned in a new breach could raise an event there you can then use YAML to perform an action with, for example flashing a light red. That might be weird and unnecessary, but when you have an API, suddenly all these things you never thought of are possible.
It took Brett Adams less than a day after we released the new domain search API last Monday for him to reach out to me with one of those ideas. He wanted to build a Splunk app (Brett is a Splunk MVP so this was right up his alley) to surface breached data about an organisation's domains right into the place where so many security engineers spend their days. He just wanted 2 new APIs to make the user experience the best it could be:
That seems so ridiculously obvious, why didn't I think of that originally?! But hey, easy fix, so the next day Brett had his APIs. And today, you also have the APIs because they're now all publicly documented and ready for you to consume. You also have Brett's Splunk app and because he's published it to Splunkbase, you can go and pull it into your own Splunk instance, plug in your HIBP API key and it's job done!
I'll leave you with a bunch of screen caps from Brett's work, starting with a zoomed in grab of what I suspect folks will find the most valuable - the addresses on their domains and their appearances across breaches:
That's a fragment of the broader dashboard that also breaks down the incidents over time:
The starting point for this is simply plugging your API key into the interface:
I like these headline figures and I picture particularly large organisations that have gone through various acquisitions of different brands with various domains finding this really useful:
And speaking of breaches, there's a lot of them which Brett has visualised across the course of time:
So that's it, you can see all the APIs documented on the HIBP website and you can grab Brett's app right now from Splunkbase. You can also find all the code for this in Brett's GitHub repo should you wish to have a read through it.
The HIBP APIs are there for other people to build awesome things. If you're one of those people, please get in touch with me and show me what you've created, I can't wait to see more integrations like Brett's 😊
This week hasd been manic! Non-stop tickets related to the new HIBP domain subscription service, scrambling to support invoicing and resellers, struggling our way through some odd Stripe things and so on and so forth. It's all good stuff and there have been very few issues of note (and all of those have merely been people getting to grips with the new model), so all in all, it's happy days 😊
There's a "hidden" API on HIBP. Well, it's not "hidden" insofar as it's easily discoverable if you watch the network traffic from the client, but it's not meant to be called directly, rather only via the web app. It's called "unified search" and it looks just like this:
It's been there in one form or another since day 1 (so almost a decade now), and it serves a sole purpose: to perform searches from the home page. That is all - only from the home page. It's called asynchronously from the client without needing to post back the entire page and by design, it's super fast and super easy to use. Which is bad. Sometimes.
To understand why it's bad we need to go back in time all the way to when I first launched the API that was intended to be consumed programmatically by other people's services. That was easy, because it was basically just documenting the API that sat behind the home page of the website already, the predecessor to the one you see above. And then, unsurprisingly in retrospect, it started to be abused so I had to put a rate limit on it. Problem is, that was a very rudimentary IP-based rate limit and it could be circumvented by someone with enough IPs, so fast forward a bit further and I put auth on the API which required a nominal payment to access it. At the same time, that unified search endpoint was created and home page searches updated to use that rather than the publicly documented API. So, 2 APIs with 2 different purposes.
The primary objective for putting a price on the public API was to tackle abuse. And it did - it stopped it dead. By attaching a rate limit to a key that required a credit card to purchase it, abusive practices (namely enumerating large numbers of email addresses) disappeared. This wasn't just about putting a financial cost to queries, it was about putting an identity cost to them; people are reluctant to start doing nasty things with a key traceable back to their own payment card! Which is why they turned their attention to the non-authenticated, non-documented unified search API.
Let's look at a 3 day period of requests to that API earlier this year, keeping in mind this should only ever be requested organically by humans performing searches from the home page:
This is far from organic usage with requests peaking at 121.3k in just 5 minutes. Which poses an interesting question: how do you create an API that should only be consumed asynchronously from a web page and never programmatically via a script? You could chuck a CAPTCHA on the front page and require that be solved first but let's face it, that's not a pleasant user experience. Rate limit requests by IP? See the earlier problem with that. Block UA strings? Pointless, because they're easily randomised. Rate limit an ASN? It gets you part way there, but what happens when you get a genuine flood of traffic because the site has hit the mainstream news? It happens.
Over the years, I've played with all sorts of combinations of firewall rules based on parameters such as geolocations with incommensurate numbers of requests to their populations, JA3 fingerprints and, of course, the parameters mentioned above. Based on the chart above these obviously didn't catch all the abusive traffic, but they did catch a significant portion of it:
If you combine it with the previous graph, that's about a third of all the bad traffic in that period or in other words, two thirds of the bad traffic was still getting through. There had to be a better way, which brings us to Cloudflare's Turnstile:
With Turnstile, we adapt the actual challenge outcome to the individual visitor or browser. First, we run a series of small non-interactive JavaScript challenges gathering more signals about the visitor/browser environment. Those challenges include, proof-of-work, proof-of-space, probing for web APIs, and various other challenges for detecting browser-quirks and human behavior. As a result, we can fine-tune the difficulty of the challenge to the specific request and avoid ever showing a visual puzzle to a user.
"Avoid ever showing a visual puzzle to a user" is a polite way of saying they avoid the sucky UX of CAPTCHA. Instead, Turnstile offers the ability to issue a "non-interactive challenge" which implements the sorts of clever techniques mentioned above and as it relates to this blog post, that can be an invisible non-interactive challenge. This is one of 3 different widget types with the others being a visible non-interactive challenge and a non-intrusive interactive challenge. For my purposes on HIBP, I wanted a zero-friction implementation nobody saw, hence the invisible approach. Here's how it works:
Get it? Ok, let's break it down further as it relates to HIBP, starting with when the front page first loads and it embeds the Turnstile widget from Cloudflare:
<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>The widget takes responsibility for running the non-interactive challenge and returning a token. This needs to be persisted somewhere on the client side which brings us to embedding the widget:
<div ID="turnstileWidget" class="cf-turnstile" data-sitekey="0x4AAAAAAADY3UwkmqCvH8VR" data-callback="turnstileCompleted"></div>Per the docs in that link, the main thing here is to have an element with the "cf-turnstile" class set on it. If you happen to go take a look at the HIBP HTML source right now, you'll see that element precisely as it appears in the code block above. However, check it out in your browser's dev tools so you can see how it renders in the DOM and it will look more like this:
Expand that DIV tag and you'll find a whole bunch more content set as a result of loading the widget, but that's not relevant right now. What's important is the data-token attribute because that's what's going to prove you're not a bot when you run the search. How you implement this from here is up to you, but what HIBP does is picks up the token and sets it in the "cf-turnstile-response" header then sends it along with the request when that unified search endpoint is called:
So, at this point we've issued a challenge, the browser has solved the challenge and received a token back, now that token has been sent along with the request for the actual resource the user wanted, in this case the unified search endpoint. The final step is to validate the token and for this I'm using a Cloudflare worker. I've written a lot about workers in the past so here's the short pitch: it's code that runs in each one of Cloudflare's 300+ edge nodes around the world and can inspect and modify requests and responses on the fly. I already had a worker to do some other processing on unified search requests, so I just added the following:
const token = request.headers.get('cf-turnstile-response');
if (token === null) {
return new Response('Missing Turnstile token', { status: 401 });
}
const ip = request.headers.get('CF-Connecting-IP');
let formData = new FormData();
formData.append('secret', '[secret key goes here]');
formData.append('response', token);
formData.append('remoteip', ip);
const turnstileUrl = 'https://challenges.cloudflare.com/turnstile/v0/siteverify';
const result = await fetch(turnstileUrl, {
body: formData,
method: 'POST',
});
const outcome = await result.json();
if (!outcome.success) {
return new Response('Invalid Turnstile token', { status: 401 });
}That should be pretty self-explanatory and you can find the docs for this on Cloudflare's server-side validation page which goes into more detail, but in essence, it does the following:
And because this is all done in a Cloudflare worker, any of those 401 responses never even touch the origin. Not only do I not need to process the request in Azure, the person attempting to abuse my API gets a nice speedy response directly from an edge node near them 🙂
So, what does this mean for bots? If there's no token then they get booted out right away. If there's a token but it's not valid then they get booted out at the end. But can't they just take a previously generated token and use that? Well, yes, but only once:
If the same response is presented twice, the second and each subsequent request will generate an error stating that the response has already been consumed.
And remember, a real browser had to generate that token in the first place so it's not like you can just automate the process of token generation then throw it at the API above. (Sidenote: that server-side validation link includes how to handle idempotency, for example when retrying failed requests.) But what if a real human fails the verification? That's entirely up to you but in HIBP's case, that 401 response causes a fallback to a full page post back which then implements other controls, for example an interactive challenge.
Time for graphs and stats, starting with the one in the hero image of this page where we can see the number of times Turnstile was issued and how many times it was solved over the week prior to publishing this post:
That's a 91% hit rate of solved challenges which is great. That remaining 9% is either humans with a false positive or... bots getting rejected 😎
More graphs, this time how many requests to the unified search page were rejected by Turnstile:
That 990k number doesn't marry up with the 476k unsolved ones from before because they're 2 different things: the unsolved challenges are when the Turnstile widget is loaded but not solved (hopefully due to it being a bot rather than a false positive), whereas the 401 responses to the API is when a successful (and previously unused) Turnstile token isn't in the header. This could be because the token wasn't present, wasn't solved or had already been used. You get more of a sense of how many of these rejected requests were legit humans when you drill down into attributes like the JA3 fingerprints:
In other words, of those 990k failed requests, almost 40% of them were from the same 5 clients. Seems legit 🤔
And about a third were from clients with an identical UA string:
And so on and so forth. The point being that the number of actual legitimate requests from end users that were inconvenienced by Turnstile would be exceptionally small, almost certainly a very low single-digit percentage. I'll never know exactly because bots obviously attempt to emulate legit clients and sometimes legit clients look like bots and if we could easily solve this problem then we wouldn't need Turnstile in the first place! Anecdotally, that very small false positive number stacks up as people tend to complain pretty quickly when something isn't optimal, and I implemented this all the way back in March. Yep, 5 months ago, and I've waited this long to write about it just to be confident it's actually working. Over 100M Turnstile challenges later, I'm confident it is - I've not seen a single instance of abnormal traffic spikes to the unified search endpoint since rolling this out. What I did see initially though is a lot of this sort of thing:
By now it should be pretty obvious what's going on here, and it should be equally obvious that it didn't work out real well for them 😊
The bot problem is a hard one for those of us building services because we're continually torn in different directions. We want to build a slick UX for humans but an obtrusive one for bots. We want services to be easily consumable, but only in the way we intend them to... which might be by the good bots playing by the rules!
I don't know exactly what Cloudflare is doing in that challenge and I'll be honest, I don't even know what a "proof-of-space" is. But the point of using a service like this is that I don't need to know! What I do know is that Cloudflare sees about 20% of the internet's traffic and because of that, they're in an unrivalled position to look at a request and make a determination on its legitimacy.
If you're in my shoes, go and give Turnstile a go. And if you want to consume data from HIBP, go and check out the official API docs, the uh, unified search doesn't work real well for you any more 😎
Somehow in this week's video, I forgot to talk about the single blog post I wrote this week! So here's the elevator pitch: Cloudflare's Turnstile is a bot-killing machine I've had enormous success with for the "API" (quoted because it's not meant to be consumed by others), behind the front page of HIBP. It's unintrusive, is super easy to implement and kills bots dead. There you go, how's that for a last minute pitch? 😊
Today, the US Justice Department announced a multinational operation involving actions in the United States, France, Germany, the Netherlands, and the United Kingdom to disrupt the botnet and malware known as Qakbot and take down its infrastructure. Beyond just taking down the backbone of the operation, the FBI began actively intercepting traffic from the botnet and instructing infected machines the uninstall the malware:
To disrupt the botnet, the FBI was able to redirect Qakbot botnet traffic to and through servers controlled by the FBI, which in turn instructed infected computers in the United States and elsewhere to download a file created by law enforcement that would uninstall the Qakbot malware
As part of the operation, the FBI have requested support from Have I Been Pwned (HIBP) to help notify impacted victims of their exposure to the malware. We provided similar support in 2021 with the Emotet botnet, although this time around with a grand total of 6.43M impacted email addresses. These are now all searchable in HIBP albeit with the incident is flagged as "sensitive" so you'll need to verify you control the email address via the notification service first, or you can search any domains you control via the domain search feature. Further, the passwords from the malware will shortly be searchable in the Pwned Passwords service which can either be checked online or via the API. Pwned Passwords is presently requested 5 and a half billion times each month to help organisations prevent people from using known compromised passwords.
Guidance for those impacted by this incident is the same tried and tested advice given after previous malware incidents:
Last week I was contacted by CERT Poland. They'd observed a phishing campaign that had collected 68k credentials from unsuspecting victims and asked if HIBP may be used to help alert these individuals to their exposure. The campaign began with a typical email requesting more information:
In this case, the email contained a fake purchase order attachment which requested login credentials that were then posted back to infrastructure controlled by the attacker:
All in all, CERT Poland identified 202 other phishing campaigns using the same infrastructure which has subsequently been taken offline. Data accumulated by the malicious activity spanned from October 2022 until just last week.
The advice to impacted individuals is as follows:
I'm super late pushing out this week's video, I mean to the point where I now have a couple of days before doing the next one. Travel from the opposite side of the world is the obvious excuse, then frankly, just wanting to hang out with friends and relax. And now, I somehow find myself publishing this from the most mind-bending set of circumstances:
Heading to 31C. Cold beer. Warm pool. How is this in England?! 🤯 pic.twitter.com/tQSbHaoLhG
— Troy Hunt (@troyhunt) September 6, 2023
On that note, straight into the video, links below and I'll do it all again in a couple of days from Spain:
I'm in Spain! Alicante, to be specific, where we've spent the last few days doing family wedding things, and I reckon we scrubbed up pretty well:
Getting fancy in Spain 😍 pic.twitter.com/iDFmBORnHa
— Troy Hunt (@troyhunt) September 9, 2023
Next stop is Amsterdam and by the end of today, we'll be sipping cold beer canal side in the 31C heat 😎 Meanwhile, this week's video focuses mostly on the Dymocks breach and the noteworthiness of what appears to be excessive data retention. After recording this video, someone also pointed out that the data is already being abused in a pretty traceable fashion:
@troyhunt not sure if this is particularly useful but I just received this scam attempt. I use iCloud's Hide My Email service and the address this email was sent to was the same address iCloud generated for use with my Dymocks account. pic.twitter.com/GiFZ7EIDo2
— matt (@matt_0833) September 9, 2023
That's all for this week, a little shorter as I was rushing for the wedding, I'll come to you next week from our second home, Oslo 🇳🇴
It's another week of travels, this time from our "second home", Oslo. That's off the back of 4 days in the Netherlands and starting tomorrow, another 4 in Prague. But today, the 17th of September, is extra special 😊
1 year today ❤️ pic.twitter.com/vsRChdDshn
— Troy Hunt (@troyhunt) September 17, 2023
We'll be going out and celebrating accordingly as soon as I get this post published so I'll be brief: enjoy this week's video!
Well that's it, Europe is done! I've spent the week in Prague with highlights including catching up with Josef Prusa, keynoting at Experts Live EU and taking a "beer spa" complete with our own endless supply of tap beer. Life is good 🍻
That’s it - we’ve peaked - life is all downhill from here 🤣 🍻 #BeerSpa pic.twitter.com/ezCpUC6XEK
— Troy Hunt (@troyhunt) September 21, 2023
All that and more in this week's video, next week I'll come to you from back home in the sunshine 😎
Ah, home 😊 It's been more than a month since I've been able to sit at this desk and stream a weekly video. And now I'm doing it with the glorious spring weather just outside my window, which I really must make more time to start enjoying. Anyway, this week is super casual due to having had zero prep time, but I hope the discussion about the ABC's piece on HIBP and I in particular is interesting. I feel like this whole story has a long way to go yet, hopefully now having a few months at home will give us an opportunity to lay the foundation for the next phase. Stay tuned!
Imagine you wanted to buy some shit on the internet. Not the metaphorical kind in terms of "I bought some random shit online", but literal shit. Turds. Faeces. The kind of thing you never would have thought possible to buy online until... Shitexpress came along. Here's a service that enables you to send an actual piece of smelly shit to "An irritating colleague. School teacher. Your ex-wife. Filthy boss. Jealous neighbour. That successful former classmate. Or all those pesky haters." But it would be weird if the intended recipient of the aforementioned shit knew it came from you, so, Shitexpress makes a bold commitment:
100% anonymous! Not 90%, not 95% but the full whack 100%! And perhaps they really did deliver on that promise, at least until one day last year:
New sensitive breach: Faeces delivery service Shitexpress had 24k email addresses breached last week. Data also included IP and physical addresses, names, and messages accompanying the posted shit. 76% were already in @haveibeenpwned. Read more: https://t.co/7R7vdi1ftZ
— Have I Been Pwned (@haveibeenpwned) August 16, 2022
When you think about it now, the simple mechanics of purchasing either metaphorical or literal shit online dictates collecting information that, if disclosed, leaves you anything but anonymous. At the very least, you're probably going to provide your own email address, your IP will be logged somewhere and payment info will be provided that links back to you (Bitcoin was one of many payment options and is still frequently traceable to an identity). Then of course if it's a physical good, there's a delivery address although in the case above, that's inevitably not going to be the address of the purchaser (sending yourself shit would also just be weird). Which is why following the Shitexpress data breach, we can now easily piece together information such as this:
Here we have an individual who one day last year, went on an absolute (literal) shit-posting bender posting off half a dozen boxes of excrement to heavy hitters in the US justice system. For 42 minutes, this bright soul (whose IP address was logged with each transaction), sent abusive messages from their iPhone (the user agent is also in the logs) to some of the most powerful people in the land. Did they only do this on the assumption of being "100% anonymous"? Possibly, it certainly doesn't seem like the sort of activity you'd want to put your actual identity to but hey, here we are. Who knows if there were any precautions taken by this individual to use an IP that wasn't easily traceable back to them, but that's not really the point; an attribute that will very likely be tied back to a specific individual if required was captured, stored and then leaked. IP not enough to identify someone? Hmmm... I wonder what other information might be captured during a purchase...
Uh, yeah, that's all pretty personally identifiable! And there are nearly 10k records in the "invoices_stripe.csv" file that include invoice IDs so if you paid by credit card, good luck not having that traced back to you (KYC obligations ain't real compatible with anonymously posting shit).
Now, where have we heard all this before? The promise of anonymity and data protection? Hmmm...
"Anonymous". "Discreet". That was July 2015, and we all know what happened next. It wasn't just the 30M+ members of the adultery website that were exposed in the breach, it was also the troves of folks who joined the service, thought better of it, paid to have their data deleted and then realised the "full delete" service, well, didn't. Why did they think their data would actually be deleted? Because the website told them it would be.
Vastaamo, the Finnish service referred to "the McDonalds of psychotherapy" was very clear around the privacy of the data they collected:
Until a few years ago when the worst conceivable scenario was realised:
A security flaw in the company’s IT systems had exposed its entire patient database to the open internet—not just email addresses and social security numbers, but the actual written notes that therapists had taken.
What made the Vastaamo incident particularly insidious was that after failing to extract the ransom demand from the company itself, the perpetrator (for whom things haven't worked out so well this year), then proceeded to ransom the individuals:
If we do not receive this payment within 24 hours, you still have another 48 hours to acquire and send us 500 euros worth of Bitcoins. If we still don't receive our money after this, your information will be published: your address, phone number, social security number, and your exact patient report, which includes e.g. transcriptions of your conversations with the Receptionist's therapist/psychiatrist.
And then it was all dumped publicly anyway.
Here's what I'm getting at with all this:
Assurances of safety, security and anonymity aren't statements of fact, they're objectives, and they may not be achieved
I've written this post as I have so many others so that it may serve as a reference in the future. Time and time again, I see the same promises as above as though somehow words on a webpage are sufficient to ensure data security. You can trust those words just about as much as you can trust the promise of being able to choose the animal the excrement is sourced from, which turns out to be total horseshit 🐎
This must be my first "business as usual" weekly update since August and damn it's nice to be back to normal! New sponsor, new breaches, new blog post and if you're in this part of the world, a brand new summer creeping over the horizon. I've now got a couple of months with very little in the way of travel plans and a goal to really knock a bunch of new HIBP features out of the park, some of which I talk about in this week's video. Enjoy! 🍻
There seemed to be an awful lot of time gone on the 23andMe credential stuffing situation this week, but I think it strikes a lot of important chords. We're (us as end users) still reusing credentials, still not turning on MFA and still trying to sue when we don't do these things. And we as builders are still creating systems that allow this to happen en mass. All that said, I don't know how we build systems that are resilient to a single person coming along and entering someone else's (probably) reused credentials into a normal browser session, at least not without introducing additional barriers to entry that will upset the marketing manager. And so, I'm back at the only logical conclusion I think we can all agree on right now: it's a great time to be working in this industry 😊
I did it again - I tweeted about Twitter doing something I thought was useful and the hordes did descend on Twitter to tweet about how terrible Twitter is. Right, gotcha, so 1.3M views of that tweet later... As I say in this week's video, there's a whole bunch of crazy arguments in there but the thing that continues to get me the most in every one of these discussions is the argument that Elon is a poo poo head. No, seriously, I explain it at the end of the video how so constantly the counterarguments have no rational base and they constantly boil down to a dislike of the guy. Ironically, continuing to use Twitter to have a rant about stuff just shows that Twitter is just the same as it always was 🤣
So I wrapped up this week's live stream then promptly blew hours mucking around with Zigbee on Home Assistant. Is it worth it, as someone asked in the chat? Uh, yeah, kinda, mostly. But seriously, having a highly automated house is awesome and I suggest that most people watching these vids harbour the same basic instinct as I do to try and improve our lives through technology. The coordination of lights with times of day, the security checks around open doors, the controlling of fans and air conditioning to keep everyone comfy, it just rocks... when it works 😎
Yes, the Lenovo is Chinese. No, I'm not worried about Superfish. Yes, I'm running windows. No, I don't want a Framework laptop. Seemed to be a lot of time this week gone on talking all things laptops, and there are clearly some very differing views on the topic. Some good suggestions, some neat alternatives and some ideas that, well, just seem a little crazy. But hey, I'm super happy with the machine, it's an absolute beast and I expect I'll get many years of hard work out of it. That and more in this week's video, enjoy 😊
Edit (1 day later): After posting this, the party responsible for leaking the data turned around and said "that was only a small part of it, here's the whole thing", and released records encompassing a further 14M records. I've added those into HIBP and will shortly be re-sending notifications to people monitoring domains as the count of impacted addresses will likely have changed. Everything else about the subsequent dataset is consistent with what you'll read below in terms of structure, patterns and conclusions.
The same threat actor has leaked larger amounts of data from LinkedIn dated 2023. They claim this new data contains 35M lines and is 12 GB uncompressed. They also issue an apology to @troyhunt. #Breach #Clearnet #DarkWeb #DarkWebInformer #Database #Leaks #Leaked #LinkedIn https://t.co/qBFAofvppU pic.twitter.com/Clg5o92b6t
— Dark Web Informer (@DarkWebInformer) November 7, 2023
I like to think of investigating data breaches as a sort of scientific search for truth. You start out with a theory (a set of data coming from an alleged source), but you don't have a vested interested in whether the claim is true or not, rather you follow the evidence and see where it leads. Verification that supports the alleged source is usually quite straightforward, but disproving a claim can be a rather time consuming exercise, especially when a dataset contains fragments of truth mixed in with data that is anything but. Which is what we have here today.
To lead with the conclusion and save you reading all the details if you're not inclined, the dataset so many people flagged me this week titled "Linkedin Database 2023 2.5 Millions" turned out to be a combination of publicly available LinkedIn profile data and 5.8M email addresses mostly fabricated from a combination of first and last name. It all began with this tweet:
A threat actor has allegedly leaked a database from LinkedIn @LinkedIn dated 2023. They claim the database shows emails, profile data, phones, full names, and more confidential info. #Breach #Clearnet #DarkWeb #DarkWebInformer #Database #Leaks #Leaked #LinkedIn pic.twitter.com/8MQecKc1vz
— Dark Web Informer (@DarkWebInformer) November 4, 2023
All good lies are believable at face value; is it feasible a massive corpus of LinkedIn data is floating around? Well, they were proper breached in 2012 to the tune of 164M records (by which I mean that incident was genuinely internal data such as email addresses and passwords extracted out by a vulnerability), then they were massively scraped in 2021 with another 126M records going into Have I Been Pwned (HIBP). So, when you see a claim like the one above, it seems highly feasible at face value which is what many people take it at. But I'm a bit more suspicious than most people 🙂
First, the claim:
This one is similar to my twitter data scrapped [sic] but for linkedin plus 2023
Now, there's a whole debate about whether scraped data is breached data and indeed whether the definition of it even matters. With the rising prevalence of scraped data, this topic came up enough that I wrote a dedicated blog post about it a couple of years ago and concluded the following in terms of how we should define the term "breach":
A data breach occurs when information is obtained by an unauthorised party in a fashion in which it was not intended to be made available
Which makes scrapes like this alleged one a breach. If indeed it was accurate, LinkedIn data had been taken and redistributed in a way it was never intended to be by either the service itself or the individuals whose data was in this corpus. So, it's something to take seriously, and that warranted further investigation.
I scrolled through the 10M+ rows of data (many records spanned multiple rows due to line returns), and my eyes fell on a fellow Aussie who for the purposes of this exercise we'll call "EM", being the initials of her first and last name. Whilst the data I'm going to refer to is either public by design or fabricated, I don't want to use a real person as an example without their consent so let's just play it safe. Here's a fragment of EM's record:
There are 5 noteworthy parts of this I that immediately caught my attention:
On its own, this record would be unremarkable. It'd be entirely feasible - this could very well be legit - except when you keep looking through the remainder of the data. A pattern quickly emerged and I'm going to bold it here because it's the smoking gun that ultimately indicates that a bunch of this data is fake:
Every single record with multiple email addresses had exactly the same alias on completely unrelated domains and it was almost always in the form of "[first name].[last name]@".
Representing email addresses in this fashion is certainly common, but it's far from ubiquitous, and that's easy to demonstrate. For example, I have tons of emails from Pluralsight so I dig one out from my friend "CU":
There's no dot, rather a dash. Every single real Pluralsight email address I looked at was a dash rather than a dot, yet when I delved into the alleged LinkedIn data and dig out another sample Pluralsight address, here's what I found:
That's not LM's real address because it has a dot instead of a dash. Every. Single. One. Is. Fake.
Let's try this the other way around and load up the existing breached accounts in HIBP for the domain of one of EM's alleged email addresses and see how they're formed:
That's definitely not the same format as EM's address, not by a long shot. And time and time again, the same pattern of addresses in the corpus of data in the original tweet emerged, drawing me to what seems to be a pretty logical conclusion:
Each email address was fabricated by taking the actual domain of a company the individual legitimately worked at and then constructing the alias from their name.
And these are legitimate companies too because every single LinkedIn profile I checked had all the cues of accurate information and each domain I checked in the corpus of data was indeed the correct one for the company they worked at. I imagine someone has effectively worked through the following logic:
On that final point, what is the point? The data wasn't being sold in that original tweet, rather it was freely downloadable. But per the date on EM's profile, the data could have been obtained much earlier and previously monetised. And on that, the date wasn't constant across records, rather there was a broad range of them as recent as July last year and as old as... well, I stopped when the records got older than me. What is this?!
I suspect the answer may partly lie in the column headings which I've pasted here in their entirety:
"PROFILE_KEY", "PROFILE_USERNAMES", "PROFILE_SPENDESK_IDS", "PROFILE_LINKEDIN_PUBLIC_IDENTIFIER", "PROFILE_LINKEDIN_ID", "PROFILE_SALES_NAVIGATOR_ID", "PROFILE_LINKEDIN_MEMBER_ID", "PROFILE_SALESFORCE_IDS", "PROFILE_AUTOPILOT_IDS", "PROFILE_PIPL_IDS", "PROFILE_HUBSPOT_IDS", "PROFILE_HAS_LINKEDIN_SOURCE", "PROFILE_HAS_SALES_NAVIGATOR_SOURCE", "PROFILE_HAS_SALESFORCE_SOURCE", "PROFILE_HAS_SPENDESK_SOURCE", "PROFILE_HAS_ASGARD_SOURCE", "PROFILE_HAS_AUTOPILOT_SOURCE", "PROFILE_HAS_PIPL_SOURCE", "PROFILE_HAS_HUBSPOT_SOURCE", "PROFILE_FETCHED_AT", "PROFILE_LINKEDIN_FETCHED_AT", "PROFILE_SALES_NAVIGATOR_FETCHED_AT", "PROFILE_SALESFORCE_FETCHED_AT", "PROFILE_SPENDESK_FETCHED_AT", "PROFILE_ASGARD_FETCHED_AT", "PROFILE_AUTOPILOT_FETCHED_AT", "PROFILE_PIPL_FETCHED_AT", "PROFILE_HUBSPOT_FETCHED_AT", "PROFILE_LINKEDIN_IS_NOT_FOUND", "PROFILE_SALES_NAVIGATOR_IS_NOT_FOUND", "PROFILE_EMAILS", "PROFILE_PERSONAL_EMAILS", "PROFILE_PHONES", "PROFILE_FIRST_NAME", "PROFILE_LAST_NAME", "PROFILE_TEAM", "PROFILE_HIERARCHY", "PROFILE_PERSONA", "PROFILE_GENDER", "PROFILE_COUNTRY_CODE", "PROFILE_SUMMARY", "PROFILE_INDUSTRY_NAME", "PROFILE_BIRTH_YEAR", "PROFILE_MARVIN_SEARCHES", "PROFILE_POSITION_STARTED_AT", "PROFILE_POSITION_TITLE", "PROFILE_POSITION_LOCATION", "PROFILE_POSITION_DESCRIPTION", "PROFILE_COMPANY_NAME", "PROFILE_COMPANY_LINKEDIN_ID", "PROFILE_COMPANY_LINKEDIN_UNIVERSAL_NAME", "PROFILE_COMPANY_SALESFORCE_ID", "PROFILE_COMPANY_SPENDESK_ID", "PROFILE_COMPANY_HUBSPOT_ID", "PROFILE_SKILLS", "PROFILE_LANGUAGES", "PROFILE_SCHOOLS", "PROFILE_EXTERNAL_SEARCHES", "PROFILE_LINKEDIN_HEADLINE", "PROFILE_LINKEDIN_LOCATION", "PROFILE_SALESFORCE_CREATED_AT", "PROFILE_SALESFORCE_STATUS", "PROFILE_SALESFORCE_LAST_ACTIVITY_AT", "PROFILE_SALESFORCE_OWNER_CONTACT_ID", "PROFILE_SALESFORCE_OWNER_CONTACT_NAME", "PROFILE_SPENDESK_SIGNUP_AT", "PROFILE_SPENDESK_DELETED_AT", "PROFILE_SPENDESK_ROLES", "PROFILE_SPENDESK_AVERAGE_NPS_SCORE", "PROFILE_SPENDESK_NPS_SCORES_COUNT", "PROFILE_SPENDESK_FIRST_NPS_SCORE", "PROFILE_SPENDESK_LAST_NPS_SCORE", "PROFILE_SPENDESK_LAST_NPS_SCORE_SENT_AT", "PROFILE_SPENDESK_PAYMENTS_COUNT", "PROFILE_SPENDESK_TOTAL_EUR_SPENT", "PROFILE_SPENDESK_ACTIVE_SUBSCRIPTIONS_COUNT", "PROFILE_SPENDESK_LAST_ACTIVITY_AT", "PROFILE_AUTOPILOT_MAIL_CLICKED_COUNT", "PROFILE_AUTOPILOT_LAST_MAIL_CLICKED_AT", "PROFILE_AUTOPILOT_MAIL_OPENED_COUNT", "PROFILE_AUTOPILOT_LAST_MAIL_OPENED_AT", "PROFILE_AUTOPILOT_MAIL_RECEIVED_COUNT", "PROFILE_AUTOPILOT_LAST_MAIL_RECEIVED_AT", "PROFILE_AUTOPILOT_MAIL_UNSUBSCRIBED_AT", "PROFILE_AUTOPILOT_MAIL_REPLIED_AT", "PROFILE_AUTOPILOT_LISTS", "PROFILE_AUTOPILOT_SEGMENTS", "PROFILE_HUBSPOT_CFO_CONNECT_SLACK_MEMBER_STATUS", "PROFILE_HUBSPOT_IS_CFO_CONNECT_MEETUPS_MEMBER", "PROFILE_HUBSPOT_CFO_CONNECT_AREAS_OF_EXPERTISE", "PROFILE_HUBSPOT_CORPORATE_FINANCE_EXPERIENCE_YEARS_RANGE"
Check out some of those names: LinkedIn is obviously there, but so is Salesforce and Spendesk and Hubspot, among others. This reads more like an aggregation of multiple sources than it does data solely scraped from LinkedIn. My hope is that in posting this someone might pop up and say "I recognise those column headings, they're from..." Who knows.
So, here's where that leaves us: this data is a combination of information sourced from public LinkedIn profiles, fabricated emails address and in part (anecdotally based on simply eyeballing the data this is a small part), the other sources in the column headings above. But the people are real, the companies are real, the domains are real and in many cases, the email addresses themselves are real. There are over 1.8k HIBP subscribers in the data set and this is folks that have double opted-in so they've successfully received an email to that address in the past. Further, when the data was loaded into HIBP there were nearly a million email addresses that were already in the system so evidently, they were addresses that had previously been in use. Which stands to reason because even if every address was constructed by an algorithm, the pattern is common enough that there'll be a bunch of hits.
Because the conclusion is that there's a significant component of legitimate data in this corpus, I've loaded it into HIBP. But because there are also a significant number of fabricated email addresses in there, I've flagged it as a spam list which means the addresses won't impact the scale of anyone's paid subscription if they're monitoring domains. And whilst I know some people will suggest it shouldn't go in at all, time and time again when I've polled the public about similar incidents the overwhelming majority of people have said "we want to know about it then we'll make up our own minds what action needs to be taken". And in this case, even if you find an email address on your domain that doesn't actually exist, that person who either currently works at your company or previously did has still had their personal data dumped in this corpus. That's something most people will still want to know.
Lastly, one of the main reasons I decided to invest hours into this today is that I loathe disinformation and I hate people using that to then make statements that are completely off base. I'm looking at my Twitter feed now and see people angry at LinkedIn for this, blaming an insider due to recent layoffs there, accusing them of mishandling our data and so on and so forth. No, not this time, the evidence has led us somewhere completely different.
Most of this week's video went on the scraped (and faked) LinkedIn data, but it's the ransomware discussion that keeps coming back to mind. Even just this morning, 2 days after recording this live stream, I ended up on nation TV talking about the DP World security incident and whilst we don't have any confirmation yet, it has all the hallmarks of another ransomware case. In advance of that interview, I was trawling through various ransomware Tor sites and the volume of big names appearing there is just staggering. It does get me thinking: how many other individuals and corporations alike are being exposed through these and are never told about it? I wonder...
Allegedly, Acuity had a data breach. That's the context that accompanied a massive trove of data that was sent to me 2 years ago now. I looked into it, tried to attribute and verify it then put it in the "too hard basket" and moved onto more pressing issues. It was only this week as I desperately tried to make some space to process yet more data that I realised why I was short on space in the first place:
Ah, yeah - Acuity - that big blue 437GB blob. What follows is the process I went through trying to work out what an earth this thing is, the confusion surrounding the data, the shady characters dealing with it and ultimately, how it's now searchable in Have I Been Pwned (HIBP), which may be what brought you to this blog post in the first place.
One of the first things I do after receiving a data breach is to literally just Google it: acuity data breach. Which immediately yielded this top result from June:
Ah, so Acuity is a healthcare company. But wait - here's the next result:
That's not about healthcare, that's Acuity Brands. How many companies called "Acuity" that have been breached are there?! Let's see what references I have in my email:
Another one 🤦♂️ That "breach" could be circumstantial, so we'll call it a "maybe", but it's yet another Acuity with a question mark next to it. So how many "Acuity" companies are out there in total?! Just in the course of investigating this data, I came across a total of 6 of them that as far as I can tell, are completely unrelated:
Ugh, great. We'll work through them and try to figure out where they fit into the picture in a moment, but first let's look at the actual data. We already know it's 437GB, but it's the breadth of column headings that's most stunning; here's all 414 of them:
Just by eyeballing these, it really doesn't feel like the sort of data that comes from a healthcare provider, a brands company or a scheduler. The other 3, however... Maybe.
Some more data points before going further:
On that final point, here's an example of what I'm talking about:
The last names are the same, as are the salutations. The physical addresses are spot on accurate in their structure as are the phone numbers; there are no spaces, no dashes and no other artifacts typical of millions of different humans entering data. This is clean - too clean.
The "datasource" field is another interesting data point with the top 10 values being:
Each of these entries appeared at least hundreds of thousands of times, if not millions. Does that mean that Netflix, for example, provided customer data to this list? Almost certainly no, but it does feel reminiscent of the Acxiom / Live Ramp misattribution post I wrote a year ago where I listed full counts of a similar column. One of the top values there was also "TAGGED.COM" (also all in uppercase), alongside several other values that also appeared in both sources.
Back to attribution and a post on a popular hacking forum jumps out:
Many things here line up, for example the column names that are very unique to this data source, including "estimatedincomecode", "del_point_check_digit" and "secondaryaddresspresent". The attribution is to the insurance company named "Acuity", but is that accurate? Insurance companies collect a lot of data as it's relevant to how they run their business, but that data is highly unlikely to include fields such as:
That's much more in the "data enrichment" space where a company sells a massive data set so that it can expand the profile data of the purchaser's existing customer base. It's a legitimate, honest, legal business model. It's also indistinguishable from this:
Hey, it's 437GB! And the column names line up! And it's called Acuity! Slightly different column count to mine (and similar but different to the hacker forum post), and slightly different email count, but the similarities remain striking. How I got to this resource is also interesting, having come by someone I was discussing the data with a couple of years ago:
The YouTube video is a walkthrough of a campaign management tool to send emails to customers. Could that indicate the data as coming from Acuity Ads (now Illumin)? No, not in and of itself, the walkthrough there isn't that dissimilar to other campaign tools I've used in the past. No matter how much I looked, I just couldn't find a solid lead back to Acuity Ads and anything even remotely related was merely circumstantial. It could be from them, but it could also be from many other places and the mere fact that a near identical corpus of data was sitting there on an outright spam site only makes the whole mystery that much deeper. There was just one more interesting data point in that email:
i myself am in that dataset and i've been getting 100x more phishing/scam calls, emails, and physical mail
Let me end this with a best guess: this feels like the same situation as the massive Master Deeds incident in South Africa in 2017. In that case, a legally operating data aggregator (I think you know how I feel about those by now...) sold personal information to a real estate business who then left it publicly exposed. I say it feels the same because it's just such a clean set of data and it's clearly very comprehensive in terms of the columns. It's exactly what I'd expect a data aggregator to prepare and sell to other businesses so they could identify which of their existing customers likes needlework.
In the past, publishing blog posts like this has helped identify an origin service and if that happens again here then I'll be sure to provide an update. For now, I've loaded it into HIBP and flagged it as a spam list which means it won't impact the size of anyone's domains and bump them into a different subscription level. If you do have any interesting insights on this data, please leave a comment below and with any luck, one of the Acuity entities out there will emerge as the source.
Note: just after loading the data, I ran the calcs on how many of the addresses were pre-existing in HIBP. This seems like a statistically significant number 😲
So, 100% (just under actually, but it rounded up). Working through a bunch of sample addresses, they appeared across all sorts of other existing spam lists and dodgy data aggregator breaches. Who knows which ones came first, just more data in the big swimming pool of breaches. https://t.co/Ux2rw6uaAk
— Troy Hunt (@troyhunt) November 15, 2023
Think about it like this: in 2015, we all lost our proverbial minds at the idea of the Kazakhstan government mandating the installation of root certificates on their citizens' devices. We were outraged at the premise of a government mandating the implementation of a model that could, at their bequest, allow them to intercept traffic without any transparency or accountability. The EFF said the following at the time:
If the country's ruling regime were to successfully implement this plan, it would be able to snoop on, impersonate, and alter the online communications of anyone within their borders—effectively performing a Man in the Middle attack on its entire population.
Now watch the video, listen to Scott and ask yourself how different the technical capacity he discusses is from the Kazakhstan situation. Not from a policy perspective or the intentions of the respective government bodies, but rather it terms of the capabilities and lack of transparency it results in. It's nuts. But hey, it's a good time to be in this industry!
For a weekly update with no real agenda, we sure did spend a lot of time talking about the ridiculous approach Harvey Norman took to dealing with heavy traffic on Black Friday. It was just... unfathomable. A bunch of people chimed into the tweet thread and suggested it may have been by design, but they certainly wouldn't have set out to achieve the sorts of headlines that adorned the news afterwards. Who knows, but it made for entertaining content this week 🙂
I'm irrationally excited about the new Prusa 3D printer on order, and I think that's mostly to do with planning for the NDC Oslo talk I plan to do with Elle, my 11-year old daughter. I'm all for getting the kids exposure not just to tech, but also to being able to talk to others about tech and involving them in conference talks since a young age has been a big part of that. But what I'm especially excited about is that this won't just be an "aw, isn't it cute seeing kids talk at a conference" kinda thing; she genuinely knows enough about this technology that together, we can make a talk that adults will learn something from. That's cool 😎
A decade ago to the day, I published a tweet launching what would surely become yet another pet project that scratched an itch, was kinda useful to a few people but other than that, would shortly fade away into the same obscurity as all the other ones I'd launched over the previous couple of decades:
It's alive! "Have I been pwned?" by @troyhunt is now up and running. Search for your account across multiple breaches http://t.co/U0QyHZxP6k
— Have I Been Pwned (@haveibeenpwned) December 4, 2013
And then, as they say, things kinda escalated quickly. The very next day I published a blog post about how I made it so fast to search through 154M records and thus began a now 185-post epic where I began detailing the minutiae of how I built this thing, the decisions I made about how to run it and commentary on all sorts of different breaches. And now, a 10th birthday blog post about what really sticks out a decade later. And that's precisely what this 185th blog post tagging HIBP is - the noteworthy things of the years past, including a few things I've never discussed publicly before.
You know why it's called "Have I Been Pwned"? Try coming up with almost any conceivable normal sounding English name and getting a .com domain for it. Good luck! That was certainly part of it, but another part of the name choice was simply that I honestly didn't expect this thing to go anywhere. It's like I said in the intro of this post where I fully expected this to be another failed project, so why does the name matter?
But it's weird how "pwned" has stuck and increasingly, become synonymous with HIBP. For many people, the first time they ever hear the word is in the context of "Have I Been..." with an ensuing discussion often explaining the origins of the term as it relates to gaming culture. And if you do go and look for a definition of the term online, you'll come across resources such as How “PWNED” went from hacker slang to the internet’s favourite taunt:
Then in 2013, when various web services and sites saw an uptick in personal data breaches, security expert Troy Hunt created the website “Have I Been Pwned?” Anyone can type in an email address into the site to check if their personal data has been compromised in a security breach.
And somehow, this little project is now referenced in the definition of the name it emerged from. Weird.
But, because it's such an odd name that has so frequently been mispronounced or mistyped, I've ended up with a whole raft of bizarre domain names including haveibeenpaened.com, haveibeenpwnded.com, haveibeenporned.com and my personal favourite, haveibeenprawned.com (because a journo literally pronounced it that way in a major news segment 🤦♂️). Not to mention all the other weird variations including haveibeenburned.com, haveigotpwned.com, haveibeenrekt.com and after someone made the suggestion following the revelation that PornHub follows me, haveibeenfucked.com 🤷♂️
It's difficult to even know where to start here. How does the little site with the weird name end up in the press? Inevitably, "because data breaches", and it's nuts just how much exposure this project has had because of them. These are often mainstream news events and what reporters often want to impart to people is along the lines of "Here's what you should do if you've been impacted", which often boils down to checking HIBP.
Press is great for raising awareness of the project, but it has also quite literally DDoS'd the service with the Martin Lewis Money Show in the UK knocking it offline in 2016. Cool! No, for real, I learned some really valuable lessons from that experience which, of course, I shared in a blog post. And then ensured could never happen again.
Back in 2018, Gizmodo reckoned HIBP was one of the top 100 websites that shaped the internet as we knew it, alongside the likes of Wikipedia, Google, Amazon and Goatse (don't Google it). Only the year after it launched, TIME magazine reckon'd it was one of the 50 best websites of the year. And every time I do a Google search for a major news outlet, I find this little website. The Wall Street Journal. The Standard (nice headline!) USA Today. Toronto Star. De Telegraaf. VG. Le Monde. Corriere della Sera. It's wild - I just kept Googling for the largest newspapers in various parts of the world and kept getting hits!
The point is that it's had impact, and nobody is more surprised about that than me.
How on earth did I end up here?!
6 years and a few days ago now, I found myself in a place I'd only ever seen before in the movies: Congress. American Congress. Saying "pwned"!
For reasons I still struggle to completely grasp, the folks there thought it would be a good idea if I flew to the other side of the world and talked about the impact of data breaches on identity verification. "You know they're just trying to get you to DC so they can arrest you for all that stolen data you have, right?! 🤣", the internet quipped. But instead, I had one of the most memorable moments of my career as I read my testimony (these are public hearings so it's all recorded and available to watch), responded to questions from congressmen and congresswomen and rounded out the trip staring down at where they inaugurate presidents:
Today, that photo adorns the wall outside my office and dozens of times a day I look at it and ask the same question - how did it all lead to this?!
The potential sale of HIBP was a very painful, very expensive chapter of life, announced in a blog post from June 2019. For the most part, I was as transparent and honest as I could be about the reasons behind the decision, including the stress:
To be completely honest, it's been an enormously stressful year dealing with it all.
More than one year later, I finally wrote about the source of so much of that stress: divorce. Relationship circumstances had put a huge amount of pressure on me and I needed a relief valve which at the time, I thought would be the sale of the project I loved so much but was becoming increasingly demanding. Ultimately, Project Svalbard (the code name for the sale of HIBP), had the opposite effect as years of bitter legal battles with my ex ensued, in part due to the perceived value that would have been realised had it been sold and some big tech company owned my arse for years to come. The project I built out of a passion to do community good was now being used as a tool to extract as much money out of me as possible. There's a wild story to be told there one day but whilst that saga is now well and truly behind me, the scars are still raw.
There were many times throughout Project Svalbard where I felt like I was living out an episode of Silicon Valley, especially as I hopped between interviews at the who's-who of tech firms in San Francisco to meet potential acquirers. But there was one moment in particular that I knew at the time would form an indelible memory, so I took a photo of it:
I'm sitting in a rental car in Yosemite whilst driving from the aforementioned meetings in SF and onto Vegas for the annual big cyber-events. I had a scheduled call with a big tech firm who was a potential acquirer and should that deal go through, the guy I was speaking to would be my new boss. I'd done that dozens of times by now and I don't know if it was because I was especially tired or emotional or if there was something in the way he phrased the question, but this triggered something deep inside me:
So Troy, what would your perfect day in the office look like?
I didn't say it this directly, but I kid you not this is exactly what popped into my mind:
I get on my jet ski and I do whatever the fuck I want
My potential new overlord had somehow managed to find exactly the raw nerve to touch that made me realise how valuable independence had become to me. 6 months later, Project Svalbard was dead after a deal I'd struck fell through. I still can't talk about the precise circumstances due to being NDA'd up to wazoo, but the term we chose to use was "a change of business circumstances on behalf of the purchaser". With the benefit of hindsight, I've never been so happy to have lost so much 😊
10 years ago, I certainly didn't see this on the cards:
This is so cool, thanks @FBI 😊 pic.twitter.com/aqMi3as91O
— Troy Hunt (@troyhunt) June 28, 2023
Nor did I expect them to be actively feeding data into HIBP. Or the UK's NCA to be feeding data in. Or various other law enforcement agencies the world over. And I never envisioned a time where dozens of national governments would be happy to talk about using the service.
A couple of months ago, the ABC wrote a long piece on how this whole thing is, to use their term, a strange sign of the times.
He’s just “a dude on the web”, but Troy Hunt has ended up playing an oddly central role in global cybersecurity.
It's strange until you look at through the lens of aligned objectives: the whole idea of HIBP was "to do good things after bad things happen" which is well aligned with the mandates of law enforcement agencies. You could call it... common ground:
This is something I suspect a lot of people don't understand - that law enforcement agencies often work in conjunction with private enterprise to further their goals of protecting people just like you and me. It's something I certainly didn't understand 10 years ago, and I still remember the initial surprise when agencies started reaching out. Many years on, these have become really productive relationships with a bunch of top notch people, a number of whom I now count as friends and make an effort to spend time with on my travels.
This was never on the cards originally. In fact, I'd always been adamant that there should never be passwords in HIBP although in my defence, the sentiment was that they should never appear next to the username to which they originally accompanied. But looking at passwords through the lens of how breach data can be used to do good things, a list of known compromised passwords disassociated from any form of PII made a lot of sense. So, in 2017, Pwned Passwords was born. You know what I was saying earlier about things escalating quickly? Yeah:
Setting all new records for Pwned Passwords this week: biggest day ever yesterday at 282M requests and biggest rolling 30 days ever, now passing the 6 *billion* requests mark! pic.twitter.com/dQiuQim3da
— Troy Hunt (@troyhunt) September 12, 2023
As if to make the point, I just checked the latest stats and last week we did 301.6M requests in a single day. 100% of those requests - and that's not a rounded number either, it's 100.0000000000% - were served from Cloudflare's cache 🤯
There's so much I love about this service. I love that it's free, there's no auth, it's entirely open source (both code and data), the FBI feeds data into it and perhaps most importantly, it has real impact on security. It's such a simple thing, but every time you see a headline such as "Big online website hit with credential stuffing attack", a significant portion of the accounts being taken over have passwords that could easily have been blocked.
On multiple occasions now, I've had conversations that can best be paraphrased as follows:
Random Internet Person: I'm going to report you to the FBI for having all that stolen data
Me: Maybe you should start by Googling "troy hunt fbi" first...
But I understand where they're coming from and the paradox I refer to is the perceived conflict between handling what is usually the output of a crime whilst simultaneously trying to perform a community good. It's the same discussion I've often had with people citing privacy laws in their corner of the world (often the EU and GDPR) as the reason why HIBP shouldn't exist: "but you're processing data without informed consent!", they'll claim. The issue of there being other legal bases for processing aside, nobody consents to being in a data breach! The natural progression of that conversation is that being in a data breach is a parallel discussion to HIBP then indexing it and making it searchable, which is something I've devoted many words to addressing in the past.
But for all the bluster the occasional random internet person can have (and honestly, I could count the number of annual instances of this on one hand), nothing has come of any complaints. And when I say "complaints", it's often nothing more than a polite conversation which may simply conclude with an acknowledgment of opposing views and that's it. There has been one exception in the entire decade of running this service where a complaint did come via a government privacy regulator, I responded to all the questions that were asked and that was the end of it.
When you have a pet project like HIBP was in the beginning, it's usually just you putting in the hours. That's fine, it's a hobby and you're scratching an itch, so what does it matter that there's nobody else involved? Like many similar passion projects, HIBP consumed a lot of hours from early on, everything from obviously building the service then sourcing data breaches, verifying and disclosing them, writing up descriptions and even editing every single one of those 700+ logos by hand to be just the right dimensions and file size. But in the beginning, if I'd just stopped one day, what would happen? Nothing. But today, a genuinely important part of the internet that a huge number of individuals, corporations and governments have built dependencies on would stop working if I lost interest.
The dependency on just me was partly behind the possible sale in 2019, but clearly that didn't eventuate. There was always the option to employ people and build it out like most people would a normal company, but every time I gave that consideration it just didn't stack up for a whole bunch of reasons. It was certainly feasible from the perspective of building some sort of valuable commercial entity, but in just the same way as that question about my perfect day in the office sucked the soul from my body, so did the prospect of being responsible for other people. Employment contracts. Salary negotiations. Performance reviews. Sick leave and annual leave and all sorts of other people issues from strangers I'd need to entrust with "my baby". So, bringing in more people was a really unattractive idea, with 2 exceptions:
In early 2021, my (soon to be at the time) wife Charlotte started working for HIBP.
Charlotte had spent the last 8 years working with people just like me; software nerds. As a project manager for the NDC conferences based out of Norway, she'd dealt with hundreds of speakers (including me on many occasions), and thousands of attendees at the best conference I've ever been a part of. Plus, she spent a great deal of time coordinating sponsors, corporate attendees and all sorts of other folks that live in the tech world HIBP inhabited. For Charlotte, even though she's not a technical person (her qualifications are in PR and entrepreneurial studies), this was very familiar territory.
So, for the last few years, Charlotte has done absolutely everything that she can to ensure that I can focus on the things that need my attention. She onboards new corporate subscribers, handles masses of tickets for API and domain subscribers and does all the accounting and tax work. And she does this tirelessly every single day at all sorts of hours whether we're at home or travelling. She is... amazing 🤩
Earlier this year, Stefán Jökull Sigurðarson started working for us part time writing code, cleaning up code, migrating code and, well, doing lots of different code things.
Just today I asked Stefán what I should write about him, thinking he'd give me some bullet points I'd massage and then incorporate into this blog post. Instead, I reckon what he wrote was so spot on that I'm just going to quote the entire thing here:
"Just" that having had my eye on the service since it was released and then developing one of the first big integrations with the PwnedPasswords v2 API in EVE, coinciding with us meeting for the first time at NDC Oslo in 2018 shortly after, HIBP has managed to take me on this awesome journey where it has been a part of launching my public speaking career, contributing to OSS with Pwned Passwords, becoming an MVP and helped me meet a bunch of awesome people and allowed me to contribute to a better and hopefully safer internet. I'm very happy and honoured to a be a part of this project which is full of awesome challenges and interesting problems to deal with. Having meeting invites from the FBI in my inbox a few years after doing a few experimental rest calls to the Pwned Passwords API in early 2018 was definitely not something I was expecting 😅
What really resonated with me in Stefán's message is that for him, this isn't just a job, it's a passion. His journey is my journey in that we freely devoted our time to do something we love and it led to many wonderful things, including MVP roles and speaking at "Charlotte's" conference, NDC. Stefán is based in Iceland, but we've still had many opportunities to share beers together and establish a relationship that transcends merely writing code. I can't think of anyone better to do what he does today.
731 breaches later, here we are. So, what stands out? Just going off the top of my head here:
Ashley Madison. Every knows the name so it needs no introduction, but that incident in 2015 had a major impact on HIBP in terms of use of the service, and also a major impact on me in terms of the engagements I had with impacted parties. My blog post on Here’s what Ashley Madison members have told me still feels harrowing to read.
Collection #1. This is the one that really contributed to my stress levels in early 2019 and had a profound impact on my decision to look at selling the service. Read about where those 773M records came from (still the largest breach in HIBP to date).
Rosebutt. Don't make a joke about it, don't make a joke about it, don't... aw man, thanks The Register! (link to an archive.org version as they seem to have thought better of their image choice later on...) The point is that even serious data breaches can have their moments of levity.
Shit Express. Sometimes, you just need a bit of hilarity in your data breach. Shit Express is literally a site to send other people pieces of that - anonymously - and they got breached, thus somewhat affecting their anonymity. The more serious point is that as I later wrote, claims of anonymity are often highly misleading.
I often joke about my life being very much about getting up each morning, reading my emails and events from overnight and then just winging it from there. Of course there are the occasional scheduled things not to mention travel commitments, but for the most part it's very much just rolling with whatever is demanding attention on the day. This is also probably a significant part of why I don't really want to see this thing grow into a larger concern with more responsibilities, I just don't want to lose that freedom. Yet...
We're gradually moving in a direction where things become more formalised. 3 years ago, I did 100% of everything myself. 1 year ago, I did everything technical myself. 6 months ago, we had no ticketing system for support. But these are small, incremental steps forward and that's what I'd like to see continuing. I want HIBP to outlive me, I just don't want it to become a burden I'm beholden to in the process. I'd like to have more people involved but as you can see from above, that's been a very slow process with only those very close to me playing a role.
The only thing I have real certainty on at the moment is that there will be more breaches. I've commented many times recently that the scourge that is ransomware feels like it's really accelerated lately, I wonder how many of the people in the emails and documents and all sorts of other data that get dumped there ever learn of their exposure? It's a non-trivial exercise to index that (for all sorts of reasons), but it also seems like an increasingly worthy exercise. Who knows, let's see how I feel when I get up tomorrow morning 🙂
Finally, for this week's regular video, I'm going to make a birthday special and do it live with Charlotte. Please come and join us, I'm not entirely sure what we'll cover (I'll work it out on the morning!) but let's make a virtual 10th birthday party out of it 🎂
10 years later... 🤯 Seriously, how did this thing turn into this?! It was the humblest of beginning with absolutely no expectations of anything, and now it's, well, massive! I'm a bit lost for words if I'm honest, I hope the chat with Charlotte adds some candour to this week's update, she's seen this thing grow since before its first birthday, through the hardest times and the best times and now lives and breathes HIBP day in day out with me. I hope you enjoy this video, and we'd both love to hear those swag ideas from you too 😊
I'd say the balloon fetish segment was the highlight of this week's video. No, seriously, it's a moment of levity in an otherwise often serious industry. It's still a bunch of personal info exposed publicly and that suchs regardless of the nature of the site, but let's be honest, the subject matter did make for some humorous comments 🤣
It's that time of the year again, time to head from the heat to the cold as we jump on the big plane(s) back to Europe. The next 4 weekly updates will all be from places of varying degrees colder than home, most of them done with Scott Helme too so they'll be a little different to usual. For now, here's a pretty casual Christmas edition, see you next week from the other side 🙂
We're in Paris! And feeling proper relaxed after several days of wine and cheese too, I might add. This was a very impromptu end of 2023 weekly update as we balanced family time with doing the final video for the year. On the cyber side, the constant theme over the last week has been ransomware; big firms, little firms, Aussie firms, American firms - it's just completely indiscriminate. Anecdotally, this seems to have really ramped up over 2023 so on that basis, 2024 will bring... well, let's wait and see, this industry is nothing if not full of surprises. Happy New Year friends 😊
