Geez we've been getting hammered down here: Optus, MyDeal, Vinomofo, Medibank and now Australian Clinical Labs. It's crazy how much press interest there's been down here and whilst I think some of it is a bit hyperbolic, bringing the issue to the forefront and ensuring it's being discussed is certainly a good thing. Anyway, let's see what happens between now and next week's video, at this rate there'll be at least one more major Aussie breach to talk about!
Just over 3 years ago now, I sat down at a makeshift desk (ok, so it was a kitchen table) in an Airbnb in Olso and built the authenticated API for Have I Been Pwned (HIBP). As I explained at the time, the primary goal was to combat abuse of the service and by adding the need to supply a credit card, my theory was that the bad guys would be very reluctant to, well, be bad guys. The theory checked out, and now with the benefit of several years of data, I can confidently say abuse is near non-existent. I just don't see it. Which is awesome π
But there were other things I also didn't see, and it's taken a while for me to get around to addressing them. Some of them are fixed now (like right now, already in production), and some of them will be fixed very, very soon. I think it's all pretty cool, let me explain:
A little more background will help me explain this better: in the opening sentence of this blog post I mentioned building the original authenticated API out on a kitchen table at an Airbnb in Oslo. By that time, everyone knew I was going through an M&A process with HIBP I called Project Svalbard, which ultimately failed. What most people didn't know at the time was the other very stressful goings on in my life which combined, had me on a crazy rollercoaster ride I had little control over. It was in that environment that I created the authenticated API, complete with the Azure API Management (APIM) component and Stripe integration. It was rough, and I wish I'd done it better. Now, I have.
In the beginning, I pushed as much of the payment processing as possible to the HIBP website. This was due to a combination of me wanting to create a slick UX and frankly, not understanding Stripe's own UI paradigms. It looked like this:
Cards never ended up hitting HIBP directly, rather the site did a dance with Stripe that involved the card data going to them directly from the client side, a token coming back and then that being used for the processing. It worked, but it had numerous problems ranging from lack of support for things like 3D Secure payments, no support for other payments mechanisms such as Google Pay and Apple Pay and increasingly, large amounts of plumbing required to tie it all together. For example, there were hundreds of lines of code on my end to process payments, change the default card and show a list of previous receipts. The Stripe APIs are extraordinarily clever, but I couldn't escape writing large troves of my own code to make it work the way I originally designed it.
Two new things from Stripe since I originally wrote the code have opened up a whole new way of doing this:
Rolling to these services removed a huge amount of code from HIBP with the bulk of what's left being email address verification, API key management and handling callbacks from Stripe when a payment is successful. What all this means is that when you first create a subscription, after verifying your email address, you see these two screens:
That's the embeddable pricing table following by Stripe's own hosted payment page. I left the browser address bar in the latter to highlight that this is served by Stripe rather than HIBP. I love distancing myself from any sort of card processing and what's more, everything to do with actually taking the payment is now Stripe's problem π If you're interested in the mechanics of this, a successful payment calls a webhook on HIBP with the customer's details which updates their account with a month of API key whilst the screen above redirects them over to the HIBP website where they can grab their key. Easy peasy.
I silently rolled this out a week ago, watched it being used, made a few little tweaks and then waited until now to write about it. The rollout coincided with a typical email I've received so many times before:
First of all I would like to thank you for the wonderful service that helps people to keep track of their email breaches. I was trying to build a product to provide your services via my website, something similar to Firefox, avast and 100's of other companies doing. We were trying to do it according to the guidelines mentioned in the website. However I am not able to renew my purchase due to payment gateway failures at stripe payment. Requesting you to kindly check the same and advise me on alternate methods for making the payment.
The old model often caused payments to be rejected, especially from subscribers in India. The painful thing for me when trying to help folks is that Stripe would simply report the failed payment as follows:
However, going back to the individual who raised the query above after rolling out this update, things changed very dramatically:
To the title of this section, I simply wasn't "Striping" right. I'm sure there's a way with enough plumbing that it's feasible, but why bother? I cut hundreds of lines of code out just by delegating more of the workload back to them. Further, with ever tightening PCI DSS standards (read Scott's piece, interesting stuff) the less I have to do with cards, the better.
This was a "penny drop" moment for me and it's already made a big difference in a positive way. But there's another penny that dropped for me at the same time: one-off keys were an unnecessary problem.
It was at the moment I was ripping out those hundreds of lines of code that I wondered: why do I have all the additional kludge to support the paradigm of a one-off key that only lasts a month? Why had I built (and was now maintaining) server side code to handle different types of purchases and UX paradigms to represent one-off versus recurring keys? My gut feel was that most payments formed part of an ongoing subscription but hey, who needs gut feels when you have real data?! So I pulled the numbers:
Only 7% of payments were one-offs, with 93% of payments forming part of ongoing subscriptions.
And so I killed the one-off keys. Kinda, because you can still have a key for only one month, you just purchase a monthly subscription then immediately cancel it via the Stripe Customer Portal:
That's linked into from the API key dashboard on HIBP and it'll take all of 5 seconds to do (also note the ability to change payment method directly on the Stripe site). I've added text to that effect on the HIBP website (you may have spotted that in the earlier screen cap) so in practice, the ability to purchase a one-off key is still there and the main upside of this is that I've just killed a trove of code I no longer have to worry about π Because this is the internet, I'm sure someone will still be upset, but if you only want a key for a month then that capability still well and truly exists.
All of this so far amounts to doing the same things that were always there but better. Now let's talk about the all new stuff!
The title is self-explanatory and "very soon" is in about 2 weeks from now π
Let me illustrate the first part of that title with a message I received recently:
Is there a way to procure a 10 year API key? Our client wants to use the Have I been Pwned plugin for [redacted service name]; however, the $3.50 monthly subscription is too small to go through procurement.
What's that saying about no good deed going unpunished? In my naivety, I made the pricing low with the thinking that was a good thing, yet here we are with that posing barriers! This was a recurring message over and over again with folks simply struggling to get their $3.50 reimbursed. I should have seen this coming after years of living the corporate life myself (I have vivid flashbacks of how hard it was to get small sums reimbursed), and filling out an untold number of expense reports. Speaking of which, this was another recurring theme:
Is there a way to pay yearly for HIBP API access vs monthly? Β Monthly adds overhead in paperwork.
And again, I get it, this is a painful process. It somehow feels even more painful due to the fact the sum is so low; how much time are people burning trying to justify $3.50 to their boss?! It's painful, and this likely explains why the request for annual payments is the second most requested idea on HIBP's UserVoice. The comments there speak for themselves, and I'm having corporate PTSD flashbacks just reading them again now!
Sticking with the UserVoice theme, the 5th most requested feature is for different pricing on different rate limits. This is mostly self-explanatory but what I wasn't aware of until I went and pulled the stats was just how many people were hacking around the rate limit problem. There are heaps of API accounts like this:
hibp+1@domain.com
hibp+2@domain.com
hibp+3@domain.com
...
Because there can only be one key per email address, organisations are creating heaps of unique sub-addressed emails in order to buy multiple keys. This would have been a manual, laborious process; there's no automated way to do this, quite the contrary with anti-automation controls built into the process. Further, each key has it's own rate limit so I imagine they were also building a bunch of plumbing in the back end to then distribute requests across a collection of keys which, yeah, I get it, but man that seems like hard work! When I say "a collection of keys", I'm not just talking about a few of them either; the largest number of active in-use keys by a single organisation is 112. One hundred and twelve! The next largest is 110. I never expected that π€― (Incidentally, these orgs and the others obtaining multiple keys are all precisely the kinds I want using the API to do good things.)
Building the mechanics of annual billing and different rate limits is only part of the challenge and most of that is already done, the harder part is pricing it. I'm pulling troves of analytics from APIM at present to better understand the usage patterns, and it's quite interesting to see the data as it relates to requests for the API:
There's no persistent logging of the actual queries themselves, but APIM makes it easy to understand both the volume of queries and how many of them are successful versus failed, namely because they exceed the existing rate limit or were made with an invalid (likely expired) key. So, that's what I need to work out over the next couple of weeks when I'll launch everything and write it up, as always, in detail π
The HIBP API has become an increasingly important part of all sorts of different tools and systems that use the data to help protect people impacted by data breaches. The changes I've pushed out over the last week help make the service more accessible and easier to manage, but it's the coming changes I'm most excited about. These are the ones that will make life so much easier on so many people integrating the service and, I sincerely hope, will enable them to do things that make a much more profound impact on all of us who've been pwned before.
Go and check out how the whole API key process works, I'd love to hear your feedback π
Aussie breachapalooza! That what it feels like this week between Optus (ok, it was weeks ago but it's still in the news), Vinomofo, My Deal and the mother of all of them (at least as far as media interest goes), Medibank. That last one totally smashed my week out with unprecedented press enquiries, so is it any wonder I totally missed the Microsoft one? I read through that last one live in this week's video and as you'll hear, a breach of any kind is never a good look but what stands out for me about this one isn't the breach itself, rather the marketing effort SOCRadar has made around it. As I say in the video, it just feels... icky. See if you agree.
I decided to do something a bit different this week and mostly just answer questions from my talk at GOTO Copenhagen last week. I wasn't actually in Denmark this time, but a heap of really good questions came through and as I started reading them, I thought "this would actually make for a really good weekly update". So here we are, and those questions then spurned on a whole heap more from the live audience too so this week's video became one large Q&A. I hope you enjoy this one, let me know if I should do more of these in the future.
Geez it's nice to be home π It's nice to live in a home that makes you feel that way when returning from a place as beautiful as Bali π This week's video is dominated by the whole discussion around this tweet:
I love that part of the Microsoft Security Score for Identity in Azure improves your score if you *don't* enforce password rotation, what a sign of the times! Who out there still works somewhere that forces rotation (because "reasons")? pic.twitter.com/a2yQQvNRpa
β Troy Hunt (@troyhunt) October 6, 2022
I love this for the way it throws traditional logic out the window, logic we all knew sucked and I suspect the massive engagement the tweet drove is due to precisely that: Microsoft giving us all a good reason to whinge about a sucky practice that still prevails so broadly. So... I hope you enjoy listening to just how bad enforced password rotation sucks π
How's this weeks video for a view?! It's a stunning location here in Bali and it's just been the absolute most perfect spot for a honeymoon, especially after weeks of guests and celebrations. But whoever hacked and ransom'd Optus didn't care about me taking time out and I've done more media in the last week than I have in a long time. I don't mind, it's a fascinating story the way this has unfolded and that's where most of the time in this week's video has gone, I hope you enjoy my analysis of what has become a pretty crazy story back home in Australia.
Wow, what a week! Of course there's lots of cyber / tech stuff in this week's update, but it was really only the embedded tweet below on my mind so I'm going to leave you with this then come to you from somewhere much more exotic than usual (and I reckon that's a pretty high bar for me!) next week π
Absolutely over the moon to formally make @Charlotte_Hunt_ a part of our family β€οΈ π pic.twitter.com/XfahXElboC
β Troy Hunt (@troyhunt) September 21, 2022
I came so close to skipping this week's video. I'm surrounded by family, friends and my amazing wife to be in only a couple of days. But... this video has been my constant companion through very difficult times, and I'm happy to still being doing it at the best of times π So, with that, I'm signing out and heading off to do something much more important. See you next week.
Taking a bit of time off Twitter while @charlottelyng and I do more important things π π°ββοΈ pic.twitter.com/9JJrPM9kWX
β Troy Hunt (@troyhunt) September 13, 2022
I'm so excited to see the book finally out and awesome feedback coming in, but I'm disappointed with this week's video. I frankly wasn't in the right frame of mind to do it justice (it's been a very hard road up until this point, for various reasons), then my connection dropped out halfway through and I had to roll to 5G, and now I'm hearing (both from other people and with my own ears), a constant background noise being picked up by the mic. Argh! But, that's the reality of scheduled live streams and for better or worse, you end up getting the "warts and all" version. It is what it is, and next week's will be better π
The first time I ever wrote publicly about a company's security vulnerabilities, my boss came to have a word with me after seeing my name in the news headlines.
One of the worst days I've ever had was right in the middle of the Have I Been Pwned sale process, and it left me an absolute emotional wreck.
When I wrote about how I deal with online abuse, it was off the back of some pretty nasty stuff... which I've now included in this book π
These are the stories behind the stories and finally, the book about it all is here:
I announced the book back in April last year after Rob, Charlotte and I had already invested a heap of effort before releasing a preview in October. I'd hoped to have it out by Christmas... but it wasn't perfect. Ok, so it'll never be perfect without faults of any kind, but it had to meet an extremely high bar for me to be happy with an end result we could charge people money for. I completely rewrote the intro, changed a bunch of the posts we decided to include, reordered them all and edited a heap of the personal stuff. Rob wrote an amazing intro (I genuinely felt emotional reading it, given the time of both our lives he refers to), and both Charlotte and I wrote pieces at the end of it all. That bit in particular is very personal, and it's what was the most natural to us at the time. With our wedding being only next week, it just felt... right. That's why we're publishing today; to get this story about the earlier phases of my life out before the next phase begins. There are things we both wanted to say, and I hope you enjoying reading them π
We actually pushed the book out to a preview audience a couple of weeks ago and requested feedback to make it even better before releasing to the masses. We got two kinds of feedback: typos, which we've now fixed and testimonials, which have been awesome:
Great read! Troy has the technical know-how, and is able to effectively communicate complex topics so that anyone understands them. The personal stories kept me hooked!
- Mikko HyppΓΆnnen
I haven't been able to put the book down. The added intros and epilogue on each post in particular and the retrospectives from today's perspective are particularly interesting... Captivating stuff, apart from infosec, you really feel as though youβve been taken on a journey with Troy through the years of living in paradise a.k.a. Gold Coast, craft beer, good coffeeβs, travel and jet skis. Top of my list for resources I point anyone new to the space to. Simply Outstanding read - 10/10.
- Henk Brink
Love, love, LOVE the intros - i'm only familiar with Lars in terms of his online identity and videos, but the "commendations" from Richard Campbell, and in particular Rob Conery are fantastic and really anchor an emotional aspect to the book, that i was not expecting. Great to see a book deliver this authenticity - we're all only human after all!. I "cheated" and also skipped to read Charlotte's epilogue and again was blown away by the depth and genuine nature of the emotion on display. I honestly was not expecting there to be so much heart on display, but am very glad there is.
- C. Morgan
A famous American newsman used to call this "The rest of the story." This is the kind of insight that only comes with time, as the author can reflect and pick out important details that didn't get the coverage they deserved and these then find a life of their own. This book provides "the rest of the story" behind other stories which broke months or years ago. It gives these stories new life, and new significance.
- Pat Phelan
PWNED! Troy Hunt takes us on his life journey, ups and downs, explaining how haveIbeenpwned came to be, raising awareness of the worldβs poor password and online security habits... This book has it all. Plenty of tech, data breaches, career hacks, IoT, Cloud, password management, application security, and more, delivered in a fun way. This info is gold, and has improved and complemented my career in IT, and also my digital life at home too. I highly recommend this book as it explains cyber security so well, and hope you have your eureka moment and improve your cyber hygiene too.
- George Ousak
I never imagined turning blog posts into a book, but here we are, and it's come out awesome π This book is a labour of love the three of us have poured ourselves into over the last couple of years. It's a personal story I'm publishing at a very personal time and it's now live at book.troyhunt.com
Well, after a crazy amount of work, a lot of edits, reflection, and feedback cycles, "Pwned" is almost here:
This better be a sizzling read @troyhunt or I'll be crashing the wedding in ways never done before.
β Mike Thompson (@AppSecBloke) August 30, 2022
Also, I thought they'd cancelled Neighbours? πβ€οΈ pic.twitter.com/jrYIKtL0Uh
The preview cycle is in full swing with lots of feedback coming in and revisions being made before we push it live to the masses. This is really exciting and I can't wait to get the book out there in front of everyone, stay tuned π
By all accounts, this was one of the best weekly updates ever courtesy of a spam caller giving me a buzz at the 38:40 mark and struggling with "pwn" versus "porn". It resulted in an entertaining little on-air call and subsequently caused me to go out and register both haveibeeninpwn.com and haveibeeninporn.com. I figure these will result in much ongoing hilarity the next time I get a call of this nature about one of those domains π€£ Oh - and there's a whole bunch of data breach stuff this week, enjoy!
Right off the back of a visit to our wedding venue (4 weeks and counting!) and a few hours before heading to the snow (yes, Australia has snow), I managed to slip in a weekly update earlier today. I've gotta say, the section on Shitexpress is my favourite because there's just so much to give with this one; a service that literally ships shit with a public promise of multiple kinds of animal shit whilst data that proves only horse shit was ever shipped, a promise of 100% anonymity whilst the data set clearly shows both shit-senders and shit-receivers and possibly the most eye-opening of all, the messages accompanying the shit. So, uh, yeah, enjoy! π©
It was all a bit last minute today after travel, office works and then a quick rebuild of desk and PC before doing this livestream (didn't even have time to comb my hair!) So yes, I took a shortcut with the description of this video, but it all worked out well in the end IMHO with plenty of content that wasn't entirely data breach related, but yeah, that does seem to be a bit of a recurring theme in these vids. Enjoy π
A very early weekly update this time after an especially hectic week. The process with the couple of data breaches in particular was a real time sap and it shouldn't be this hard. Seriously, the amount of effort that goes into trying to get organisations to own their breach (or if they feel strongly enough about it, help attribute it to another party) is just nuts. It's not getting any better either π Regardless, listen to how these couple went and as always, if you've got any bright ideas about how to make this process less painful then I'd love to hear them.
How best to punish spammers? I give this topic a lot of thought because I spend a lot of time sifting through the endless rubbish they send me. And that's when it dawned on me: the punishment should fit the crime - robbing me of my time - which means that I, in turn, need to rob them of their time. With the smallest possible overhead on my time, of course. So, earlier this year I created Password Purgatory with the singular goal of putting spammers through the hellscape that is attempting to satisfy really nasty password complexity criteria. And I mean really nasty criteria, like much worse than you've ever seen before. I opened-sourced it, took a bunch of PRs, built out the API to present increasingly inane password complexity criteria then left it at that. Until now because finally, it's live, working and devilishly beautiful π
This is the easy bit - I didn't have to do anything for this step! But let me put it into context and give you a real world sample:
Ugh. Nasty stuff, off to hell for them it is, and it all begins with filing the spam into a special folder called "Send Spammer to Password Purgatory":
That's the extent of work involved on a spam-by-spam basis, but let's peel back the covers and look at what happens next.
Microsoft Power Automate (previously "Microsoft Flow") is a really neat way of triggering a series of actions based on an event, and there's a whole lot of connectors built in to make life super easy. Easy on us as the devs, that is, less easy on the spammers because here's what happens as soon as I file an email in the aforementioned folder:
Using the built in connector to my Microsoft 365 email account, the presence of a new email in that folder triggers a brand new instance of a flow. Following, I've added the "HTTP" connector which enables me to make an outbound request:
All this request does is makes a POST to an API on Password Purgatory called "create-hell". It passes an API key because I don't want just anyone making these requests as it will create data that will persist at Cloudflare. Speaking of which, let's look at what happens over there.
Let's start with some history: Back in the not too distant past, Cloudflare wasn't a host and instead would just reverse proxy requests through to origin services and do cool stuff with them along the way. This made adding HTTPS to any website easy (and free), added heaps of really neat WAF functionality and empowered us to do cool things with caching. But this was all in-transit coolness whilst the app logic, data and vast bulk of the codebase sat at that origin site. Cloudflare Workers started to change that and suddenly we had code on the edge running in hundreds of nodes around the world, nice and close to our visitors. Did that start to make Cloudflare a "host"? Hmm... but the data itself was still on the origin service (transient caching aside). Fast forward to now and there are multiple options to store data on Cloudflare's edges including their (presently beta) R2 service, Durable Objects, the (forthcoming) D1 SQL database and of most importance to this blogpost, Workers KV. Does this make them a host if you can now build entire apps within their environment? Maybe so, but let's skip the titles for now and focus on the code.
All the code I'm going to refer to here is open source and available in the public Password Purgatory Logger Github repo. Very early on in the index.js file that does all the work, you'll see a function called "createHell" which is called when the flow step above runs. That code creates a GUID then stores it in KV after which I can easily view it in the Cloudflare dashboard:
There's no value yet, just a key and it's returned via a JSON response in a property called "kvKey". To read that back in the flow, I need a "Parse JSON" step with a schema I generated from a sample:
At this point I now have a unique ID in persistent storage and it's available in the flow, which means it's time to send the spammer an email.
Because it would be rude not to respond, I'd like to send the spammer back an email and invite them to my very special registration form. To do this, I've grabbed the "Reply to email" connector and fed the kvKey through to a hyperlink:
It's an HTML email with the key hidden within the hyperlink tag so it doesn't look overtly weird. Using this connector means that when the email sends, it looks precisely like I've lovingly crafted it myself:
With the entire flow now executed, we can view the history of each step and see how the data moves between them:
Now, we play the waiting game π
Wasting spammer time in and of itself is good. Causing them pain by having them attempt to pass increasingly obtuse password complexity criteria is better. But the best thing - the piΓ¨ce de rΓ©sistance - is to log that pain and share it publicly for our collective entertainment π€£
So, by following the link the spammer ends up here (you're welcome to follow that link and have a play with it):
The kvKey is passed via the query string and the page invites the spammer to begin the process of becoming a partner. All they need to leave is an email address... and a password. That page then embeds 2 scripts from the Password Purgatory website, both of which you can find in the open source and public Github repository I created in the original blog post. Each attempt at creating an account sends off the password only to the original Password Purgatory API I created months ago, after which it responds with the next set of criteria. But each attempt also sends off both the criteria that was presented (none on the first go, then something increasingly bizarre on each subsequent go), the password they tried to use to satisfy the criteria and the kvKey so it can all be tied together. What that means is that the Cloudflare Workers KV entry created earlier gradually builds up as follows:
There are a couple of little conditions built into the code:
That's everything needed to lure the spammer in and record their pain, now for the really fun bit π
The very first time the spammer's password attempt is logged, the Cloudflare Worker sends me an email to let me know I have a new spammer hooked (this capability using MailChannels only launched this year):
It was so exciting getting this email yesterday, I swear it's the same sensation as literally getting a fish on your line! That link is one I can share to put the spammer's pain on display for the world to see. This is achieved with another Cloudflare Workers route that simply pulls out the logs for the given kvKey and formats it neatly in an HTML response:
Ah, satisfaction π I listed the amount of time the spammer burned with a goal to further refining the complexity criteria in the future to attempt to keep them "hooked" for longer. Is the requirement for a US post code in the password a bit too geographically specific, for example? Time will tell and I wholeheartedly welcome PRs to that effect in the original Password Purgatory API repo.
Oh - and just to ensure traction and exposure are maximised, there's a neatly formatted Twitter card that includes the last criteria and password used, you know, the ones that finally broke the spammer's spirit and caused them to give up:
Spammer burned a total of 80 seconds in Password Purgatory π #PasswordPurgatory https://t.co/VwSCHNZ2AW
β Troy Hunt (@troyhunt) August 3, 2022
Clearly, I've taken a great deal of pleasure in messing with spammers and I hope you do too. I've gotta be honest - I've never been so excited to go through my junk mail! But I also thoroughly enjoyed putting this together with Power Automate and Workers KV, I think it's super cool that you can pull an app together like this with a combination of browser-based config plus code and storage that runs directly in hundreds of globally distributed edge nodes around the world. I hope the spammers appreciate just how elegant this all is π€£
I didn't intend for a bunch of this week's vid to be COVID related, but between the breach of an anti-vaxxer website and the (unrelated) social comments directed at our state premier following some pretty simple advice, well, it just kinda turned out that way. But there's more on other breaches too, in particular the alleged Paytm one and the actual Customer.io one.
I'm really looking forward to next week's update, here's a little teaser of what you can expect to hear about then π€£
I broke Yoda's stick! 3D printing woes, and somehow I managed to get through the explanation without reverting to a chorus of My Stick by a Bad Lip Reading (and now you'd got that song stuck in your head). Loads of data breaches this week and whilst "legacy", still managed to demonstrate how bad some practices remain today (hi Shadi.com π). Never a dull moment in data breach land, more from there next week π