Login
In November 2022, the password manager service LastPass disclosed a breach in which hackers stole password vaults containing both encrypted and plaintext data for more than 25 million users. Since then, a steady trickle of six-figure cryptocurrency heists targeting security-conscious people throughout the tech industry has led some security experts to conclude that crooks likely have succeeded at cracking open some of the stolen LastPass vaults.
Taylor Monahan is lead product manager of MetaMask, a popular software cryptocurrency wallet used to interact with the Ethereum blockchain. Since late December 2022, Monahan and other researchers have identified a highly reliable set of clues that they say connect recent thefts targeting more than 150 people. Collectively, these individuals have been robbed of more than $35 million worth of crypto.
Monahan said virtually all of the victims she has assisted were longtime cryptocurrency investors, and security-minded individuals. Importantly, none appeared to have suffered the sorts of attacks that typically preface a high-dollar crypto heist, such as the compromise of one’s email and/or mobile phone accounts.
“The victim profile remains the most striking thing,” Monahan wrote. “They truly all are reasonably secure. They are also deeply integrated into this ecosystem, [including] employees of reputable crypto orgs, VCs [venture capitalists], people who built DeFi protocols, deploy contracts, run full nodes.”
Monahan has been documenting the crypto thefts via Twitter/X since March 2023, frequently expressing frustration in the search for a common cause among the victims. Then on Aug. 28, Monahan said she’d concluded that the common thread among nearly every victim was that they’d previously used LastPass to store their “seed phrase,” the private key needed to unlock access to their cryptocurrency investments.
MetaMask owner Taylor Monahan on Twitter. Image: twitter.com/tayvano_
Armed with your secret seed phrase, anyone can instantly access all of the cryptocurrency holdings tied to that cryptographic key, and move the funds to anywhere they like.
Which is why the best practice for many cybersecurity enthusiasts has long been to store their seed phrases either in some type of encrypted container — such as a password manager — or else inside an offline, special-purpose hardware encryption device, such as a Trezor or Ledger wallet.
“The seed phrase is literally the money,” said Nick Bax, director of analytics at Unciphered, a cryptocurrency wallet recovery company. “If you have my seed phrase, you can copy and paste that into your wallet, and then you can see all my accounts. And you can transfer my funds.”
Bax said he closely reviewed the massive trove of cryptocurrency theft data that Taylor Monahan and others have collected and linked together.
“It’s one of the broadest and most complex cryptocurrency investigations I’ve ever seen,” Bax said. “I ran my own analysis on top of their data and reached the same conclusion that Taylor reported. The threat actor moved stolen funds from multiple victims to the same blockchain addresses, making it possible to strongly link those victims.”
Bax, Monahan and others interviewed for this story say they’ve identified a unique signature that links the theft of more than $35 million in crypto from more than 150 confirmed victims, with roughly two to five high-dollar heists happening each month since December 2022.
KrebsOnSecurity has reviewed this signature but is not publishing it at the request of Monahan and other researchers, who say doing so could cause the attackers to alter their operations in ways that make their criminal activity more difficult to track.
But the researchers have published findings about the dramatic similarities in the ways that victim funds were stolen and laundered through specific cryptocurrency exchanges. They also learned the attackers frequently grouped together victims by sending their cryptocurrencies to the same destination crypto wallet.
A graphic published by @tayvano_ on Twitter depicting the movement of stolen cryptocurrencies from victims who used LastPass to store their crypto seed phrases.
By identifying points of overlap in these destination addresses, the researchers were then able to track down and interview new victims. For example, the researchers said their methodology identified a recent multi-million dollar crypto heist victim as an employee at Chainalysis, a blockchain analysis firm that works closely with law enforcement agencies to help track down cybercriminals and money launderers.
Chainalysis confirmed that the employee had suffered a high-dollar cryptocurrency heist late last month, but otherwise declined to comment for this story.
Bax said the only obvious commonality between the victims who agreed to be interviewed was that they had stored the seed phrases for their cryptocurrency wallets in LastPass.
“On top of the overlapping indicators of compromise, there are more circumstantial behavioral patterns and tradecraft which are also consistent between different thefts and support the conclusion,” Bax told KrebsOnSecuirty. “I’m confident enough that this is a real problem that I’ve been urging my friends and family who use LastPass to change all of their passwords and migrate any crypto that may have been exposed, despite knowing full well how tedious that is.”
LastPass declined to answer questions about the research highlighted in this story, citing an ongoing law enforcement investigation and pending litigation against the company in response to its 2022 data breach.
“Last year’s incident remains the subject of an ongoing investigation by law enforcement and is also the subject of pending litigation,” LastPass said in a written statement provided to KrebsOnSecurity. “Since last year’s attack on LastPass, we have remained in contact with law enforcement and continue to do so.”
Their statement continues:
“We have shared various technical information, Indicators of Compromise (IOCs), and threat actor tactics, techniques, and procedures (TTPs) with our law enforcement contacts as well as our internal and external threat intelligence and forensic partners in an effort to try and help identify the parties responsible. In the meantime, we encourage any security researchers to share any useful information they believe they may have with our Threat Intelligence team by contacting securitydisclosure@lastpass.com.”
On August 25, 2022, LastPass CEO Karim Toubba wrote to users that the company had detected unusual activity in its software development environment, and that the intruders stole some source code and proprietary LastPass technical information. On Sept. 15, 2022, LastPass said an investigation into the August breach determined the attacker did not access any customer data or password vaults.
But on Nov. 30, 2022, LastPass notified customers about another, far more serious security incident that the company said leveraged data stolen in the August breach. LastPass disclosed that criminal hackers had compromised encrypted copies of some password vaults, as well as other personal information.
In February 2023, LastPass disclosed that the intrusion involved a highly complex, targeted attack against a DevOps engineer who was one of only four LastPass employees with access to the corporate vault.
“This was accomplished by targeting the DevOps engineer’s home computer and exploiting a vulnerable third-party media software package, which enabled remote code execution capability and allowed the threat actor to implant keylogger malware,” LastPass officials wrote. “The threat actor was able to capture the employee’s master password as it was entered, after the employee authenticated with MFA, and gain access to the DevOps engineer’s LastPass corporate vault.”
Dan Goodin at Ars Technica reported and then confirmed that the attackers exploited a known vulnerability in a Plex media server that the employee was running on his home network, and succeeded in installing malicious software that stole passwords and other authentication credentials. The vulnerability exploited by the intruders was patched back in 2020, but the employee never updated his Plex software.
As it happens, Plex announced its own data breach one day before LastPass disclosed its initial August intrusion. On August 24, 2022, Plex’s security team urged users to reset their passwords, saying an intruder had accessed customer emails, usernames and encrypted passwords.
A basic functionality of LastPass is that it will pick and remember lengthy, complex passwords for each of your websites or online services. To automatically populate the appropriate credentials at any website going forward, you simply authenticate to LastPass using your master password.
LastPass has always emphasized that if you lose this master password, that’s too bad because they don’t store it and their encryption is so strong that even they can’t help you recover it.
But experts say all bets are off when cybercrooks can get their hands on the encrypted vault data itself — as opposed to having to interact with LastPass via its website. These so-called “offline” attacks allow the bad guys to conduct unlimited and unfettered “brute force” password cracking attempts against the encrypted data using powerful computers that can each try millions of password guesses per second.
“It does leave things vulnerable to brute force when the vaults are stolen en masse, especially if info about the vault HOLDER is available,” said Nicholas Weaver, a researcher at University of California, Berkeley’s International Computer Science Institute (ICSI) and lecturer at UC Davis. “So you just crunch and crunch and crunch with GPUs, with a priority list of vaults you target.”
How hard would it be for well-resourced criminals to crack the master passwords securing LastPass user vaults? Perhaps the best answer to this question comes from Wladimir Palant, a security researcher and the original developer behind the Adblock Plus browser plugin.
In a December 2022 blog post, Palant explained that the crackability of a LastPass master password depends largely on two things: The complexity of the master password, and the default settings for LastPass users, which appear to have varied quite a bit based on when those users began patronizing the service.
LastPass says that since 2018 it has required a twelve-character minimum for master passwords, which the company said “greatly minimizes the ability for successful brute force password guessing.”
But Palant said while LastPass indeed improved its master password defaults in 2018, it did not force all existing customers who had master passwords of lesser lengths to pick new credentials that would satisfy the 12-character minimum.
“If you are a LastPass customer, chances are that you are completely unaware of this requirement,” Palant wrote. “That’s because LastPass didn’t ask existing customers to change their master password. I had my test account since 2018, and even today I can log in with my eight-character password without any warnings or prompts to change it.”
Palant believes LastPass also failed to upgrade many older, original customers to more secure encryption protections that were offered to newer customers over the years. One important setting in LastPass is the number of “iterations,” or how many times your master password is run through the company’s encryption routines. The more iterations, the longer it takes an offline attacker to crack your master password.
Palant noted last year that for many older LastPass users, the initial default setting for iterations was anywhere from “1” to “500.” By 2013, new LastPass customers were given 5,000 iterations by default. In February 2018, LastPass changed the default to 100,100 iterations. And very recently, it upped that again to 600,000.
Palant said the 2018 change was in response to a security bug report he filed about some users having dangerously low iterations in their LastPass settings.
“Worse yet, for reasons that are beyond me, LastPass didn’t complete this migration,” Palant wrote. “My test account is still at 5,000 iterations, as are the accounts of many other users who checked their LastPass settings. LastPass would know how many users are affected, but they aren’t telling that. In fact, it’s painfully obvious that LastPass never bothered updating users’ security settings. Not when they changed the default from 1 to 500 iterations. Not when they changed it from 500 to 5,000. Only my persistence made them consider it for their latest change. And they still failed implementing it consistently.”
A chart on Palant’s blog post offers an idea of how increasing password iterations dramatically increases the costs and time needed by the attackers to crack someone’s master password. Palant said it would take a single GPU about a year to crack a password of average complexity with 500 iterations, and about 10 years to crack the same password run through 5,000 iterations.
Image: palant.info
However, these numbers radically come down when a determined adversary also has other large-scale computational assets at their disposal, such as a bitcoin mining operation that can coordinate the password-cracking activity across multiple powerful systems simultaneously.
Weaver said a password or passphrase with average complexity — such as “Correct Horse Battery Staple” is only secure against online attacks, and that its roughly 40 bits of randomness or “entropy” means a graphics card can blow through it in no time.
“An Nvidia 3090 can do roughly 4 million [password guesses] per second with 1000 iterations, but that would go down to 8 thousand per second with 500,000 iterations, which is why iteration count matters so much,” Weaver said. “So a combination of ‘not THAT strong of a password’ and ‘old vault’ and ‘low iteration count’ would make it theoretically crackable but real work, but the work is worth it given the targets.”
Reached by KrebsOnSecurity, Palant said he never received a response from LastPass about why the company apparently failed to migrate some number of customers to more secure account settings.
“I know exactly as much as everyone else,” Palant wrote in reply. “LastPass published some additional information in March. This finally answered the questions about the timeline of their breach – meaning which users are affected. It also made obvious that business customers are very much at risk here, Federated Login Services being highly compromised in this breach (LastPass downplaying as usual of course).”
Palant said upon logging into his LastPass account a few days ago, he found his master password was still set at 5,000 iterations.
KrebsOnSecurity interviewed one of the victims tracked down by Monahan, a software engineer and startup founder who recently was robbed of approximately $3.4 million worth of different cryptocurrencies. The victim agreed to tell his story in exchange for anonymity because he is still trying to claw back his losses. We’ll refer to him here as “Connor” (not his real name).
Connor said he began using LastPass roughly a decade ago, and that he also stored the seed phrase for his primary cryptocurrency wallet inside of LastPass. Connor chose to protect his LastPass password vault with an eight character master password that included numbers and symbols (~50 bits of entropy).
“I thought at the time that the bigger risk was losing a piece of paper with my seed phrase on it,” Connor said. “I had it in a bank security deposit box before that, but then I started thinking, ‘Hey, the bank might close or burn down and I could lose my seed phrase.'”
Those seed phrases sat in his LastPass vault for years. Then, early on the morning of Sunday, Aug. 27, 2023, Connor was awoken by a service he’d set up to monitor his cryptocurrency addresses for any unusual activity: Someone was draining funds from his accounts, and fast.
Like other victims interviewed for this story, Connor didn’t suffer the usual indignities that typically presage a cryptocurrency robbery, such as account takeovers of his email inbox or mobile phone number.
Connor said he doesn’t know the number of iterations his master password was given originally, or what it was set at when the LastPass user vault data was stolen last year. But he said he recently logged into his LastPass account and the system forced him to upgrade to the new 600,000 iterations setting.
“Because I set up my LastPass account so early, I’m pretty sure I had whatever weak settings or iterations it originally had,” he said.
Connor said he’s kicking himself because he recently started the process of migrating his cryptocurrency to a new wallet protected by a new seed phrase. But he never finished that migration process. And then he got hacked.
“I’d set up a brand new wallet with new keys,” he said. “I had that ready to go two months ago, but have been procrastinating moving things to the new wallet.”
Connor has been exceedingly lucky in regaining access to some of his stolen millions in cryptocurrency. The Internet is swimming with con artists masquerading as legitimate cryptocurrency recovery experts. To make matters worse, because time is so critical in these crypto heists, many victims turn to the first quasi-believable expert who offers help.
Instead, several friends steered Connor to Flashbots.net, a cryptocurrency recovery firm that employs several custom techniques to help clients claw back stolen funds — particularly those on the Ethereum blockchain.
According to Connor, Flashbots helped rescue approximately $1.5 million worth of the $3.4 million in cryptocurrency value that was suddenly swept out of his account roughly a week ago. Lucky for him, Connor had some of his assets tied up in a type of digital loan that allowed him to borrow against his various cryptocurrency assets.
Without giving away too many details about how they clawed back the funds, here’s a high level summary: When the crooks who stole Connor’s seed phrase sought to extract value from these loans, they were borrowing the maximum amount of credit that he hadn’t already used. But Connor said that left open an avenue for some of that value to be recaptured, basically by repaying the loan in many small, rapid chunks.
According to MetaMask’s Monahan, users who stored any important passwords with LastPass — particularly those related to cryptocurrency accounts — should change those credentials immediately, and migrate any crypto holdings to new offline hardware wallets.
“Really the ONLY thing you need to read is this,” Monahan pleaded to her 70,000 followers on Twitter/X: “PLEASE DON’T KEEP ALL YOUR ASSETS IN A SINGLE KEY OR SECRET PHRASE FOR YEARS. THE END. Split up your assets. Get a hw [hardware] wallet. Migrate. Now.”
If you also had passwords tied to banking or retirement accounts, or even just important email accounts — now would be a good time to change those credentials as well.
I’ve never been comfortable recommending password managers, because I’ve never seriously used them myself. Something about putting all your eggs in one basket. Heck, I’m so old-fashioned that most of my important passwords are written down and tucked away in safe places.
But I recognize this antiquated approach to password management is not for everyone. Connor says he now uses 1Password, a competing password manager that recently earned the best overall marks from Wired and The New York Times.
1Password says that three things are needed to decrypt your information: The encrypted data itself, your account password, and your Secret Key. Only you know your account password, and your Secret Key is generated locally during setup.
“The two are combined on-device to encrypt your vault data and are never sent to 1Password,” explains a 1Password blog post ‘What If 1Password Gets Hacked?‘ “Only the encrypted vault data lives on our servers, so neither 1Password nor an attacker who somehow manages to guess or steal your account password would be able to access your vaults – or what’s inside them.
Weaver said that Secret Key adds an extra level of randomness to all user master passwords that LastPass didn’t have.
“With LastPass, the idea is the user’s password vault is encrypted with a cryptographic hash (H) of the user’s passphrase,” Weaver said. “The problem is a hash of the user’s passphrase is remarkably weak on older LastPass vaults with master passwords that do not have many iterations. 1Password uses H(random-key||password) to generate the password, and it is why you have the QR code business when adding a new device.”
Weaver said LastPass deserves blame for not having upgraded iteration counts for all users a long time ago, and called the latest forced upgrades “a stunning indictment of the negligence on the part of LastPass.”
“That they never even notified all those with iteration counts of less than 100,000 — who are really vulnerable to brute force even with 8-character random passwords or ‘correct horse battery staple’ type passphrases — is outright negligence,” Weaver said. “I would personally advocate that nobody ever uses LastPass again: Not because they were hacked. Not because they had an architecture (unlike 1Password) that makes such hacking a problem. But because of their consistent refusal to address how they screwed up and take proactive efforts to protect their customers.”
Bax and Monahan both acknowledged that their research alone can probably never conclusively tie dozens of high-dollar crypto heists over the past year to the LastPass breach. But Bax says at this point he doesn’t see any other possible explanation.
“Some might say it’s dangerous to assert a strong connection here, but I’d say it’s dangerous to assert there isn’t one,” he said. “I was arguing with my fiance about this last night. She’s waiting for LastPass to tell her to change everything. Meanwhile, I’m telling her to do it now.”
What if I told you... that you could run a website from behind Cloudflare and only have 385 daily requests miss their cache and go through to the origin service?
No biggy, unless... that was out of a total of more than 166M requests in the same period:
Yep, we just hit "five nines" of cache hit ratio on Pwned Passwords being 99.999%. Actually, it was 99.9998% but we're at the point now where that's just splitting hairs, let's talk about how we've managed to only have two requests in a million hit the origin, beginning with a bit of history:
Optimising Caching on Pwned Passwords (with Workers)- @troyhunt - https://t.co/KjBtCwmhmT pic.twitter.com/BSfJbWyxMy
— Cloudflare (@Cloudflare) August 9, 2018
Ah, memories 😊 Back then, Pwned Passwords was serving way fewer requests in a month than what we do in a day now and the cache hit ratio was somewhere around 92%. Put another way, instead of 2 in every million requests hitting the origin it was 85k. And we were happy with that! As the years progressed, the traffic grew and the caching model was optimised so our stats improved:
There it is - Pwned Passwords is now doing north of 2 *billion* requests a month, peaking at 91.59M in a day with a cache-hit ratio of 99.52%. All free, open source and out there for the community to do good with 😊 pic.twitter.com/DSJOjb2CxZ
— Troy Hunt (@troyhunt) May 24, 2022
And that's pretty much where we levelled out, at about the 99-and-a-bit percent mark. We were really happy with that as it was now only 5k requests per million hitting the origin. There was bound to be a number somewhere around that mark due to the transient nature of cache and eviction criteria inevitably meaning a Cloudflare edge node somewhere would need to reach back to the origin website and pull a new copy of the data. But what if Cloudflare never had to do that unless explicitly instructed to do so? I mean, what if it just stayed in their cache unless we actually changed the source file and told them to update their version? Welcome to Cloudflare Cache Reserve:
Ok, so I may have annotated the important bit but that's what it feels like - magic - because you just turn it on and... that's it. You still serve your content the same way, you still need the appropriate cache headers and you still have the same tiered caching as before, but now there's a "cache reserve" sitting between that and your origin. It's backed by R2 which is their persistent data store and you can keep your cached things there for as long as you want. However, per the earlier link, it's not free:
You pay based on how much you store for how long, how much you write and how much you read. Let's put that in real terms and just as a brief refresher (longer version here), remember that Pwned Passwords is essentially just 16^5 (just over 1 million) text files of about 30kb each for the SHA-1 hashes and a similar number for the NTLM ones (albeit slight smaller file sizes). Here are the Cache Reserve usage stats for the last 9 days:
We can now do some pretty simple maths with that and working on the assumption of 9 days, here's what we get:
2 bucks a day 😲 But this has taken nearly 16M requests off my origin service over this period of time so I haven't paid for the Azure Function execution (which is cheap) nor the egress bandwidth (which is not cheap). But why are there only 16M read operations over 9 days when earlier we saw 167M requests to the API in a single day? Because if you scroll back up to the "insert magic here" diagram, Cache Reserve is only a fallback position and most requests (i.e. 99.52% of them) are still served from the edge caches.
Note also that there are nearly 1M write operations and there are 2 reasons for this:
An untold number of businesses rely on Pwned Passwords as an integral part of their registration, login and password reset flows. Seriously, the number is "untold" because we have no idea who's actually using it, we just know the service got hit three and a quarter billion times in the last 30 days:
Giving consumers of the service confidence that not only is it highly resilient, but also massively fast is essential to adoption. In turn, more adoption helps drive better password practices, less account takeovers and more smiles all round 😊
As those remaining hash prefixes populate Cache Reserve, keep an eye on the "cf-cache-status" response header. If you ever see a value of "MISS" then congratulations, you're literally one in a million!
Full disclosure: Cloudflare provides services to HIBP for free and they helped in getting Cache Reserve up and running. However, they had no idea I was writing this blog post and reading it live in its entirety is the first anyone there has seen it. Surprise! 👋
FreshRSS 