Login
In large metropolitan areas, tourists are often easy to spot because they’re far more inclined than locals to gaze upward at the surrounding skyscrapers. Security experts say this same tourist dynamic is a dead giveaway in virtually all computer intrusions that lead to devastating attacks like data theft and ransomware, and that more organizations should set simple virtual tripwires that sound the alarm when authorized users and devices are spotted exhibiting this behavior.
In a blog post published last month, Cisco Talos said it was seeing a worrisome “increase in the rate of high-sophistication attacks on network infrastructure.” Cisco’s warning comes amid a flurry of successful data ransom and state-sponsored cyber espionage attacks targeting some of the most well-defended networks on the planet.
But despite their increasing complexity, a great many initial intrusions that lead to data theft could be nipped in the bud if more organizations started looking for the telltale signs of newly-arrived cybercriminals behaving like network tourists, Cisco says.
“One of the most important things to talk about here is that in each of the cases we’ve seen, the threat actors are taking the type of ‘first steps’ that someone who wants to understand (and control) your environment would take,” Cisco’s Hazel Burton wrote. “Examples we have observed include threat actors performing a ‘show config,’ ‘show interface,’ ‘show route,’ ‘show arp table’ and a ‘show CDP neighbor.’ All these actions give the attackers a picture of a router’s perspective of the network, and an understanding of what foothold they have.”
Cisco’s alert concerned espionage attacks from China and Russia that abused vulnerabilities in aging, end-of-life network routers. But at a very important level, it doesn’t matter how or why the attackers got that initial foothold on your network.
It might be zero-day vulnerabilities in your network firewall or file-transfer appliance. Your more immediate and primary concern has to be: How quickly can you detect and detach that initial foothold?
The same tourist behavior that Cisco described attackers exhibiting vis-a-vis older routers is also incredibly common early on in ransomware and data ransom attacks — which often unfurl in secret over days or weeks as attackers methodically identify and compromise a victim’s key network assets.
These virtual hostage situations usually begin with the intruders purchasing access to the target’s network from dark web brokers who resell access to stolen credentials and compromised computers. As a result, when those stolen resources first get used by would-be data thieves, almost invariably the attackers will run a series of basic commands asking the local system to confirm exactly who and where they are on the victim’s network.
This fundamental reality about modern cyberattacks — that cybercriminals almost always orient themselves by “looking up” who and where they are upon entering a foreign network for the first time — forms the business model of an innovative security company called Thinkst, which gives away easy-to-use tripwires or “canaries” that can fire off an alert whenever all sorts of suspicious activity is witnessed.
“Many people have pointed out that there are a handful of commands that are overwhelmingly run by attackers on compromised hosts (and seldom ever by regular users/usage),” the Thinkst website explains. “Reliably alerting when a user on your code-sign server runs whoami.exe can mean the difference between catching a compromise in week-1 (before the attackers dig in) and learning about the attack on CNN.”
These canaries — or “canary tokens” — are meant to be embedded inside regular files, acting much like a web beacon or web bug that tracks when someone opens an email.
The Canary Tokens website from Thinkst Canary lists nearly two-dozen free customizable canaries.
“Imagine doing that, but for file reads, database queries, process executions or patterns in log files,” the Canary Tokens documentation explains. “Canarytokens does all this and more, letting you implant traps in your production systems rather than setting up separate honeypots.”
Thinkst operates alongside a burgeoning industry offering so-called “deception” or “honeypot” services — those designed to confuse, disrupt and entangle network intruders. But in an interview with KrebsOnSecurity, Thinkst founder and CEO Haroon Meer said most deception techniques involve some degree of hubris.
“Meaning, you’ll have deception teams in your network playing spy versus spy with people trying to break in, and it becomes this whole counterintelligence thing,” Meer said. “Nobody really has time for that. Instead, we are saying literally the opposite: That you’ve probably got all these [security improvement] projects that are going to take forever. But while you’re doing all that, just drop these 10 canaries, because everything else is going to take a long time to do.”
The idea here is to lay traps in sensitive areas of your network or web applications where few authorized users should ever trod. Importantly, the canary tokens themselves are useless to an attacker. For example, that AWS canary token sure looks like the digital keys to your cloud, but the token itself offers no access. It’s just a lure for the bad guys, and you get an alert when and if it is ever touched.
One nice thing about canary tokens is that Thinkst gives them away for free. Head over to canarytokens.org, and choose from a drop-down menu of available tokens, including:
-a web bug / URL token, designed to alert when a particular URL is visited;
-a DNS token, which alerts when a hostname is requested;
-an AWS token, which alerts when a specific Amazon Web Services key is used;
-a “custom exe” token, to alert when a specific Windows executable file or DLL is run;
-a “sensitive command” token, to alert when a suspicious Windows command is run.
-a Microsoft Excel/Word token, which alerts when a specific Excel or Word file is accessed.
Much like a “wet paint” sign often encourages people to touch a freshly painted surface anyway, attackers often can’t help themselves when they enter a foreign network and stumble upon what appear to be key digital assets, Meer says.
“If an attacker lands on your server and finds a key to your cloud environment, it’s really hard for them not to try it once,” Meer said. “Also, when these sorts of actors do land in a network, they have to orient themselves, and while doing that they are going to trip canaries.”
Meer says canary tokens are as likely to trip up attackers as they are “red teams,” security experts hired or employed by companies seeking to continuously probe their own computer systems and networks for security weaknesses.
“The concept and use of canary tokens has made me very hesitant to use credentials gained during an engagement, versus finding alternative means to an end goal,” wrote Shubham Shah, a penetration tester and co-founder of the security firm Assetnote. “If the aim is to increase the time taken for attackers, canary tokens work well.”
Thinkst makes money by selling Canary Tools, which are honeypots that emulate full blown systems like Windows servers or IBM mainframes. They deploy in minutes and include a personalized, private Canarytoken server.
“If you’ve got a sophisticated defense team, you can start putting these things in really interesting places,” Meer said. “Everyone says their stuff is simple, but we obsess over it. It’s really got to be so simple that people can’t mess it up. And if it works, it’s the best bang for your security buck you’re going to get.”
Further reading:
Dark Reading: Credential Canaries Create Minefield for Attackers
NCC Group: Extending a Thinkst Canary to Become an Interactive Honeypot
Cruise Automation’s experience deploying canary tokens
A New York fintech biz is set to pay $1 million in fines under a US Securities and Exchange Commission order that claims it advertised "annualized" returns on Titan Crypto of up to 2,700 percent, a number based on a "purely hypothetical account."…
Webinar You could be forgiven for wondering if anything can ever again be completely straightforward or demonstrably authentic in a world where generative AI can masquerade convincingly as your mother, or express itself in the exact language your best friend might use.…
Updated Apple last year introduced a security feature called App Management that's designed to prevent one application from modifying another without authorization under macOS Ventura – but a developer claims it’s not very good at its job under some circumstances.…
A critical authentication bypass bug in MobileIron Sentry has been exploited in the wild, its maker Ivanti said in an advisory on Monday.…
With America outspending the rest of the world on space technologies, those systems and their blueprints are a highly alluring and lucrative target for sticky-fingered spies, Uncle Sam has reminded industry.…
Insiders are to blame for a May data breach at Tesla, the company claimed in filings after news of the incident was reported months ago by German media.…
Users of the popular WinRAR compression and archiving tool should update now to avoid a vulnerability that allows code to be run when a user opens a RAR file.…
There's a "hidden" API on HIBP. Well, it's not "hidden" insofar as it's easily discoverable if you watch the network traffic from the client, but it's not meant to be called directly, rather only via the web app. It's called "unified search" and it looks just like this:
It's been there in one form or another since day 1 (so almost a decade now), and it serves a sole purpose: to perform searches from the home page. That is all - only from the home page. It's called asynchronously from the client without needing to post back the entire page and by design, it's super fast and super easy to use. Which is bad. Sometimes.
To understand why it's bad we need to go back in time all the way to when I first launched the API that was intended to be consumed programmatically by other people's services. That was easy, because it was basically just documenting the API that sat behind the home page of the website already, the predecessor to the one you see above. And then, unsurprisingly in retrospect, it started to be abused so I had to put a rate limit on it. Problem is, that was a very rudimentary IP-based rate limit and it could be circumvented by someone with enough IPs, so fast forward a bit further and I put auth on the API which required a nominal payment to access it. At the same time, that unified search endpoint was created and home page searches updated to use that rather than the publicly documented API. So, 2 APIs with 2 different purposes.
The primary objective for putting a price on the public API was to tackle abuse. And it did - it stopped it dead. By attaching a rate limit to a key that required a credit card to purchase it, abusive practices (namely enumerating large numbers of email addresses) disappeared. This wasn't just about putting a financial cost to queries, it was about putting an identity cost to them; people are reluctant to start doing nasty things with a key traceable back to their own payment card! Which is why they turned their attention to the non-authenticated, non-documented unified search API.
Let's look at a 3 day period of requests to that API earlier this year, keeping in mind this should only ever be requested organically by humans performing searches from the home page:
This is far from organic usage with requests peaking at 121.3k in just 5 minutes. Which poses an interesting question: how do you create an API that should only be consumed asynchronously from a web page and never programmatically via a script? You could chuck a CAPTCHA on the front page and require that be solved first but let's face it, that's not a pleasant user experience. Rate limit requests by IP? See the earlier problem with that. Block UA strings? Pointless, because they're easily randomised. Rate limit an ASN? It gets you part way there, but what happens when you get a genuine flood of traffic because the site has hit the mainstream news? It happens.
Over the years, I've played with all sorts of combinations of firewall rules based on parameters such as geolocations with incommensurate numbers of requests to their populations, JA3 fingerprints and, of course, the parameters mentioned above. Based on the chart above these obviously didn't catch all the abusive traffic, but they did catch a significant portion of it:
If you combine it with the previous graph, that's about a third of all the bad traffic in that period or in other words, two thirds of the bad traffic was still getting through. There had to be a better way, which brings us to Cloudflare's Turnstile:
With Turnstile, we adapt the actual challenge outcome to the individual visitor or browser. First, we run a series of small non-interactive JavaScript challenges gathering more signals about the visitor/browser environment. Those challenges include, proof-of-work, proof-of-space, probing for web APIs, and various other challenges for detecting browser-quirks and human behavior. As a result, we can fine-tune the difficulty of the challenge to the specific request and avoid ever showing a visual puzzle to a user.
"Avoid ever showing a visual puzzle to a user" is a polite way of saying they avoid the sucky UX of CAPTCHA. Instead, Turnstile offers the ability to issue a "non-interactive challenge" which implements the sorts of clever techniques mentioned above and as it relates to this blog post, that can be an invisible non-interactive challenge. This is one of 3 different widget types with the others being a visible non-interactive challenge and a non-intrusive interactive challenge. For my purposes on HIBP, I wanted a zero-friction implementation nobody saw, hence the invisible approach. Here's how it works:
Get it? Ok, let's break it down further as it relates to HIBP, starting with when the front page first loads and it embeds the Turnstile widget from Cloudflare:
FreshRSS 
<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>The widget takes responsibility for running the non-interactive challenge and returning a token. This needs to be persisted somewhere on the client side which brings us to embedding the widget:
<div ID="turnstileWidget" class="cf-turnstile" data-sitekey="0x4AAAAAAADY3UwkmqCvH8VR" data-callback="turnstileCompleted"></div>Per the docs in that link, the main thing here is to have an element with the "cf-turnstile" class set on it. If you happen to go take a look at the HIBP HTML source right now, you'll see that element precisely as it appears in the code block above. However, check it out in your browser's dev tools so you can see how it renders in the DOM and it will look more like this:
Expand that DIV tag and you'll find a whole bunch more content set as a result of loading the widget, but that's not relevant right now. What's important is the data-token attribute because that's what's going to prove you're not a bot when you run the search. How you implement this from here is up to you, but what HIBP does is picks up the token and sets it in the "cf-turnstile-response" header then sends it along with the request when that unified search endpoint is called:
So, at this point we've issued a challenge, the browser has solved the challenge and received a token back, now that token has been sent along with the request for the actual resource the user wanted, in this case the unified search endpoint. The final step is to validate the token and for this I'm using a Cloudflare worker. I've written a lot about workers in the past so here's the short pitch: it's code that runs in each one of Cloudflare's 300+ edge nodes around the world and can inspect and modify requests and responses on the fly. I already had a worker to do some other processing on unified search requests, so I just added the following:
const token = request.headers.get('cf-turnstile-response');
if (token === null) {
return new Response('Missing Turnstile token', { status: 401 });
}
const ip = request.headers.get('CF-Connecting-IP');
let formData = new FormData();
formData.append('secret', '[secret key goes here]');
formData.append('response', token);
formData.append('remoteip', ip);
const turnstileUrl = 'https://challenges.cloudflare.com/turnstile/v0/siteverify';
const result = await fetch(turnstileUrl, {
body: formData,
method: 'POST',
});
const outcome = await result.json();
if (!outcome.success) {
return new Response('Invalid Turnstile token', { status: 401 });
}That should be pretty self-explanatory and you can find the docs for this on Cloudflare's server-side validation page which goes into more detail, but in essence, it does the following:
And because this is all done in a Cloudflare worker, any of those 401 responses never even touch the origin. Not only do I not need to process the request in Azure, the person attempting to abuse my API gets a nice speedy response directly from an edge node near them 🙂
So, what does this mean for bots? If there's no token then they get booted out right away. If there's a token but it's not valid then they get booted out at the end. But can't they just take a previously generated token and use that? Well, yes, but only once:
If the same response is presented twice, the second and each subsequent request will generate an error stating that the response has already been consumed.
And remember, a real browser had to generate that token in the first place so it's not like you can just automate the process of token generation then throw it at the API above. (Sidenote: that server-side validation link includes how to handle idempotency, for example when retrying failed requests.) But what if a real human fails the verification? That's entirely up to you but in HIBP's case, that 401 response causes a fallback to a full page post back which then implements other controls, for example an interactive challenge.
Time for graphs and stats, starting with the one in the hero image of this page where we can see the number of times Turnstile was issued and how many times it was solved over the week prior to publishing this post:
That's a 91% hit rate of solved challenges which is great. That remaining 9% is either humans with a false positive or... bots getting rejected 😎
More graphs, this time how many requests to the unified search page were rejected by Turnstile:
That 990k number doesn't marry up with the 476k unsolved ones from before because they're 2 different things: the unsolved challenges are when the Turnstile widget is loaded but not solved (hopefully due to it being a bot rather than a false positive), whereas the 401 responses to the API is when a successful (and previously unused) Turnstile token isn't in the header. This could be because the token wasn't present, wasn't solved or had already been used. You get more of a sense of how many of these rejected requests were legit humans when you drill down into attributes like the JA3 fingerprints:
In other words, of those 990k failed requests, almost 40% of them were from the same 5 clients. Seems legit 🤔
And about a third were from clients with an identical UA string:
And so on and so forth. The point being that the number of actual legitimate requests from end users that were inconvenienced by Turnstile would be exceptionally small, almost certainly a very low single-digit percentage. I'll never know exactly because bots obviously attempt to emulate legit clients and sometimes legit clients look like bots and if we could easily solve this problem then we wouldn't need Turnstile in the first place! Anecdotally, that very small false positive number stacks up as people tend to complain pretty quickly when something isn't optimal, and I implemented this all the way back in March. Yep, 5 months ago, and I've waited this long to write about it just to be confident it's actually working. Over 100M Turnstile challenges later, I'm confident it is - I've not seen a single instance of abnormal traffic spikes to the unified search endpoint since rolling this out. What I did see initially though is a lot of this sort of thing:
By now it should be pretty obvious what's going on here, and it should be equally obvious that it didn't work out real well for them 😊
The bot problem is a hard one for those of us building services because we're continually torn in different directions. We want to build a slick UX for humans but an obtrusive one for bots. We want services to be easily consumable, but only in the way we intend them to... which might be by the good bots playing by the rules!
I don't know exactly what Cloudflare is doing in that challenge and I'll be honest, I don't even know what a "proof-of-space" is. But the point of using a service like this is that I don't need to know! What I do know is that Cloudflare sees about 20% of the internet's traffic and because of that, they're in an unrivalled position to look at a request and make a determination on its legitimacy.
If you're in my shoes, go and give Turnstile a go. And if you want to consume data from HIBP, go and check out the official API docs, the uh, unified search doesn't work real well for you any more 😎
Opinion Information wants to be free. This usefully ambiguous battle cry has been the mischievous slogan of hackers since early networking thinker Stuart Brand coined it in the early 1980s. Intended as part of a discussion about the inherent contradictions of intellectual property, it has bestowed irony in many other places since.…
Infosec in brief Someone at Microsoft has some explaining to do after a messed-up DNS record caused emails sent from accounts using Microsoft's Outlook Hotmail service to be rejected and directed to spam folders starting on Thursday.…
I started a free newsletter to share threat hunting tips.
This week hasd been manic! Non-stop tickets related to the new HIBP domain subscription service, scrambling to support invoicing and resellers, struggling our way through some odd Stripe things and so on and so forth. It's all good stuff and there have been very few issues of note (and all of those have merely been people getting to grips with the new model), so all in all, it's happy days 😊
An Interpol-led operation arrested 14 suspects and identified 20,674 "suspicious" networks spanning 25 African countries that international cops have linked to more than $40 million in cybercrime losses.…
Here's a heads up. Another version of BlackCat ransomware has been spotted extorting victims. This variant embeds two tools, we're told: the network toolkit Impacket for lateral movement within compromised environments, and Remcom for remote code execution.…
