Login
In part one of this issue of our Black Hat Asia NOC blog, you will find:
Cisco Meraki was asked by Black Hat Events to be the Official Wired and Wireless Network Equipment, for Black Hat Asia 2022, in Singapore, 10-13 May 2022; in addition to providing the Mobile Device Management (since Black Hat USA 2021), Malware Analysis (since Black Hat USA 2016), & DNS (since Black Hat USA 2017) for the Network Operations Center. We were proud to collaborate with NOC partners Gigamon, IronNet, MyRepublic, NetWitness and Palo Alto Networks.
To accomplish this undertaking in a few weeks’ time, after the conference had a green light with the new COVID protocols, Cisco Meraki and Cisco Secure leadership gave their full support to send the necessary hardware, software licenses and staff to Singapore. Thirteen Cisco engineers deployed to the Marina Bay Sands Convention Center, from Singapore, Australia, United States and United Kingdom; with two additional remote Cisco engineers from the United States.
From attendee to press to volunteer – coming back to Black Hat as NOC volunteer by Humphrey Cheung
Loops in the networking world are usually considered a bad thing. Spanning tree loops and routing loops happen in an instant and can ruin your whole day, but over the 2nd week in May, I made a different kind of loop. Twenty years ago, I first attended the Black Hat and Defcon conventions – yay Caesars Palace and Alexis Park – a wide-eyed tech newbie who barely knew what WEP hacking, Driftnet image stealing and session hijacking meant. The community was amazing and the friendships and knowledge I gained, springboarded my IT career.
In 2005, I was lucky enough to become a Senior Editor at Tom’s Hardware Guide and attended Black Hat as accredited press from 2005 to 2008. From writing about the latest hardware zero-days to learning how to steal cookies from the master himself, Robert Graham, I can say, without any doubt, Black Hat and Defcon were my favorite events of the year.
Since 2016, I have been a Technical Solutions Architect at Cisco Meraki and have worked on insanely large Meraki installations – some with twenty thousand branches and more than a hundred thousand access points, so setting up the Black Hat network should be a piece of cake right? Heck no, this is unlike any network you’ve experienced!
As an attendee and press, I took the Black Hat network for granted. To take a phrase that we often hear about Cisco Meraki equipment, “it just works”. Back then, while I did see access points and switches around the show, I never really dived into how everything was set up.
A serious challenge was to secure the needed hardware and ship it in time for the conference, given the global supply chain issues. Special recognition to Jeffry Handal for locating the hardware and obtaining the approvals to donate to Black Hat Events. For Black Hat Asia, Cisco Meraki shipped:
Let’s start with availability. iPads and iPhones are scanning QR codes to register attendees. Badge printers need access to the registration system. Training rooms all have their separate wireless networks – after all, Black Hat attendees get a baptism by fire on network defense and attack. To top it all off, hundreds of attendees gulped down terabytes of data through the main conference wireless network.
All this connectivity was provided by Cisco Meraki access points, switches, security appliances, along with integrations into SecureX, Umbrella and other products. We fielded a literal army of engineers to stand up the network in less than two days… just in time for the training sessions on May 10 to 13th and throughout the Black Hat Briefings and Business Hall on May 12 and 13.
Let’s talk security and visibility. For a few days, the Black Hat network is probably one of the most hostile in the world. Attendees learn new exploits, download new tools and are encouraged to test them out. Being able to drill down on attendee connection details and traffic was instrumental on ensuring attendees didn’t get too crazy.
On the wireless front, we made extensive use of our Radio Profiles to reduce interference by tuning power and channel settings. We enabled band steering to get more clients on the 5GHz bands versus 2.4GHz and watched the Location Heatmap like a hawk looking for hotspots and dead areas. Handling the barrage of wireless change requests – enable or disabling this SSID, moving VLANs (Virtual Local Area Networks), enabling tunneling or NAT mode, – was a snap with the Meraki Dashboard.
While the Cisco Meraki Dashboard is extremely powerful, we happily supported exporting of logs and integration in major event collectors, such as the NetWitness SIEM and even the Palo Alto firewall. On Thursday morning, the NOC team found a potentially malicious Macbook Pro performing vulnerability scans against the Black Hat management network. It is a balance, as we must allow trainings and demos connect to malicious websites, download malware and execute. However, there is a Code of Conduct to which all attendees are expected to follow and is posted at Registration with a QR code.
The Cisco Meraki network was exporting syslog and other information to the Palo Alto firewall, and after correlating the data between the Palo Alto Dashboard and Cisco Meraki client details page, we tracked down the laptop to the Business Hall.
We briefed the NOC management, who confirmed the scanning was violation of the Code of Conduct, and the device was blocked in the Meraki Dashboard, with the instruction to come to the NOC.
The device name and location made it very easy to determine to whom it belonged in the conference attendees.
A delegation from the NOC went to the Business Hall, politely waited for the demo to finish at the booth and had a thoughtful conversation with the person about scanning the network.
Coming back to Black Hat as a NOC volunteer was an amazing experience. While it made for long days with little sleep, I really can’t think of a better way to give back to the conference that helped jumpstart my professional career.
Meraki MR, MS, MX and Systems Manager by Paul Fidler
With the invitation extended to Cisco Meraki to provide network access, both from a wired and wireless perspective, there was an opportunity to show the value of the Meraki platform integration capabilities of Access Points (AP), switches, security appliances and mobile device management.
The first amongst this was the use of the Meraki API. We were able to import the list of MAC addresses of the Meraki MRs, to ensure that the APs were named appropriately and tagged, using a single source of truth document shared with the NOC management and partners, with the ability to update en masse at any time.
On the first day of NOC setup, the Cisco team walked around the venue to discuss AP placements with the staff of the Marina Bay Sands. Whilst we had a simple Powerpoint showing approximate AP placements for the conference, it was noted that the venue team had an incredibly detailed floor plan of the venue. This was acquired in PDF and uploaded into the Meraki Dashboard; and with a little fine tuning, aligned perfectly with the Google Map.
Meraki APs were then placed physically in the venue meeting and training rooms, and very roughly on the floor plan. One of the team members then used a printout of the floor plan to mark accurately the placement of the APs. Having the APs named, as mentioned above, made this an easy task (walking around the venue notwithstanding!). This enabled accurate heatmap capability.
The Location Heatmap was a new capability for Black Hat NOC, and the client data visualized in NOC continued to be of great interest to the Black Hat management team, such as which training, briefing and sponsor booths drew the most interest.
The ability to use SSID Availability was incredibly useful. It allowed ALL of the access points to be placed within a single Meraki Network. Not only that, because of the training events happening during the week, as well as TWO dedicated SSIDs for the Registration and lead tracking iOS devices (more of which later), one for initial provisioning (which was later turned off), and one for certificated based authentication, for a very secure connection.
We were able to monitor the number of connected clients, network usage, the persons passing by the network and location analytics, throughout the conference days. We provided visibility access to the Black Hat NOC management and the technology partners (along with full API access), so they could integrate with the network platform.
Meraki alerts are exactly that: the ability to be alerted to something that happens in the Dashboard. Default behavior is to be emailed when something happens. Obviously, emails got lost in the noise, so a web hook was created in SecureX orchestration to be able to consume Meraki alerts and send it to Slack (the messaging platform within the Black Hat NOC), using the native template in the Meraki Dashboard. The first alert to be created was to be alerted if an AP went down. We were to be alerted after five minutes of an AP going down, which is the smallest amount of time available before being alerted.
The bot was ready; however, the APs stayed up the entire time!
Applying the lessons learned at Black Hat Europe 2021, for the initial configuration of the conference iOS devices, we set up the Registration iPads and lead retrieval iPhones with Umbrella, Secure Endpoint and WiFi config. Devices were, as in London, initially configured using Apple Configurator, to both supervise and enroll the devices into a new Meraki Systems Manager instance in the Dashboard.
However, Black Hat Asia 2022 offered us a unique opportunity to show off some of the more integrated functionality.
System Apps were hidden and various restrictions (disallow joining of unknown networks, disallow tethering to computers, etc.) were applied, as well as a standard WPA2 SSID for the devices that the device vendor had set up (we gave them the name of the SSID and Password).
We also stood up a new SSID and turned-on Sentry, which allows you to provision managed devices with, not only the SSID information, but also a dynamically generated certificate. The certificate authority and radius server needed to do this 802.1x is included in the Meraki Dashboard automatically! When the device attempts to authenticate to the network, if it doesn’t have the certificate, it doesn’t get access. This SSID, using SSID availability, was only available to the access points in the Registration area.
Using the Sentry allowed us to easily identify devices in the client list.
One of the alerts generated with SysLog by Meraki, and then viewable and correlated in the NetWitness SIEM, was a ‘De Auth’ event that came from an access point. Whilst we had the IP address of the device, making it easy to find, because the event was a de auth, meaning 802.1x, it narrowed down the devices to JUST the iPads and iPhones used for registration (as all other access points were using WPA2). This was further enhanced by seeing the certificate name used in the de-auth:
Along with the certificate name was the name of the AP: R**
One of the inherent problems with iOS device location is when devices are used indoors, as GPS signals just aren’t strong enough to penetrate modern buildings. However, because the accurate location of the Meraki access points was placed on the floor plan in the Dashboard, and because the Meraki Systems Manager iOS devices were in the same Dashboard organization as the access points, we got to see a much more accurate map of devices compared to Black Hat Europe 2021 in London.
When the conference Registration closed on the last day and the Business Hall Sponsors all returned their iPhones, we were able to remotely wipe all of the devices, removing all attendee data, prior to returning to the device contractor.
Meraki Scanning API Receiver by Christian Clasen
Leveraging the ubiquity of both WiFi and Bluetooth radios in mobile devices and laptops, Cisco Meraki’s wireless access points can detect and provide location analytics to report on user foot traffic behavior. This can be useful in retail scenarios where customers desire location and movement data to better understand the trends of engagement in their physical stores.
Meraki can aggregate real-time data of detected WiFi and Bluetooth devices and triangulate their location rather precisely when the floorplan and AP placement has been diligently designed and documented. At the Black Hat Asia conference, we made sure to properly map the AP locations carefully to ensure the highest accuracy possible.
This scanning data is available for clients whether they are associated with the access points or not. At the conference, we were able to get very detailed heatmaps and time-lapse animations representing the movement of attendees throughout the day. This data is valuable to conference organizers in determining the popularity of certain talks, and the attendance at things like keynote presentations and foot traffic at booths.
This was great for monitoring during the event, but the Dashboard would only provide 24-hours of scanning data, limiting what we could do when it came to long-term data analysis. Fortunately for us, Meraki offers an API service we can use to capture this treasure trove offline for further analysis. We only needed to build a receiver for it.
The Scanning API requires that the customer stand up infrastructure to store the data, and then register with the Meraki cloud using a verification code and secret. It is composed of two endpoints:
Returns the validator string in the response body
[GET] https://yourserver/
This endpoint is called by Meraki to validate the receiving server. It expects to receive a string that matches the validator defined in the Meraki Dashboard for the respective network.
Accepts an observation payload from the Meraki cloud
[POST] https://yourserver/
This endpoint is responsible for receiving the observation data provided by Meraki. The URL path should match that of the [GET] request, used for validation.
The response body will consist of an array of JSON objects containing the observations at an aggregate per network level. The JSON will be determined based on WiFi or BLE device observations as indicated in the type parameter.
What we needed was a simple technology stack that would contain (at minimum) a publicly accessible web server capable of TLS. In the end, the simplest implementation was a web server written using Python Flask, in a Docker container, deployed in AWS, connected through ngrok.
In fewer than 50 lines of Python, we could accept the inbound connection from Meraki and reply with the chosen verification code. We would then listen for the incoming POST data and dump it into a local data store for future analysis. Since this was to be a temporary solution (the duration of the four-day conference), the thought of registering a public domain and configuring TLS certificates wasn’t particularly appealing. An excellent solution for these types of API integrations is ngrok (https://ngrok.com/). And a handy Python wrapper was available for simple integration into the script (https://pyngrok.readthedocs.io/en/latest/index.html).
We wanted to easily re-use this stack next time around, so it only made sense to containerize it in Docker. This way, the whole thing could be stood up at the next conference, with one simple command. The image we ended up with would mount a local volume, so that the ingested data would remain persistent across container restarts.
Ngrok allowed us to create a secure tunnel from the container that could be connected in the cloud to a publicly resolvable domain with a trusted TLS certificate generated for us. Adding that URL to the Meraki Dashboard is all we needed to do start ingesting the massive treasure trove of location data from the Aps – nearly 1GB of JSON over 24 hours.
This “quick and dirty” solution illustrated the importance of interoperability and openness in the technology space when enabling security operations to gather and analyze the data they require to monitor and secure events like Black Hat, and their enterprise networks as well. It served us well during the conference and will certainly be used again going forward.
Check out part two of the blog, Black Hat Asia 2022 Continued: Cisco Secure Integrations, where we will discuss integrating NOC operations and making your Cisco Secure deployment more effective:
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team: Aditya Sankar, Aldous Yeung, Alejo Calaoagan, Ben Greenbaum, Christian Clasen, Felix H Y Lam, George Dorsey, Humphrey Cheung, Ian Redden, Jeffrey Chua, Jeffry Handal, Jonny Noble, Matt Vander Horst, Paul Fidler and Steven Fan.
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially James Holland), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, James Pope, Steve Fink and Steve Oldenbourg).
For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In part one of our Black Hat Asia 2022 NOC blog, we discussed building the network with Meraki:
In this part two, we will discuss:
SecureX: Bringing Threat Intelligence Together by Ian Redden
In addition to the Meraki networking gear, Cisco Secure also shipped two Umbrella DNS virtual appliances to Black Hat Asia, for internal network visibility with redundancy, in addition to providing:
Cisco Secure Threat Intelligence (correlated through SecureX)
Donated Partner Threat Intelligence (correlated through SecureX)
alphaMountain.ai threat intelligence
Open-Source Threat Intelligence (correlated through SecureX)
Continued Integrations from past Black Hat events
New Integrations Created at Black Hat Asia 2022
Device type spoofing event by Jonny Noble
Overview
During the conference, a NOC Partner informed us that they received an alert from May 10 concerning an endpoint client that accessed two domains that they saw as malicious:
Client details from Partner:
Based on the user agent, the partner derived that the device type was an Apple iPhone.
SecureX analysis
Umbrella Investigate analysis
Umbrella Investigate positions both domains as low risk, both registered recently in Poland, and both hosted on the same IP:
Despite the low-risk score, the nameservers have high counts of malicious associated domains:
Targeting users in ASA, UK, and Nigeria:
Meraki analysis
Based on the time of the incident, we can trace the device’s location (based on its IP address). This is thanks to the effort we invested in mapping out the exact location of all Meraki APs, which we deployed across the convention center with an overlay of the event map covering the area of the event:
Further analysis and conclusions
The device name (LAPTOP-8MLGXXXXXX) and MAC address seen (f4:XX:XX:XX:XX:XX) both matched across the partner and Meraki, so there was no question that we were analyzing the same device.
Based on the useragent captured by the partner, the device type was an Apple iPhone. However, Meraki was reporting the Device and its OS as “Intel, Android”
A quick look up for the MAC address confirmed that the OUI (organizationally unique identifier) for f42679 was Intel Malaysia, making it unlikely that this was an Apple iPhone.
The description for the training “Web Hacking Black Belt Edition” can be seen here:
https://www.blackhat.com/asia-22/training/schedule/#web-hacking-black-belt-edition–day-25388
It is highly likely that the training content included the use of tools and techniques for spoofing the visibility of useragent or device type.
There is also a high probability that the two domains observed were used as part of the training activity, rather than this being part of a live attack.
It is clear that integrating the various Cisco technologies (Meraki wireless infrastructure, SecureX, Umbrella, Investigate) used in the investigation of this incident, together with the close partnership and collaboration of our NOC partners, positioned us where we needed to be and provided us with the tools we needed to swiftly collect the data, join the dots, make conclusions, and successfully bring the incident to closure.
Self Service with SecureX Orchestration and Slack by Matt Vander Horst
Overview
Since Meraki was a new platform for much of the NOC’s staff, we wanted to make information easier to gather and enable a certain amount of self-service. Since the Black Hat NOC uses Slack for messaging, we decided to create a Slack bot that NOC staff could use to interact with the Meraki infrastructure as well as Palo Alto Panorama using the SecureX Orchestration remote appliance. When users communicate with the bot, webhooks are sent to Cisco SecureX Orchestration to do the work on the back end and send the results back to the user.
Design
Here’s how this integration works:
Workflow #1: Handle Slash Commands
Slash commands are a special type of message built into Slack that allow users to interact with a bot. When a Slack user executes a slash command, the command and its arguments are sent to SecureX Orchestration where a workflow handles the command. The table below shows a summary of the slash commands our bot supported for Black Hat Asia 2022:
Here’s a sample of a portion of the SecureX Orchestration workflow that powers the above commands:
And here’s a sample of firewall logs as returned from the “/pan_traffic_history” command:
Workflow #2: Handle Interactivity
A more advanced form of user interaction comes in the form of Slack blocks. Instead of including a command’s arguments in the command itself, you can execute the command and Slack will present you with a form to complete, like this one for the “/update_vlan” command:
These forms are much more user friendly and allow information to be pre-populated for the user. In the example above, the user can simply select the switch to configure from a drop-down list instead of having to enter its name or serial number. When the user submits one of these forms, a webhook is sent to SecureX Orchestration to execute a workflow. The workflow takes the requested action and sends back a confirmation to the user:
Conclusion
While these two workflows only scratched the surface of what can be done with SecureX Orchestration webhooks and Slack, we now have a foundation that can be easily expanded upon going forward. We can add additional commands, new forms of interactivity, and continue to enable NOC staff to get the information they need and take necessary action. The goal of orchestration is to make life simpler, whether it is by automating our interactions with technology or making those interactions easier for the user.
Future Threat Vectors to Consider – Cloud App Discovery by Alejo Calaoagan
Since 2017 (starting in Black Hat USA – Las Vegas), Cisco Umbrella has provided DNS security to the Black Hat attendee network, added layers of traffic visibility previously not seen. Our efforts have largely been successful, identifying thousands of threats over the years and mitigating them via Umbrella’s blocking capabilities when necessary. This was taken a step further at Black Hat London 2021, where we introduced our Virtual Appliances to provide source IP attribution to the devices making requests.
Here at Black Hat Asia 2022, we’ve been noodling on additional ways to provide advanced protection for future shows, and it starts with Umbrella’s Cloud Application Discovery’s feature, which identified 2,286 unique applications accessed by users on the attendee network across the four-day conference. Looking at a snapshot from a single day of the show, Umbrella captured 572,282 DNS requests from all cloud apps, with over 42,000 posing either high or very high risk.
Digging deeper into the data, we see not only the types of apps being accessed…
…but also see the apps themselves…
…and we can flag apps that look suspicious.
We also include risk downs breaks by category…
…and drill downs on each.
While this data alone won’t provide enough information to take action, including this data in analysis, something we have been doing, may provide a window into new threat vectors that may have previously gone unseen. For example, if we identify a compromised device infected with malware or a device attempting to access things on the network that are restricted, we can dig deeper into the types of cloud apps those devices are using and correlate that data with suspicious request activity, potential uncovering tools we should be blocking in the future.
I can’t say for certain how much this extra data set will help us uncover new threats, but, with Black Hat USA just around the corner, we’ll find out soon.
Using SecureX sign-on to streamline access to the Cisco Stack at Black Hat by Adi Sankar
From five years ago to now, Cisco has tremendously expanded our presence at Black Hat to include a multitude of products. Of course, sign-on was simple when it was just one product (Secure Malware Analytics) and one user to log in. When it came time to add a new technology to the stack it was added separately as a standalone product with its own method of logging in. As the number of products increased, so did the number of Cisco staff at the conference to support these products. This means sharing usernames and passwords became tedious and not to mention insecure, especially with 15 Cisco staff, plus partners, accessing the platforms.
The Cisco Secure stack at Black Hat includes SecureX, Umbrella, Malware Analytics, Secure Endpoint (iOS clarity), and Meraki. All of these technologies support using SAML SSO natively with SecureX sign-on. This means that each of our Cisco staff members can have an individual SecureX sign-on account to log into the various consoles. This results in better role-based access control, better audit logging and an overall better login experience. With SecureX sign-on we can log into all the products only having to type a password one time and approve one Cisco DUO Multi-Factor Authentication (MFA) push.
How does this magic work behind the scenes? It’s actually rather simple to configure SSO for each of the Cisco technologies, since they all support SecureX sign-on natively. First and foremost, you must set up a new SecureX org by creating a SecureX sign-on account, creating a new organization and integrating at least one Cisco technology. In this case I created a new SecureX organization for Black Hat and added the Secure Endpoint module, Umbrella Module, Meraki Systems Manager module and the Secure Malware Analytics module. Then from Administration à Users in SecureX, I sent an invite to the Cisco staffers that would be attending the conference, which contained a link to create their account and join the Blackhat SecureX organization. Next let’s take a look at the individual product configurations.
Meraki:
In the Meraki organization settings enable SecureX sign-on. Then under Organization à Administrators add a new user and specify SecureX sign-on as the authentication method. Meraki even lets you limit users to particular networks and set permission levels for those networks. Accepting the email invitation is easy since the user should already be logged into their SecureX sign-on account. Now, logging into Meraki only requires an email address and no password or additional DUO push.
Umbrella:
Under Admin à Authentication configure SecureX sign-on which requires a test login to ensure you can still login before using SSO for authentication to Umbrella. There is no need to configure MFA in Umbrella since SecureX sign-on comes with built in DUO MFA. Existing users and any new users added in Umbrella under Admin à Accounts will now be using SecureX sign-on to login to Umbrella. Logging into Umbrella is now a seamless launch from the SecureX dashboard or from the SecureX ribbon in any of the other consoles.
Secure Malware Analytics:
A Secure Malware Analytics organization admin can create new users in their Threat Grid tenant. This username is unique to Malware Analytics, but it can be connected to a SecureX sign-on account to take advantage of the seamless login flow. From the email invitation the user will create a password for their Malware Analytics user and accept the EULA. Then in the top right under My Malware Analytics Account, the user has an option to connect their SecureX sign-on account which is a one click process if already signed in with SecureX sign-on. Now when a user navigates to Malware Analytics login page, simply clicking “Login with SecureX Sign-On” will grant them access to the console.
Secure Endpoint:
The Secure Endpoint deployment at Blackhat is limited to IOS clarity through Meraki Systems Manager for the conference IOS devices. Most of the asset information we need about the iPhones/iPads is brought in through the SecureX Device Insights inventory. However, for initial configuration and to view device trajectory it is required to log into Secure Endpoint. A new Secure Endpoint account can be created under Accounts à Users and an invite is sent to corresponding email address. Accepting the invite is a smooth process since the user is already signed in with SecureX sign-on. Privileges for the user in the Endpoint console can be granted from within the user account.
Conclusion:
To sum it all up, SecureX sign-on is the standard for the Cisco stack moving forward. With a new SecureX organization instantiated using SecureX sign-on any new users to the Cisco stack at Black Hat will be using SecureX sign-on. SecureX sign-on has helped our user management be much more secure as we have expanded our presence at Black Hat. SecureX sign-on provides a unified login mechanism for all the products and modernized our login experience at the conference.
Malware Threat Intelligence made easy and available, with Cisco Secure Malware Analytics and SecureX by Ben Greenbaum
I’d gotten used to people’s reactions upon seeing SecureX in use for the first time. A few times at Black Hat, a small audience gathered just to watch us effortlessly correlate data from multiple threat intelligence repositories and several security sensor networks in just a few clicks in a single interface for rapid sequencing of events and an intuitive understanding of security events, situations, causes, and consequences. You’ve already read about a few of these instances above. Here is just one example of SecureX automatically putting together a chronological history of observed network events detected by products from two vendors (Cisco Umbrella and NetWitness) . The participation of NetWitness in this and all of our other investigations was made possible by our open architecture, available APIs and API specifications, and the creation of the NetWitness module described above.
In addition to the traffic and online activities of hundreds of user devices on the network, we were responsible for monitoring a handful of Black Hat-owned devices as well. Secure X Device Insights made it easy to access information about these assets, either en masse or as required during an ongoing investigation. iOS Clarity for Secure Endpoint and Meraki System Manager both contributed to this useful tool which adds business intelligence and asset context to SecureX’s native event and threat intelligence, for more complete and more actionable security intelligence overall.
SecureX is made possible by dozens of integrations, each bringing their own unique information and capabilities. This time though, for me, the star of the SecureX show was our malware analysis engine, Cisco Secure Malware Analytics (CSMA). Shortly before Black Hat Asia, the CSMA team released a new version of their SecureX module. SecureX can now query CSMA’s database of malware behavior and activity, including all relevant indicators and observables, as an automated part of the regular process of any investigation performed in SecureX Threat Response.
This capability is most useful in two scenarios:
1: determining if suspicious domains, IPs and files reported by any other technology had been observed in the analysis of any of the millions of publicly submitted file samples, or our own.
2: rapidly gathering additional context about files submitted to the analysis engine by the integrated products in the Black Hat NOC.
The first was a significant time saver in several investigations. In the example below, we received an alert about connections to a suspicious domain. In that scenario, our first course of action is to investigate the domain and any other observables reported with it (typically the internal and public IPs included in the alert). Due to the new CSMA module, we immediately discovered that the domain had a history of being contacted by a variety of malware samples, from multiple families, and that information, corroborated by automatically gathered reputation information from multiple sources about each of those files, gave us an immediate next direction to investigate as we hunted for evidence of those files being present in network traffic or of any traffic to other C&C resources known to be used by those families. From the first alert to having a robust, data-driven set of related signals to look for, took only minutes, including from SecureX partner Recorded Future, who donated a full threat intelligence license for the Black Hat NOC.
The other scenario, investigating files submitted for analysis, came up less frequently but when it did, the CSMA/SecureX integration was equally impressive. We could rapidly, nearly immediately, look for evidence of any of our analyzed samples in the environment across all other deployed SecureX-compatible technologies. That evidence was no longer limited to searching for the hash itself, but included any of the network resources or dropped payloads associated with the sample as well, easily identifying local targets who had not perhaps seen the exact variant submitted, but who had nonetheless been in contact with that sample’s Command and Control infrastructure or other related artifacts.
And of course, thanks to the presence of the ribbon in the CSMA UI, we could be even more efficient and do this with multiple samples at once.
SecureX greatly increased the efficiency of our small volunteer team, and certainly made it possible for us to investigate more alerts and events, and hunt for more threats, all more thoroughly, than we would have been able to without it. SecureX truly took this team to the next level, by augmenting and operationalizing the tools and the staff that we had at our disposal.
We look forward to seeing you at Black Hat USA in Las Vegas, 6-11 August 2022!
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team: Aditya Sankar, Aldous Yeung, Alejo Calaoagan, Ben Greenbaum, Christian Clasen, Felix H Y Lam, George Dorsey, Humphrey Cheung, Ian Redden, Jeffrey Chua, Jeffry Handal, Jonny Noble, Matt Vander Horst, Paul Fidler and Steven Fan.
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially James Holland), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, James Pope, Steve Fink and Steve Oldenbourg).
About Black Hat
For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
Costa Rica’s national health service was hacked sometime earlier this morning by a Russian ransomware group known as Hive. The intrusion comes just weeks after Costa Rican President Rodrigo Chaves declared a state of emergency in response to a data ransom attack from a different Russian ransomware gang — Conti. Ransomware experts say there is good reason to believe the same cybercriminals are behind both attacks, and that Hive has been helping Conti rebrand and evade international sanctions targeting extortion payouts to cybercriminals operating in Russia.
The Costa Rican publication CRprensa.com reports that affected systems at the Costa Rican Social Security Fund (CCSS) were taken offline on the morning of May 31, but that the extent of the breach was still unclear. The CCSS is responsible for Costa Rica’s public health sector, and worker and employer contributions are mandated by law.
A hand-written sign posted outside a public health center in Costa Rica today explained that all systems are down until further notice (thanks to @Xyb3rb3nd3r for sharing this photo).
A hand-written notice posted outside a public health clinic today in Costa Rica warned of system outages due to a cyberattack on the nation’s healthcare systems. The message reads: “Dear Users: We would like to inform you that due a problem related to the information systems of the institution, we are unable to expend any medicine prescriptions today until further notice. Thank you for your understanding- The pharmacy.”
Esteban Jimenez, founder of the Costa Rican cybersecurity consultancy ATTI Cyber, told KrebsOnSecurity the CCSS suffered a cyber attack that compromised the Unique Digital Medical File (EDUS) and the National Prescriptions System for the public pharmacies, and as a result medical centers have turned to paper forms and manual contingencies.
“Many smaller health centers located in rural areas have been forced to close due to not having the required equipment or communication with their respective central health areas and the National Retirement Fund (IVM) was completely blocked,” Jimenez said. “Taking into account that salaries of around fifty thousand employees and deposits for retired citizens were due today, so now the payments are in danger.”
Jimenez said the head of the CCSS has addressed the local media, confirming that the Hive ransomware was deployed on at least 30 out of 1,500 government servers, and that any estimation of time to recovery remains unknown. He added that many printers within the government agency this morning began churning out copies of the Hive ransom note.
Printers at the Costa Rican government health ministry went crazy this morning after the Hive ransomware group attacked. Image: Esteban Jimenez.
“HIVE has not yet released their ransom fee but attacks are expected to follow, other organizations are trying to get a hold on the emergency declaration to obtain additional funds to purchase new pieces of infrastructure, improve their backup structure amongst others,” Jimenez said.
A copy of the ransom note left behind by the intruders and subsequently uploaded to Virustotal.com indicates the CCSS intrusion was the work of Hive, which typically demands payment for a digital key needed to unlock files and servers compromised by the group’s ransomware.
A HIVE ransomware chat page for a specific victim (redacted).
On May 8, President Chaves used his first day in office to declare a national state of emergency after the Conti ransomware group threatened to publish gigabytes of sensitive data stolen from Costa Rica’s Ministry of Finance and other government agencies. Conti initially demanded $10 million, and later doubled the amount when Costa Rica refused to pay. On May 20, Conti leaked more than 670 gigabytes of data taken from Costa Rican government servers.
As CyberScoop reported on May 17, Chaves told local media he believed that collaborators within Costa Rica were helping Conti extort the government. Chaves offered no information to support this claim, but the timeline of Conti’s descent on Costa Rica is worth examining.
Most of Conti’s public communications about the Costa Rica attack have very clearly assigned credit for the intrusion to an individual or group calling itself “unc1756.” In March 2022, a new user by the same name registered on the Russian language crime forum Exploit.
A message Conti posted to its dark web blog on May 20.
On the evening of April 18, Costa Rica’s Ministry of Finance disclosed the Conti intrusion via Twitter. Earlier that same day, the user unc1756 posted a help wanted ad on Exploit saying they were looking to buy access to “special networks” in Costa Rica.
“By special networks I mean something like Haciendas,” unc1756 wrote on Exploit. Costa Rica’s Ministry of Finance is known in Spanish as the “Ministerio Hacienda de Costa Rica.” Unc1756 said they would pay $USD 500 or more for such access, and would work only with Russian-speaking people.
Experts say there are clues to suggest Conti and Hive are working together in their attacks on Costa Rica, and that the intrusions are tied to a rebranding effort by Conti. Shortly after Russia invaded Ukraine at the end of February, Conti declared its full support, aligning itself directly with Russia and against anyone who would stand against the motherland.
Conti’s threatening message this week regarding international interference in Ukraine.
Conti quickly deleted the declaration from its website, but the damage had already been done, and any favor or esteem that Conti had earned among the Ukrainian cybercriminal underground effectively evaporated overnight.
Shortly thereafter, a Ukrainian security expert leaked many months worth of internal chat records between Conti personnel as they plotted and executed attacks against hundreds of victim organizations. Those candid messages exposed what it’s like to work for Conti, how they undermined the security of their targets, as well as how the group’s leaders strategized for the upper hand in ransom negotiations.
But Conti’s declaration of solidarity with the Kremlin also made it increasingly ineffective as an instrument of financial extortion. According to cyber intelligence firm ADVIntel, Conti’s alliance with the Russian state soon left it largely unable to receive ransom payments because victim companies are being advised that paying a Conti ransom demand could mean violating U.S. economic sanctions on Russia.
“Conti as a brand became associated with the Russian state — a state that is currently undergoing extreme sanctions,” ADVIntel wrote in a lengthy analysis (PDF). “In the eyes of the state, each ransom payment going to Conti may have potentially gone to an individual under sanction, turning simple data extortion into a violation of OFAC regulation and sanction policies against Russia.”
Conti is by far the most aggressive and profitable ransomware group in operation today. Image: Chainalysis
ADVIntel says it first learned of Conti’s intrusion into Costa Rican government systems on April 14, and that it has seen internal Conti communications indicating that getting paid in the Costa Rica attack was not the goal.
Rather, ADVIntel argues, Conti was simply using it as a way to appear publicly that it was still operating as the world’s most lucrative ransomware collective, when in reality the core Conti leadership was busy dismantling the crime group and folding themselves and top affiliates into other ransomware groups that are already on friendly terms with Conti.
“The only goal Conti had wanted to meet with this final attack was to use the platform as a tool of publicity, performing their own death and subsequent rebirth in the most plausible way it could have been conceived,” ADVIntel concluded.
ADVIntel says Conti’s leaders and core affiliates are dispersing to several Conti-loyal crime collectives that use either ransomware lockers or strictly engage in data theft for ransom, including AlphV/BlackCat, AvosLocker, BlackByte, HelloKitty, Hive, and Karakurt.
Still, Hive appears to be perhaps the biggest beneficiary of any attrition from Conti: Twice over the past week, both Conti and Hive claimed responsibility for hacking the same companies. When the discrepancy was called out on Twitter, Hive updated its website to claim it was not affiliated with Conti.
Conti and Hive’s Costa Rican exploits mark the latest in a string of recent cyberattacks against government targets across Latin America. Around the same time it hacked Costa Rica in April, Conti announced it had hacked Peru’s National Directorate of Intelligence, threatening to publish sensitive stolen data if the government did not pay a ransom.
But Conti and Hive are not alone in targeting Latin American victims of late. According to data gathered from the victim shaming blogs maintained by multiple ransomware groups, over the past 90 days ransom actors have hacked and sought to extort 15 government agencies in Brazil, nine in Argentina, six in Colombia, four in Ecuador and three in Chile.
A recent report (PDF) by the Inter-American Development Bank suggests many Latin American countries lack the technical expertise or cybercrime laws to deal with today’s threats and threat actors.
“This study shows that the Latin American and Caribbean (LAC) region is not sufficiently prepared to handle cyberattacks,” the IADB document explains. “Only 7 of the 32 countries studied have a critical infrastructure protection plan, while 20 have established cybersecurity incident response teams, often called CERTs or CSIRTs. This limits their ability to identify and respond to attacks.”
A 33-year-old Illinois man was sentenced to two years in prison today following his conviction last year for operating services that allowed paying customers to launch powerful distributed denial-of-service (DDoS) attacks against hundreds of thousands of Internet users and websites.
The user interface for Downthem[.]org.
Matthew Gatrel of St. Charles, Ill. was found guilty for violations of the Computer Fraud and Abuse Act (CFAA) related to his operation of downthem[.]org and ampnode[.]com, two DDoS-for-hire services that had thousands of customers who paid to launch more than 200,000 attacks.
Despite admitting to FBI agents that he ran these so-called “booter” services (and turning over plenty of incriminating evidence in the process), Gatrel opted to take his case to trial, defended the entire time by public defenders. Gatrel’s co-defendant and partner in the business, Juan “Severon” Martinez of Pasadena, Calif., pleaded guilty just before the trial.
After a nine-day trial in the Central District of California, Gatrel was convicted on all three counts, including conspiracy to commit unauthorized impairment of a protected computer, conspiracy to commit wire fraud, and unauthorized impairment of a protected computer.
Prosecutors said Downthem sold subscriptions allowing customers to launch DDoS attacks, while AmpNode provided “bulletproof” server hosting to customers — with an emphasis on “spoofing” servers that could be pre-configured with DDoS attack scripts and lists of vulnerable “attack amplifiers” used to launch simultaneous cyberattacks on victims.
Booter and stresser services let customers pick from among a variety of attack methods, but almost universally the most powerful of these methods involves what’s known as a “reflective amplification attack.” In such assaults, the perpetrators leverage unmanaged Domain Name Servers (DNS) or other devices on the Web to create huge traffic floods.
Ideally, DNS servers only provide services to machines within a trusted domain — such as translating an Internet address from a series of numbers into a domain name, like example.com. But DNS reflection attacks rely on consumer and business routers and other devices equipped with DNS servers that are (mis)configured to accept queries from anywhere on the Web.
Attackers can send spoofed DNS queries to these DNS servers, forging the request so that it appears to come from the target’s network. That way, when the DNS servers respond, they reply to the spoofed (target) address.
The bad guys also can amplify a reflective attack by crafting DNS queries so that the responses are much bigger than the requests. For example, an attacker could compose a DNS request of less than 100 bytes, prompting a response that is 60-70 times as large. This “amplification” effect is especially pronounced if the perpetrators query dozens of DNS servers with these spoofed requests simultaneously.
The government charged that Gatrel and Martinez constantly scanned the Internet for these misconfigured devices, and then sold lists of Internet addresses tied to these devices to other booter service operators.
“Gatrel ran a criminal enterprise designed around launching hundreds of thousands of cyber-attacks on behalf of hundreds of customers,” prosecutors wrote in a memorandum submitted in advance of his sentencing. “He also provided infrastructure and resources for other cybercriminals to run their own businesses launching these same kinds of attacks. These attacks victimized wide swaths of American society and compromised computers around the world.”
The U.S. and United Kingdom have been trying to impress on would-be customers of these booter services that hiring them for DDoS attacks is illegal. The U.K. has even taken out Google ads to remind U.K. residents when they search online for terms common to booter services.
The case against Gatrel and Martinez was brought as part of a widespread crackdown on booter services in 2018, when the FBI joined law enforcement partners overseas to seize 15 different booter service domains.
Those actions have prompted a flurry of prosecutions, with wildly varying sentences when the booter service owners are invariably found guilty. However, DDoS experts say booter and stresser services that remain in operation continue to account for the vast majority of DDoS attacks launched daily around the globe.
Microsoft on Tuesday released software updates to fix 60 security vulnerabilities in its Windows operating systems and other software, including a zero-day flaw in all supported Microsoft Office versions on all flavors of Windows that’s seen active exploitation for at least two months now. On a lighter note, Microsoft is officially retiring its Internet Explorer (IE) web browser, which turns 27 years old this year.
Three of the bugs tackled this month earned Microsoft’s most dire “critical” label, meaning they can be exploited remotely by malware or miscreants to seize complete control over a vulnerable system. On top of the critical heap this month is CVE-2022-30190, a vulnerability in the Microsoft Support Diagnostics Tool (MSDT), a service built into Windows.
Dubbed “Follina,” the flaw became public knowledge on May 27, when a security researcher tweeted about a malicious Word document that had surprisingly low detection rates by antivirus products. Researchers soon learned that the malicious document was using a feature in Word to retrieve a HTML file from a remote server, and that HTML file in turn used MSDT to load code and execute PowerShell commands.
“What makes this new MS Word vulnerability unique is the fact that there are no macros exploited in this attack,” writes Mayuresh Dani, manager of threat research at Qualys. “Most malicious Word documents leverage the macro feature of the software to deliver their malicious payload. As a result, normal macro-based scanning methods will not work to detect Follina. All an attacker needs to do is lure a targeted user to download a Microsoft document or view an HTML file embedded with the malicious code.”
Kevin Beaumont, the researcher who gave Follina its name, penned a fairly damning account and timeline of Microsoft’s response to being alerted about the weakness. Beaumont says researchers in March 2021 told Microsoft they were able achieve the same exploit using Microsoft Teams as an example, and that Microsoft silently fixed the issue in Teams but did not patch MSDT in Windows or the attack vector in Microsoft Office.
Beaumont said other researchers on April 12, 2022 told Microsoft about active exploitation of the MSDT flaw, but Microsoft closed the ticket saying it wasn’t a security issue. Microsoft finally issued a CVE for the problem on May 30, the same day it released recommendations on how to mitigate the threat from the vulnerability.
Microsoft also is taking flak from security experts regarding a different set of flaws in its Azure cloud hosting platform. Orca Security said that back on January 4 it told Microsoft about a critical bug in Azure’s Synapse service that allowed attackers to obtain credentials to other workspaces, execute code, or leak customer credentials to data sources outside of Azure.
In an update to their research published Tuesday, Orca researchers said they were able to bypass Microsoft’s fix for the issue twice before the company put a working fix in place.
“In previous cases, vulnerabilities were fixed by the cloud providers within a few days of our disclosure to the affected vendor,” wrote Orca’s Avi Shua. “Based on our understanding of the architecture of the service, and our repeated bypasses of fixes, we think that the architecture contains underlying weaknesses that should be addressed with a more robust tenant separation mechanism. Until a better solution is implemented, we advise that all customers assess their usage of the service and refrain from storing sensitive data or keys in it.”
Amit Yoran, CEO of Tenable and a former U.S. cybersecurity czar, took Microsoft to task for silently patching an issue Tenable reported in the same Azure Synapse service.
“It was only after being told that we were going to go public, that their story changed…89 days after the initial vulnerability notification…when they privately acknowledged the severity of the security issue,” Yoran wrote in a post on LinkedIn. “To date, Microsoft customers have not been notified. Without timely and detailed disclosures, customers have no idea if they were, or are, vulnerable to attack…or if they fell victim to attack prior to a vulnerability being patched. And not notifying customers denies them the opportunity to look for evidence that they were or were not compromised, a grossly irresponsible policy.”
Also in the critical and notable stack this month is CVE-2022-30136, which is a remote code execution flaw in the Windows Network File System (NFS version 4.1) that earned a CVSS score of 9.8 (10 being the worst). Microsoft issued a very similar patch last month for vulnerabilities in NFS versions 2 and 3.
“This vulnerability could allow a remote attacker to execute privileged code on affected systems running NFS. On the surface, the only difference between the patches is that this month’s update fixes a bug in NFSV4.1, whereas last month’s bug only affected versions NSFV2.0 and NSFV3.0,” wrote Trend Micro’s Zero Day Initiative. “It’s not clear if this is a variant or a failed patch or a completely new issue. Regardless, enterprises running NFS should prioritize testing and deploying this fix.”
Beginning today, Microsoft will officially stop supporting most versions of its Internet Explorer Web browser, which was launched in August 1995. The IE desktop application will be disabled, and Windows users who wish to stick with a Microsoft browser are encouraged to move to Microsoft Edge with IE mode, which will be supported through at least 2029.
For a closer look at the patches released by Microsoft today and indexed by severity and other metrics, check out the always-useful Patch Tuesday roundup from the SANS Internet Storm Center. And it’s not a bad idea to hold off updating for a few days until Microsoft works out any kinks in the updates: AskWoody.com usually has the dirt on any patches that may be causing problems for Windows users.
As always, please consider backing up your system or at least your important documents and data before applying system updates. And if you run into any problems with these updates, please drop a note about it here in the comments.
FreshRSS 