Login
In part one of this issue of our Black Hat USA NOC (Network Operations Center) blog, you will find:
Adapt and Overcome, by Jessica Bair Oppenheimer
In technology, we plan as best as we can, execute tactically with the resources and knowledge we have at the time, focus on the strategic mission, adjust as the circumstances require, collaborate, and improve; with transparency and humility. In short, we adapt and we overcome. This is the only way a community can have trust and grow, together. Every deployment comes with its challenges and Black Hat USA 2022 was no exception. Looking at the three Ps (people, process, platform), flexibility, communication, and an awesome Cisco platform allowed us to build and roll with the changes and challenges in the network. I am proud of the Cisco Meraki and Secure team members and our NOC partners.
The Buck Stops Here. Full stop. I heard a comment that the Wi-Fi service in the Expo Hall was “the worst I’ve ever experienced at a conference.” There were a lot of complaints about the Black Hat USA 2022 Wi-Fi network in the Expo Hall on 10 August. I also heard a lot of compliments about the network. Despite that the Wi-Fi and wired network was generally very good the most of the conference, and before my awesome colleagues share the many successes of designing, building, securing, managing, automating and tearing down one of the most hostile networks on Earth; I want to address where and how we adapted and what we did to fix the issues that arose, as we built an evolving, enterprise class network in a week.
First, a little history of how Cisco came to be the Official Network Provider of Black Hat USA 2022, after we were already successfully serving as the Official Mobile Device Management, Malware Analysis and Domain Name Service Provider. An Official Provider, as a Premium Partner, is not a sponsorship and no company can buy their way into the NOC for any amount of money. From the beginning of Black Hat 25 years ago, volunteers built the network for the conference rather than using the hotel network. This continues today, with the staff of Black Hat hand selecting trusted partners to build and secure the network.
After stepping up to help Black Hat with the network at Black Hat Asia, we had only two and a half months until Black Hat USA, in Las Vegas, 6-11 August 2022. Cisco was invited to build and secure the network for the much larger Black Hat USA flagship conference, affectionally known as ‘Hacker Summer Camp’, as the Official Network Equipment Provider. There were few other options, given the short timeframe to plan, supply chain difficulties in procuring the networking gear and assembling a team of network engineers, to join the Cisco Secure engineers and threat hunters. All the work, effort and loaned equipment were a gift from Cisco Meraki and Cisco Secure to the community.
We were proud to collaborate with NOC partners Gigamon, IronNet, Lumen, NetWitness and Palo Alto Networks; and work with Neil ‘Grifter’ Wyler, Bart Stump, Steve Fink and James Pope of Black Hat. We built strong bonds of familial ties over the years of challenges and joint successes. I encourage you to watch the replay of the Black Hat session An Inside Look at Defending the Black Hat Network with Bart and Grifter.
In June 2022, adjacent to Cisco Live Americas, the NOC partners met with Black Hat to plan the network. Cisco Meraki already donated 45 access points (APs), seven MS switches, and two Meraki MX security and SD-WAN appliances to Black Hat, for regional conferences.
I looked at the equipment list from 2019, that was documented in the Bart and Grifter presentation, and estimated we needed to source an additional 150 Cisco Meraki MR AP (with brackets and tripods) and 70+ Cisco Meraki MS switches to build the Black Hat USA network in just a few weeks. I wanted to be prepared for any changes or new requirements on-site. We turned to JW McIntire, who leads the network operations for Cisco Live and Cisco Impact. JW was enthusiastically supportive in helping identify the equipment within the Cisco Global Events inventory and giving his approval to utilize the equipment. A full thanks to those who made this possible is in the Acknowledgements below.
Over the week-long conference, we used all but three of the switches and all the APs.
We worked off the draft floor plans from 13 June 2022, for the training rooms, briefing rooms, support rooms, keynote rooms, conference public areas, registration, and of course the Expo Hall: over two million square feet of venue. We received updated plans for the training rooms, Expo Hall and support needs 12 days before we arrived on site. There were about 60 training rooms planned, each requiring their own SSID and Virtual Local Area Network, without host isolation. The ‘most access possible’ was the requirement, to use real world malware and attacks, without attacking other classrooms, attendees, sponsors or the rest of the world. Many of the training rooms changed again nine days before the start of the network build, as the number confirmed students rose or fell, we adjusted the AP assignments.
For switching allocation, we could not plan until we arrived onsite, to assess the conference needs and the placement of the cables in the walls of the conference center. The Black Hat USA network requires that every switch be replaced, so we always have full control of the network. Every network drop to place an AP and put the other end of a cable into the new switches in the closets costs Black Hat a lot of money. It also requires the time of ‘Doc’ – the lead network engineer at the Mandalay Bay, to whom we are all deeply grateful.
The most important mission of the NOC is Access, then Security, Visibility, Automation, etc. People pay thousands of dollars to attend the trainings and the briefings; and sponsors pay tens of thousands for their booth space. They need Access to have a successful conference experience.
With that background, let’s discuss the Wi-Fi in the Expo Hall. Cisco has a service to help customers do a methodical predictive survey of their space for the best allocation of their resources. We had 74 of the modern MR57 APs for the conference and prioritized their assignment in the Expo Hall and Registration. Specifications for MR57s include a 6 GHz 4×4:4, 5 GHz 4×4:4 and 2.4 GHz 4×4:4 radio to offer a combined tri–radio aggregate frame rate of 8.35 Gbps, with up to 4,804 Mbps in 6GHz band, 2,402 Mbps 5 GHz band and, 1,147 Mbps / 574 Mbps in the 2.4 GHz band based on 40MHz / 20MHz configuration. Technologies like transmit beamforming and enhanced receive sensitivity allow the MR57 to support a higher client density than typical enterprise-class access points, resulting in better performance for more clients, from each AP.
We donated top of the line gear for use at Black Hat USA. So, what went wrong on the first day in the Expo Hall? The survey came back with the following map and suggestions of 34 MR57s in the locations below. Many assumptions were made in pre-planning, since we did not know the shapes, sizes and materials of the booths that would be present inside the allocated spaces. We added an AP in the Arsenal Lab on the far-left side, after discussing the needs with Black Hat NOC leadership.
In the Entrance area (Bayside Foyer) of the Expo Hall (bottom of the map), you can see that coverage drops. There were four MR57s placed in the Bayside Foyer for iPad Registration and attendee Wi-Fi, so they could access their emails and obtain their QR code for scanning and badge printing.
I believed that would be sufficient and we allocated other APs to the rest of the conference areas. We had positive reports on coverage in most areas of the rest of the conference. When there were reported issues, we quickly deployed Cisco Meraki engineers or NOC technical associates. to confirm and were able to make changes in radio strength, broadcasting bands, SSIDs, etc. to fine tune the network. All while managing a large amount of new or changing network requirements, as the show expanded due to its success and was fully hybrid, with the increased streaming of the sponsored sessions, briefings and keynotes and remote Registration areas in hotels.
As the attendees queued up in mass outside of the Expo Hall on the morning of 10 August, the number of attendee devices connecting to the four MR57s in the foyer grew into the thousands. This degraded the performance of the Registration network. We adjusted by making the APs closest to the registration iPads only dedicated to the Registration. This fixed Registration lag but reduced the performance of the network for the attendees, as they waited to rush into the Expo Hall. From the site survey map, it is clear that the replacement APs were now needed in the Entrance for a connected mesh network, as you entered the Expo Hall from the Bayside foyer. Here lies Lesson 1: expected people flow should be taken into account in the RF design process.
Another challenge the morning of the Expo Hall opening was that five of the 57MRs inside were not yet connected to the Internet when it opened at 10am. The APs were installed three days earlier, then placed up on tripods the afternoon prior. However, the volume of newly requested network additions, to support the expanded hybrid element required the deployment of extra cables and switches. This cascaded down and delayed the conference center team from finalizing the Expo Hall line drops until into the afternoon. Lesson 2: Layer 1 is still king; without it, no Wi-Fi or power.
A major concern for the sponsors in their booths was that as the Expo Hall filled with excited attendees, the connectivity of the 900+ iOS devices used for lead management dropped. Part of this congestion was thousands of 2.4Ghz devices connected to the Expo Hall network. We monitored this and pushed as many as possible to 5Ghz, to relieve pressure on those airwaves. Lesson 3: With Wi-Fi 6e now available in certain countries, clean spectrum awaits, but our devices need to come along as well.
We also adjusted in the Cisco Meraki Systems Manager Mobile Device Management, to allow the iPhones for scanning to connect securely to the Mandalay Bay conference network, while still protecting your personal information with Cisco SecureX, Security Connector and Umbrella DNS, to ensure access as we expanded the network capacity in the Expo Hall. Lesson 4: Extreme security by default where you can control the end point. Do not compromise when dealing with PPI.
Using the Cisco Meraki dashboard access point location heat map and the health status of the network, we identified three places in the front of the Expo Hall to deploy additional drops with the Mandalay Bay network team. Since adding network drops takes some time (and costs Black Hat extra money), we took immediate steps to deploy more MS120 switches and eight additional APs at hot spots inside the Expo Hall with the densest client traffic, at no expense to Black Hat. Lesson 5: Footfall is not only about sales analytics. It does play a role into RF planning. Thereby, allowing for a data-driven design decision.
Above is the heat map of the conference Expo Hall at noon on 11 August. You can see the extra APs at the Entrance of the Expo Hall, connected by the three drops set up by the Mandalay Bay to the Cisco Meraki switches in the closets. Also, you can see the clusters of APs connected to the extra MS120 switches. At the same time, our lead Meraki engineer, Evan Basta, did a speed test from the center left of the Expo Hall.
As I am sharing lessons learned, I want to provide visibility to another situation encountered. On the afternoon of 9 August, the last day of training, a Black Hat attendee walked the hallways outside several training rooms and deliberately attacked the network, causing students and instructors not to be able to connect to their classes. The training rooms have host isolation removed and we designed the network to provide as much safe access as possible. The attacker took advantage of this openness, spoofed the SSIDs of the many training rooms and launched malicious attacks against the network.
We must allow real malware on the network for training, demonstrations and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors from using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities and malicious websites.
The attack vector was identified by a joint investigation of the NOC teams, initiated by the Cisco Meraki Air Marshal review. Note the exact same MAC addresses of the spoofed SSIDs and malicious broadcasts. A network protection measure was suggested by the Cisco Meraki engineering team to the NOC leadership. Permission was granted to test on one classroom, to confirm it stopped the attack, while not also disrupting the training. Lesson 6: The network-as-a-sensor will help mitigate issues but will not fix the human element.
Once confirmed, the measure was implemented network wide to return resiliency and access. The NOC team continued the investigation on the spoofed MAC addresses, using syslogs, firewall logs, etc. and identified the likely app and device used. An automated security alerting workflow was put in place to quickly identify if the attacker resumed/returned, so physical security could also intervene to revoke the badge and eject the attacker from the conference for violation of the Black Hat code of conduct.
I am grateful to the 20+ Cisco engineers, plus Talos Threat Hunters, deployed to the Mandalay Bay Convention Center, from the United States, Canada, Qatar and United Kingdom who made the Cisco contributions to the Black Hat USA 2022 NOC possible. I hope you will read on, to learn more lessons learned about the network and the part two blog about Cisco Secure in the NOC
Building the Hacker Summer Camp Network, by Evan Basta
It was the challenge of my career to take on the role of the lead network engineer for Black Hat USA. The lead engineer, who I replaced, was unable to travel from Singapore, just notifying us two weeks before we were scheduled to deploy to Las Vegas.
We prepared as much as possible before arrival, using the floor plans and the inventory of equipment that was ordered and on its way from the warehouse. We met with the Black Hat NOC leadership, partners and Mandalay Bay network engineers weekly on conference calls, adjusted what we could and then went to Black Hat, ready for a rapidly changing environment.
Our team was able to remain flexible and meet all the Black Hat requests that came in, thanks to the ability of the Cisco Meraki dashboard to manage the APs and switches from the cloud. Often, we were configuring the AP or switch as it was being transported to the location of the new network segment, laptop in hand.
For the construction of the Black Hat network, let’s start with availability. Registration and training rooms had priority for connectivity. iPads and iPhones needed secure connectivity to scan QR codes of registering attendees. Badge printers needed hardline access to the registration system. Training rooms all needed their separate wireless networks, for a safe sandbox for network defense and attack. Thousands of attendees attended, ready to download and upload terabytes of data through the main conference wireless network. All the keynotes, briefings and sponsored sessions needed to be recorded and streamed. Below are all the APs stacked up for assignment, including those assigned to the Expo Hall in the foreground.
All this connectivity was provided by Cisco Meraki access points and switches along with integrations into SecureX, Umbrella, and other Cisco platforms. We fielded a literal army of engineers to stand up the network in six days.
Let’s talk security and visibility. For a few days, the Black Hat network is one of the most hostile in the world. Attendees learn new exploits, download new tools, and are encouraged to test them out. Being able to drill down on attendee connection details and traffic was instrumental in ensuring attendees followed the Black Hat code of conduct.
On the wireless front, we made extensive use of our Radio Profiles to reduce interference by tuning power and channel settings. We enabled band steering to get more clients on the 5GHz bands versus 2.4GHz and watched the Location Heatmap like a hawk looking for hotspots and dead areas. Handling the barrage of wireless change requests – enable or disabling this SSID, moving VLANs (Virtual Local Area Networks), enabling tunneling for host isolation on the general conference Wi-Fi, mitigating attacks – was a snap with the Cisco Meraki Dashboard.
Floor Plan and Location Heatmap
On the first day of NOC setup, the Cisco team worked with the Mandalay Bay networking engineers to deploy core switches and map out the switches for the closets, according to the number of cables coming in from the training and briefing rooms. The floor plans in PDF were uploaded into the Meraki Dashboard; and with a little fine tuning, aligned perfectly with the Google Map.
Cisco Meraki APs were then placed physically in the venue meeting and training rooms. Having the APs named, as mentioned above, made this an easy task. This enabled accurate heatmap capability.
The Location Heatmap provided the capability to drill into the four levels of the conference, including the Expo Hall, lower level (North Conference Center), 2nd Floor and 3rd Floor. Below is the view of the entire conference.
Network Visibility
We were able to monitor the number of connected clients, network usage, the people passing by the network and location analytics, throughout the conference days. We provided visibility access to the Black Hat NOC management and the technology partners, along with full API (Application Programming Interface) access, so they could integrate with the network platform.
Alerts
Cisco Meraki alerts provide notification when something happens in the Dashboard. Default behavior is to be emailed when something happens. Obviously, emails got lost in the noise, at Black Hat Asia 2022, we made a web hook in Cisco SecureX orchestration to be able to consume Cisco Meraki alerts and send it to Slack (the messaging platform within the Black Hat NOC), using the native template in the Cisco Meraki Dashboard.
The alert kicked off if an AP or a switch lost connectivity. At Black Hat USA, we modified this to text alerts, as these were a priority. In the following example, we knew at the audio-visual team unplugged a switch to move it and were able to deploy technical associates from the NOC to ensure it was reconnected properly.
The Cisco Stack’s Potential in Action, by Paul Fidler
As we planned for Black Hat USA, the number of iOS devices to manage and protect rose from 300+ to over 900, and finally over 1,000.
The first amongst these was the use of the Cisco Meraki API. We were able to import the list of MAC addresses of the Cisco Meraki APs, to ensure that the APs were named appropriately and tagged, using a single source of truth document shared with the NOC management and partners, with the ability to update en masse at any time. Over three quarters of the AP configuration was able to be completed before arriving on site.
Meraki Systems Manager – Initial device enrollment and provisioning
We’ll start with the positive: When it comes to creating the design to manage X number of devices, it doesn’t matter if it’s 10 devices, or 10,000… And this was certainly true for Black Hat. The requirements were straightforward:
All this configuration was done ahead of time in the Meraki Dashboard, almost a month before the conference.
Now the negatives: Of all the events that the company who supplies the devices attends; Black Hat is the only one where devices are managed. Using mass deployment techniques like Apple’s Automated Device Enrollment, therefore, is not used. The company pre-stages the devices using Apple Configurator, which allows for both Supervision and Enrollment.
It became more difficult: Whilst the pre-staged devices were fine (other than having to handle all 1,000+ devices to turn Wi-Fi to Autojoin and opening the Meraki Systems Manager app [to give us Jailbreak and Location visibility]), an extra 100 devices were supplied that were not enrolled. As these devices were enrolled elsewhere from the prior Black Hat conferences, a team of around 10 people pitched in to restore each device, adding the Wi-Fi profile and then enrollment.
Fortunately, Apple Configurator can create Blueprints:
A Blueprint is essential a list of actions, in a particular order, that Apple Configurator can run through autonomously
But why did it need a team of ten? There were several limitations:
However, the task was completed in around three hours, given the limitations! If there’s one lesson to learn from this: Use Apple’s Automated Device Enrollment.
Command vs Profile
One of the slight nuances of Apple Mobile Device Manager is the difference between a ‘command’ and ‘profile’. Within the Meraki Systems Manager dashboard, we don’t highlight the difference between the two. But it’s important to know. A ‘profile’ is something that remains on the device: If there’s a state change on the device, or the user attempts something, the profile is always on there. However, a ‘command’ is exactly that: It’s sent once, and if something changes in the future, then the command won’t have any effect.
So, why is this highlighted here? Well, in some instances, some apps weren’t pushed successfully: You’d see them on the device, but with a cloud icon next to them. The only way to resolve this would be to remove the app, and then repost it. But we were also using a Homepage Layout, which put various apps on various pages. Pushing the app would result in it appearing on the wrong page. To ensure a consistent user experience, we would push the homepage profile again to devices to take effect.
Meraki BSSID Geolocation
We’ve mentioned this before in past Black Hat events, but, given the scale of The Mandalay Bay, it’s important to circle back to this. GPS is notoriously unreliable in conference centers like this, but it was still important to know where devices are. Because we’d ensured the correct placement of the Access Points on the floor plan, and because Systems Manager was in the same organisation, it ensured that the devices reported their location accurately! If one were to ‘walk’ we could wipe it remotely to protect your personal details.
Protection of PPI (Protected Private Information)
When the conference Registration closed on the last day and the Business Hall Sponsors all returned their iPhones, we were able to remotely wipe all the devices, removing all attendee data, prior to returning to the device contractor.
APIs
As mentioned elsewhere in this blog, this was a conference of APIs. Just the sheer scale of the conference resulted in the use of APIs. Various API projects included:
From a Systems Manager perspective, there were:
Port Security, by Ryan MacLennan, Ian Redden and Paul Fidler
During the Cisco Meraki deployment, we had a requirement to shutdown ports as they went inactive to prevent malicious actors from removing an official device and plugging in theirs. This ability is not directly built into the Cisco Meraki dashboard, so we built a workflow for the Black Hat customer, using the Cisco Meraki API. To achieve this, we created a small python script that was hosted as an AWS (Amazon Web Services) Lambda function and listened for webhooks from the Cisco Meraki Dashboard when a port went down. Initially this did solve our issue, but it was not fast enough, about five minutes from the time the port went down/a cable was unplugged. This proof of concept laid the groundwork to make the system better. We migrated from using a webhook in the Cisco Meraki Dashboard to using syslogs. We also moved the script from Lambda to a local server. Now, a python script was scanning for syslogs from the switches and when it saw a port down log, it will immediately call out to the locally hosted python script that calls out to the Cisco Meraki API and disabled the port.
This challenge had many setbacks and iterations while it was being built. Before we settled on listening for syslogs, we tried using SNMP polling. After figuring out the information we needed to use, we found that trying to poll SNMP would not work because SNMP would not report the port being down if the switch to another device was fast enough. This led us to believe we might not be able to do what we needed in a timely manner. After some deliberation with fellow NOC members, we started working on a script to listen for the port down syslogs. This became the best solution and provided immediate results. The ports would be disabled within milliseconds of going downThe diagram below shows an example of what will happen: If the Workshop Trainer’s device is un-plugged and a Threat Actor tries to plug into their port, a syslog is sent from the Cisco Meraki switch to our internal server hosting the python listener. Once the python script gets the request, it sends an API call to the Cisco Meraki API gateway and the Cisco Meraki cloud then tells the switch to disable the port that went down very briefly.
However, what was apparent was that the script was working TOO well! As discussed, several times already in this blog, the needs of the conference were very dynamic, changing on a minute-by-minute basis. This was certainly true in Registration and with the Audio-Visual teams. We discovered quickly that legitimate devices were being unplugged and plugged in to various ports, even if just temporarily. Of course, the script was so quick that it disabled ports before the users in registration knew what was happening. This resulted in NOC staff having to re-enable ports. So, more development was done. The task? For a given network tag, show the status of all the ports of all the switches. Given the number of switches at the conference, tags were used to reduce the amount of data being brought back, so it was easier to read and manage.
Mapping Meraki Location Data with Python, by Christian Clausen
In the blog post we published after Black Hat Asia 2022, we provided details on how to collect Bluetooth and Wi-Fi scanning data from a Meraki organization, for long-term storage and analysis. This augmented the location data provided by the Meraki dashboard, which is limited to 24-hours. Of course, the Meraki dashboard does more than just provide location data based on Wi-Fi and Bluetooth scanning from the access points. It also provides a neat heatmap generated from this data. We decided to take our long-term data project a step further and see if we could generate our own heatmap based on the data collected from the Meraki Scanning API.
The Folium Python library “builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the leaflet.js library” to provide all kinds of useful mapping functions. We can take location data (longitude and latitude) and plot them on lots of built-in map tiles from the likes of OpenStreetMap, MapBox, Stamen, and more. Among the available Folium plugins is a class called “HeatMapWithTime.” We can use this to plot our Meraki location data and have the resulting map animate the client’s movements.
Step 1: Collect the data
During the previous conference, we used a Docker container containing a couple Flask endpoints connected via ngrok to collect the large amount of data coming from Meraki. We re-used the same application stack this time around, but moved it out from behind ngrok into our own DMZ with a public domain and TLS (Transport Layer Security) certificate, to avoid any bandwidth limitations. We ended up with over 40GB of JSON data for the conference week to give to Black Hat!
Step 2: Format the data
Folium’s HeatMapWithTime plugin requires a “list of lists of points of time.” What we wanted to do is generate an ordered dictionary in Python that is indexed by the timestamp. The data we received from the Meraki API was formatted into “apFloor” labels provided by the admin when the access points are placed. Within each “apFloor” is a list of “observations” that contain information about individual clients spotted by the AP scanners, during the scanning interval.
Here’s what the data looked like straight from the Meraki API, with some dummy values:
The “observations” list is what we wanted to parse. It contains lots of useful information, but what we wanted is MAC address, latitude and longitude numbers, and timestamp:
We used Python to iterate through the observations and to eliminate the data we did not use. After a lot of data wrangling, de-duplicating MAC addresses, and bucketizing the observations into 15-minute increments, the resulting data structure looks like this:
Now that the data is in a usable format, we can feed it into Folium and see what kind of map we get back!
Step 3: Creating the map
Folium is designed to project points onto a map tile. Map tiles can show satellite images, streets, or terrain, and are projected onto a globe. In our case, however, we want to use the blueprint of the conference center. Folium’s allows for an image’s overlay to be added, and the bounds of the image to be set by specifying the coordinates for the top-left and bottom-right corners of image. Luckily, we can get this from the Meraki dashboard.
This enabled us to overlay the floorplan image on the map. Unfortunately, the map tiles themselves limit the amount of zoom available to the map visualization. Lucky for us, we did not care about the map tile now that we have the floorplan image. We passed “None” as the map tile source and finally received our data visualization and saved the map as an HTML file for Black Hat leadership.
We opened the HTML file, and we had an auto-playing heatmap that lets us zoom at far in as we want:
Detail at 1:30pm PT, on 10 August 2022 below.
To improve this going forward, the logical next steps would be to insert the data into a database for the Black Hat conference organizers, for quick retrieval and map generation. We can then start looking at advanced use-cases in the NOC, such as tracking individual a MAC address that may be producing suspicious traffic, by cross-referencing data from other sources (Umbrella, NetWitness, etc.).
——————————————————————————————————
Network Recovery, by Jessica Bair Oppenheimer
Once the final session ended, the Expo Hall closed and the steaming switched off, dozens of conference associates, technical associates, Mandalay Bay engineers and Cisco staff spread out through two million square feet and numerous switching closets to recover the equipment for inventory and packing. It took less than four hours to tear down a network that was built and evolved 11 days prior. Matt Vander Horst made a custom app to scan in each item, separating equipment donated to Black Hat from that which needed to be returned to the warehouse for the next global Cisco event.
Adapt and overcome! Check out part two of this blog, Black Hat USA 2022 Continued: Innovation in the NOC.
Until then, thanks again to our Cisco Meraki engineers, pictured below with a MR57 access point.
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.
Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter
Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey
Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira
Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz
Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann
SecureX threat response, orchestration, device insights, custom integrations, and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor
Umbrella DNS: Christian Clasen and Alejo Calaoagan
Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
Read Part 2:
Black Hat USA 2022 Continued: Innovation in the NOC
About Black Hat
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In part one of our Black Hat USA 2022 NOC blog, we discussed building the network with Meraki:
In this part two, we will discuss:
Cisco is a Premium Partner of the Black Hat NOC, and is the Official Wired & Wireless Network Equipment, Mobile Device Management, DNS (Domain Name Service) and Malware Analysis Provider of Black Hat.
Black Hat USA is my favorite part of my professional life each year. We had an incredible staff of 20 Cisco engineers to build and secure the network. Also, for the first time, we had two Talos Threat Hunters from the Talos Incident Response (TIR) team, providing unique perspectives and skills to the attacks on the network. I really appreciated the close collaboration with the Palo Alto Networks and NetWitness team members. We created new integrations and the NOC continued to serve as an incubator for innovation.
We must allow real malware on the network for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites. So, context is what really matters when investigating a potential attack and bringing so many technologies together in SecureX really accelerated investigation and response (when needed).
All the Black Hat network traffic was supported by Meraki switches and wireless access points, using the latest Meraki gear donated by Cisco. Our Meraki team was able to block people from the Black Hat network, when an investigation showed they did something in violation of the attendee Code of Conduct, upon review and approval by the Black Hat NOC leadership.
Cisco Secure provided all the domain name service (DNS) requests on the Black Hat network through Umbrella, whenever attendees wanted to connect to a website. If there is a specific DNS attack that threatened the conference, we supported Black Hat in blocking it to protect the network. However, by default, we allow and monitor DNS requests to malware, command and control, phishing, crypto mining, and other dangerous domains, which would be blocked in a production environment. That balance of allowing cybersecurity training and demos to occur, but ready to block when needed.
In addition to the Meraki networking gear, Cisco Secure also shipped an Umbrella DNS virtual appliance to Black Hat USA, for internal network visibility with redundancy. The Intel NUC containing the virtual appliance also contained the bridge to the NetWitness on-premises SIEM, custom developed by Ian Redden.
We also deployed the following cloud-based security software:
We analyzed files that were downloaded on the network, checking them for malicious behavior. When malware is downloaded, we confirm it is for a training, briefing or demonstration, and not the start of an attack on attendees.
During an investigation, we used SecureX to visualize the threat intelligence and related artifacts, correlating data. In the example below, an attacker was attempting remote code execution on the Registration Servers, alerted by the Palo Alto team, investigated by the NOC threat hunters, and blocked by order of the NOC leadership upon the results of the investigation.
Cisco Secure Threat Intelligence (correlated through SecureX)
Donated Partner Threat Intelligence (correlated through SecureX)
Open-Source Threat Intelligence (correlated through SecureX)
Continued Integrations from past Black Hat events
New Integrations Created at Black Hat USA 2022
The NOC partners, especially NetWitness and Palo Alto Networks, were so collaborative and we left Vegas with more ideas for future integration development
Creating Custom Meraki Dashboard Tiles for SecureX, by Matt Vander Horst
One of the biggest benefits of Cisco SecureX is its open architecture. Anyone can build integrations for SecureX if they can develop an API with the right endpoints that speak the right language. In the case of SecureX, the language is the Cisco Threat Intelligence Model (CTIM). As mentioned above, Cisco Meraki powered Black Hat USA 2022 by providing wired and wireless networking for the entire conference. This meant a lot of equipment and users to keep track of. To avoid having to switch between two different dashboards in the NOC, we decided to build a SecureX integration that would provide Meraki dashboard tiles directly into our single pane of glass: SecureX.
Building an integration for SecureX is simple: decide what functionality you want your integration to offer, build an internet-accessible API that offers those functions, and then add the integration to SecureX. At Black Hat, our Meraki integration supported two capabilities: health and dashboard. Here’s a summary of those capabilities and the API endpoints they expect:
| Capability | Description | API Endpoints |
| Health | Enables SecureX to make sure the module is reachable and working properly. | /health |
| Dashboard | Provides a list of available dashboard tiles and, after a tile is added to a dashboard, the tile data itself. | /tiles
/tile-data |
With our capabilities decided, we moved on to building the API for SecureX to talk to. SecureX doesn’t care how you build this API if it has the expected endpoints and speaks the right language. You could build a SecureX-compatible API directly into your product, as a serverless Amazon Web Services (AWS) Lambda, as a Python script with Django, and so on. To enable rapid development at Black Hat, we chose to build our integration API on an existing Ubuntu server in AWS running Apache and PHP.
After building the API framework on our AWS server, we had to decide which dashboard tiles to offer. Here’s what we ended up supporting:
| Tile Name | Description |
| Top Applications | Shows the top 10 applications by flow count |
| Client Statistics | Shows a summary of clients |
| Top SSIDs by Usage in GB | Shows the top 10 SSIDs by data usage in GB |
| Access Point Status | Shows a summary of access points |
Finally, once the API was up and running, we could add the integration to SecureX. To do this, you need to create a module definition and then push it to SecureX using its IROH-INT API. After the module is created, it appears in the Available Integration Modules section of SecureX and can be added. Here’s what our module looked like after being added to the Black Hat SecureX instance:
After adding our new tiles to the SecureX dashboard, SecureX would ask our API for data. The API we built would fetch the data from Meraki’s APIs, format the data from Meraki for SecureX, and then return the formatted data. Here’s the result:
These dashboard tiles gave us useful insights into what was going on in the Meraki network environment alongside our existing dashboard tiles for other products such as Cisco Secure Endpoint, Cisco Umbrella, Cisco Secure Malware Analytics, and so on.
If you want to learn more about building integrations with SecureX, check out these resources:
Talos Threat Hunting, by Jerzy ‘Yuri’ Kramarz and Michael Kelly
Black Hat USA 2022 was our first fully supported event, where we deployed an onsite threat hunting team from Talos Incident Response (TIR). Our colleagues and friends from various business units, connected by SecureX integration, granted us access to all the underlying consoles and API points to support the threat hunting efforts enhanced by Talos Intelligence.
The threat hunting team focused on answering three key hypothesis-driven questions and matched that with data modelling across all of the different technology stacks deployed in Black Hat NOC:
To answer the above hypothesis, our analysis started with understanding of how the network architecture is laid out and what kind of data access is granted to NOC. We quickly realized that our critical partners are key to extending visibility beyond Cisco deployed technologies. Great many thanks go to our friends from NetWitness and Palo Alto Networks for sharing full access to their technologies, to ensure that hunting did not stop on just Cisco kit and contextual intelligence could be gathered across different security products.
Daily threat hunt started with gathering data from Meraki API to identify IP and DNS level requests leaving the devices connected to wireless access points across entire conference. Although Meraki does not directly filter the traffic, we wanted to find signs of malicious activity such as DNS exfiltration attempts or connections to known and malicious domains which were not part of the class teaching. Given the level of access, we were then able to investigate network traffic capture associated with suspicious connections and check for suspected Command and Control (C2) points (there were a few from different threat actors!) or attempts to connect back to malicious DNS or Fast Flux domains which indicated that some of the attendee devices were indeed compromised with malware.
That said, this is to be expected given hostility of the network we were researching and the fact that classroom environments have users who can bring their own devices for hands-on labs. SecureX allowed us to quickly plot this internally to find specific hosts which were connecting and talking with malicious endpoints while also showing a number of additional datapoints which were useful for the investigation and hunting. Below is one such investigation, using SecureX threat response.
While looking at internal traffic, we have also found and plotted quite a few different port-scans running across the internal network. While not stopping these, it was interesting to see different tries and attempts by students to find ports and devices across networks. Good thing that network isolation was in place to prevent that! We blurred out the IP and MAC addresses in the image below.
Here is another example of really nice port scan clusters that were running across both internal and external networks we have found. This time it was the case of multiple hosts scanning each other and looking to discovery ports locally and across many of the Internet-based systems. All of that was part of the class but we had to verify that as it looked quite suspicious from the outset. Again, blurred picture for anonymity.
In a few instances, we also identified remarkably interesting clear-text LDAP traffic leaving the environment and giving a clear indicator of which organization the specific device belonged to simply because of the domain name which was requested in the cleartext. It was quite interesting to see that in 2022, we still have a lot of devices talking clear text protocols such as POP3, LDAP, HTTP or FTP, which are easy to subvert via Man-In-The-Middle type of attacks and can easily disclose the content of important messages such as email or server credentials. Below is an example of the plain text email attachments, visible in NetWitness and Cisco Secure Malware Analytics.
In terms of the external attacks, Log4J exploitation attempts were pretty much a daily occurrence on the infrastructure and applications used for attendee registration along with other typical web-based attacks such as SQL injections or path traversals. Overall, we saw a good number of port scans, floods, probes and all kind of web application exploitation attempts showing up daily, at various peak hours. Fortunately, all of them were successfully identified for context (is this part of a training class or demonstration) and contained (if appropriate) before causing any harm to external systems. Given the fact that we could intercept boundary traffic and investigate specific PCAP dumps, we used all these attacks to identify various command-and-control servers for which we also hunted internally to ensure that no internal system is compromised.
The final piece of the puzzle we looked to address, while threat hunting during Black Hat 2022, was automation to discover interesting investigation avenues. Both of us investigated a possibility of threat hunting using Jupyter playbooks to find outliers that warrant a closer look. We have created and developed a set of scripts which would gather the data from API endpoints and create a data frames which could be modeled for further analysis. This allowed us to quickly gather and filter out systems and connections which were not that interesting. Then, focus on specific hosts we should be checking across different technology stacks such as NetWitness and Palo Alto.
Unmistaken Identity, by Ben Greenbaum
An unusual aspect of the Black Hat NOC and associated security operations activities is that this is an intentionally hostile network. People come to learn new tricks and to conduct what would in any other circumstance be viewed rightfully as malicious, unwanted behavior. So, determining whether this is “acceptable” or “unacceptable” malicious behavior is an added step. Additionally, this is a heavily BYOD environment and while we do not want attendees attacking each other, or our infrastructure, there is a certain amount of suspicious or indicative behavior we may need to overlook to focus on higher priority alerts.
In short, there are broadly speaking 3 levels of security event at Black Hat:
When Umbrella alerted us (via a SecureX orchestration Webex workflow) of DNS requests for a domain involved in “Illegal Activity” it was reminiscent of an event at a previous conference where an attendee was caught using the conference network to download forged vaccination documents.
Using the Cisco Secure Malware Analytics platform’s phishing investigation tools, I loaded and explored the subject domain and found it to be a tool that generates and provides pseudo-randomized fake identities, customizable in various ways to match on demographics. Certainly, something that could be used for nefarious purposes, but is not illegal in and of itself. Physical security and access control is, however, also important at Black Hat, and if this activity was part of an effort to undermine that, then this was still a concern.
This is, however, also the kind of thing that gets taught at Black Hat…
Using the reported internal host IP from Umbrella, Meraki’s connection records, and the Meraki access point map, we were able to narrow the activity down to a specific classroom. Looking up what was being taught in that room, we were able to confirm that the activity was related to the course’s subject matter
Network owners and administrators, especially businesses, typically don’t want their network to be used for crimes. However, here at Black Hat what some would consider “crimes” is just “the curriculum”. This adds a layer of complexity to securing and protecting not just Black Hat, but also Black Hat attendees. In security operations, not every investigation leads to a smoking gun. At Black Hat, even when it does, you may find that the smoking gun was fired in a safe manner at an approved target range. Having the right tools on hand can help you make these determinations quickly and free you up to investigate the next potential threat.
25 Years of Black Hat – Musings from the show (and some DNS stats), by Alejo Calaoagan
Back in Singapore, I wrote about cloud app usage and the potential threat landscape surrounding them. My original plan at Black Hat USA was to dig deeper into this vector to see what interesting tidbits I could find on our attendee network. However, given that this was the 25th anniversary of Black Hat (and my 14th in total between Vegas, Singapore, and London), I’ve decided to pivot to talk about the show itself.
I think it’s safe to say, after two difficult pandemic years, Black Hat is back. Maybe it’s the fact that almost everyone has caught COVID by now (or that a lot of people just stopped caring). I caught it myself at RSA this year back in June, the first of consecutive summer super spread events (Cisco Live Vegas was the following week). Both of those shows were in the 15-18k attendee range, well below their pre-pandemic numbers. Black Hat USA 2022 was estimated at 27,000 attendees.
If I remember correctly, 2019 was in the 25-30K range. Last year in Vegas, there were ~3,000 people at the event, tops. 2021 in London, was even lower…it felt like there were less than 1,000 attendees. Things certainly picked up in Singapore (2-3k attendees), though that event doesn’t typically see attendee numbers as high as the other locations. All in all, while the pandemic certainly isn’t over, Las Vegas gave glimpses of what things were like before the “Rona” took over our lives.
The show floor was certainly back to the norm, with swag flying off the countertops and lines for Nike sneaker and Lego giveaways wrapping around different booths. The smiles on people’s faces as they pitched, sold, hustled, and educated the masses reminded me how much I missed this level of engagement. RSA gave me this feeling as well, before COVID sidelined me midway through the show anyway.
Not everything was quite the same. The Black Hat party scene certainly is not what it used to be. There was no Rapid 7 rager this year or last, or a happy hour event thrown by a security company you’ve never heard of at every bar you walk by on the strip. There were still some good networking events here and there, and there were some awesomely random Vanilla Ice, Sugar Ray, and Smashmouth shows. For those of you familiar with Jeremiah Grossman’s annual Black Hat BJJ throwdown, that’s still, thankfully, a thing. Hopefully, in the coming years, some of that old awesomeness returns….
Enough reminiscing, here are our DNS numbers from the show:
From a sheer traffic perspective, this was the busiest Black Hat ever, with over 50 million DNS requests made…
Digging into these numbers, Umbrella observed over 1.3 million security events, including various types of malware across the attendee network. Our threat hunting team was busy all week!
We’ve also seen an increase in app usage at Black Hat:
In a real-world production environment, Umbrella can block unapproved or high-risk apps via DNS.
The increases in DNS traffic volume and Cloud App usage obviously mirrors Black Hat’s return to the center stage of security conferences, following two years of pandemic uncertainty. I’m hopeful that things will continue to trend in a positive direction leading up to London and, hopefully, we’ll see you all there.
——
Hats off to the entire NOC team. Check out Black Hat Europe in London, 5-8 December 2022!
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.
SecureX threat response, orchestration, device insights, custom integrations and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor
Umbrella DNS: Christian Clasen and Alejo Calaoagan
Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley
Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter
Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey
Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira
Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz
Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
Read Part 1:
Black Hat USA 2022: Creating Hacker Summer Camp
About Black Hat
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Just like the myriad expanding galaxies seen in the latest images from the James Webb space telescope, the cybersecurity landscape consists of a growing number of security technology vendors, each with the goal of addressing the continually evolving threats faced by customers today. In order to be effective, cybersecurity tools have to be collaborative—be it sharing relevant threat intelligence, device & user insights, acting on detection and remediation workflows, and more.
We at Cisco Secure have embraced this concept for a while now with our continually growing ecosystem of multi-vendor technology integrations. At the RSA Conference 2022 earlier this year, Jeetu Patel, Cisco’s Executive Vice President and General Manager of Security and Collaboration, spoke of how the ‘cybersecurity poverty line’ is widening and how malicious actors are taking advantage of this gaping hole to unleash persistent attacks. It is imperative that cybersecurity vendors interact with and collaborate with each other to lower this gap. To do this, security vendors must adopt open ecosystems of APIs to easily integrate with each other to provide effective ways for mutual customers to defend and react to cybersecurity attacks.
Like in prior years, this fiscal year 2022 saw us growing to include new ecosystem partners and integrations. With 22 new partners and 51 new integrations in our ecosystem, Cisco Secure Technical Alliance (CSTA) now boasts over 450 integrations, including technical integrations with Cisco Duo and Cisco Kenna. This allows our mutual customers the freedom to implement the cybersecurity tools of their choice with the knowledge that these tools can integrate with each other if they need to, thus realizing a better return on investment in their cybersecurity spending and improving cybersecurity posture.
In this annual round-up of our ecosystem, we congratulate our new partners in CSTA and existing partners as well, who have either created new integrations across our portfolio or augmented existing ones. For more details on each partner integration in this announcement, please read through the individual partner highlights below.
Happy integrating!
The AlienApp for Cisco Secure Endpoint enables you to automate threat detection and response activities between USM Anywhere and Cisco Secure Endpoint. It also enhances the threat response capabilities of USM Anywhere by providing orchestration and response actions to isolate or un-isolate hosts based on risks identified in USM Anywhere. In addition, it allows you to collect hourly events from Cisco Secure Endpoint through the USM Anywhere Job Scheduler. Read more here.
AttackIQ automates the evaluation of Cisco Secure Endpoint against the tactic categories as outlined by MITRE ATT&CK. The AttackIQ and Cisco partnership and technical integration enables organizations to validate that the Cisco Secure Endpoint is deployed correctly and configured optimally, ensuring protection for your endpoints against the latest threats. Read more here.
With Certego Tactical Response for Cisco Secure Endpoint, monitored endpoints are monitored by the Certego PanOptikon SOAR platform. When Certego IRT detects malicious activities on a specific host in the customer’s network, it can isolate compromised hosts to block the attack, even without requiring the user to access the Cisco Secure Endpoint Console. Read more about the Certego here.
Cisco Secure Endpoint is now certified for the ServiceNow ITSM San Diego release. The Cisco Secure Endpoint App on ServiceNow provides users with the ability to integrate event data from the Cisco Secure Endpoint into ServiceNow by creating ITSM incidents. The app automates the collection of events from Cisco Secure Endpoint and groups them into single incidents. Read more here.
FAMOC manage from Techstep, a Gartner-recognized MMS provider, is an MDM designed to give IT a complete view and absolute control over mobile devices used by the workforce, so that people can work more effectively and securely. With the Cisco Security Connector for iOS integration, FAMOC MDM extends its enterprise mobility management with an extra layer of network security and traffic analysis tool, giving IT admins tools to make actionable decisions and design access control policies. Read more here.
Elastic Security now supports event ingestion from Cisco Umbrella, providing visibility into user activity and attempts to access potentially malicious domains. This new integration supports Umbrella proxy, cloud firewall, IP, and DNS logs. This integration enables security analysts to detect threats and visualize Cisco Umbrella data, and also correlate Umbrella events with other data sources including endpoint, cloud, and network. This integration expands on Elastic’s on-going expansion of Cisco integrations including ASA, Nexus, Meraki, Duo and Secure Firewall Threat Defense. Read more here.
FortiSIEM brings together visibility, correlation, automated response, and remediation in a single, scalable solution. It reduces the complexity of managing network and security operations to effectively free resources, improve breach detection, and even prevent breaches. Read more here.
Hunters ingests Cisco Umbrella log and alert information into our SOC Platform; the Platform then correlates that information with all of the other (vendor agnostic) customer security telemetry, including EDR, Identity and Cloud/Network log data, in the customer’s infrastructure to synthesize and detect incidents with a higher fidelity than any single tool alone can produce. Read more here.
LearnSafe equips school leaders (K-12) with evidence-based information to better understand which students are exhibiting behavioral issues and in need of help based on what they are using, saying, and doing on the school-owned computer. With Cisco Umbrella, LearnSafe administrators are able to block access to domains their students should not be accessing. Read more here.
The Cisco Umbrella solution for Microsoft Azure Sentinel is now live! This integration enables your customers to ingest Cisco Umbrella events stored in Amazon S3 into Microsoft Sentinel using the Amazon S3 REST API. Read more here.
Sumo Logic’s cloud-native collector supports automatic ingestion of logs from Cisco Umbrella’s hosted AWS S3 buckets. Data collected from Umbrella can then be routed to Sumo’s Cloud SIEM, where it is then automatically normalized and applied to our rule’s engine. Several built-in rules for Umbrella have been created that, when triggered, will generate security signals in the platform. These and other security signals are then clustered together based on related entities (IP, email, domain name, URL, etc.) to create insights for review by the SOC. Read more here.
The Secure Firewall team and Alkira have validated Secure Firewall (Virtual) Version 7.1 to run on Alkira’s cloud network as-a-service (CNaaS) platform. The solution offers on-demand hybrid and multi-cloud connectivity, integrated network and security services, end-to-end visibility, controls and governance. Read more here.
The Secure Firewall team has validated Cyware’s STIX 1.2 threat intelligence feed for interoperability with Secure Firewall’s Threat Intelligence Director. Customers can quickly operationalize the inbound data to protect the network from the latest threats. Read more here.
Dragos protects critical infrastructure and has joined the CSTA program. Dragos inventories assets, determines risk and vulnerabilities and generates firewall policy objects that administrators can apply to their Cisco Secure Firewall deployment through its REST API. Read more here.
The Secure Firewall team and Equinix have validated Secure Firewall (Virtual) to run on Equinix’s Network Edge as a Service platform. Equinix Fabric allows you to connect digital infrastructure and services on demand via secure, software-defined interconnection (Ecosystem). Read more here.
Fastvue has joined the CSTA program. The Fastvue Site Clean engine intelligently interprets Cisco Secure Firewall log data so that non-technical employees can easily see what people are actually doing online. The data use to keep companies compliant with workplace and school policies. Read more here.
Alef Nula has developed a new integration with ISE. The Alef Nula Identity Bridge consumes identity updates published by pxGrid and serves them to ASA firewalls using the CDA/Radius protocol. Using pxGrid v2.0, it replaces unsupported Cisco CDA and allows ASA firewalls to become an identity consumer of ISE context. It can read the full identity database and can update registered ASA firewalls in Full Download mode. Read more here.
Forescout’s pxGrid Plugin integrates with existing Cisco ISE deployments so that you can benefit from Forescout visibility and assessment for policy decisions, while continuing to use ISE as an enforcement point. The pxGrid Plugin enables Forescout platform policies to detect ISE-related properties on endpoints, and to apply Cisco ISE ANC policies, including policies that assign Security Groups to devices. Read more here.
FortiManager provides automation-driven centralized management of Fortinet devices from a single console, enabling full administration and visibility of your network devices through streamlined provisioning and innovative automation tools. FortiManager dynamically collects updates from Cisco ISE with pxGrid and forwards them to FortiGate using the Fortinet Single Sign On (FSSO) protocol. This enables the use of session information collected by Cisco ISE to be leveraged in FortiOS security policies. Read more here.
Radiflow provides OT ICS policy creation and enforcement with the Radiflow iSID IDS. They recently completed a new integration with ISE leveraging pxGrid. With this integration Cisco ISE receives enriched data of OT devices from Radiflow iSID and will process it according to the profiles and policies which have been configured. Enriching ISE with OT specific insights available with iSID’s DPI engine enables better decision making within ISE by providing additional context to categorize devices by their type/function within the OT environment. Read more here.
XTENDISE is a simple web application connected to Cisco ISE. It is designed for administrators, helpdesk, operators or anyone who needs to work with ISE and helps them with everyday routine tasks related to 802.1X without the need to train them in Cisco ISE. XTENDISE saves administrators’ time, prevents errors and increases network security. Read more here.
The Cisco Secure Malware Analytics Add-On for Splunk leverages the Threat Grid API to enrich events within Splunk. The add-on is now updated for Splunk 8 and is available on Splunkbase. Read more here.
Censys now has an integration with SecureX threat response, which returns Sightings of IP and IPv6 Observables (IOCs) in an investigation. Read more about the Censys relay module here.
The new Exabeam integration empowers users to investigate an observable and determine if it is contained in a log message stored in Exabeam Fusion SIEM Data Lake. It provides users with the date and time the observable was seen in the log, the forwarder that sent the log, and the raw log messages. When you pivot into Exabeam and search for an observable in all the log messages, the results of the search are displayed in the Exabeam UI. This integration allows you to query IPv4, IPv6, SHA-1, SHA-256, MD5, domain, URL, file path, user and email data types and it returns sightings of an observable from each log message. Read more here.
The LogRhythm integration empowers users to investigate an observable and determine if it is contained in an event stored in LogRhythm. It provides users with the date and time the observable was seen in the event and the raw event data. This integration allows you to query IPv4 and IPv6 data types and it returns sightings of an observable from each event. Read more here.
A proof-of-concept integration with RSA NetWitness SIEM was built for the RSAC SOC and Black Hat NOCs. The SecureX Concrete Relay implementation using NetWitness as a third-party Cyber Threat Intelligence service provider. The Relay itself is just a simple application written in Python that can be easily packaged and deployed. Read more here.
Cisco SecureX threat response integration with SecOps is now certified for the ServiceNow San Diego release. The module allows ServiceNow SecOps to leverage the Verdicts, Refer and Response capabilities provided by SecureX threat response to assist the security analyst in their investigation workflow. Read more here.
The Sumo Logic Cloud SIEM integration provides security analysts with enhanced visibility across the enterprise to thoroughly understand the impact and context of an attack. Streamlined workflows automatically triage alerts to maximize security analyst efficiency and focus. This integration indicates to users that the observable in an investigation is contained in an insight and/or signal within Sumo Logic Cloud SIEM. It allows you to query IPv4, IPv6, SHA-1, SHA-256, MD5, domain, and URL data types. It also returns sightings and indicators of an observable from each insight and signal retrieved from Sumo Logic Cloud SIEM. Read more here.
APIVoid provides JSON APIs useful for cyber threat analysis, threat detection and threat prevention. The following APIVoid atomic actions for SecureX Orchestration Workflows are now available: Get Domain Reputation, Get IP Reputation, Get URL Reputation, Get URL Status. Access the workflows here.
Censys is a company that allows users to discover the devices, networks, and infrastructure on the Internet and monitor how it changes over time. SecureX orchestration atomic actions for Censys is now available and includes: Basic Search. Access the workflows here.
This integration radically reduces the time and resources enterprises spend to detect, investigate, and remediate ransomware threats to data. It empowers SecOps, ITOps and NetOps with visibility and automation to collaborate in countering ransomware — regardless of whether data resides on-premises or in the cloud — delivering enterprise-wide confidence in deterring, detecting, and recovering fast from cyberattacks. Cohesity’s next-gen data management enhances Cisco SecureX by adding visibility and context to data, complementing Cisco’s existing capabilities for networks, endpoints, clouds, and apps. Read more here.
SecureX orchestration atomic actions for workflows are now available for Farsight Security DNSDB. They include various items like DKIM key inspections, DNS Resource Records and more. Access the workflows here.
SecureX orchestration workflows for Fortinet FortiGate are now available: Block URL, IP and Domain Threat Containment. Access the workflows here.
SecureX orchestration workflows for Jamf Pro include: Lock Computer, Lock Mobile Device. Access the workflows here.
SecureX orchestration workflows for Palo Alto Networks Panorama are now available: Block URL, IP, Domain Threat Containment. Access the workflows here.
A new Orchestration action provides top MacOS IR Indicators to ServiceNow This workflow runs multiple Orbital queries on the endpoint provided to look for top incident response indicators of compromise. The results are then posted to a ServiceNow incident. Supported observables: ip, mac_address, amp_computer_guid, hostname. Access the workflow here.
Shodan is a database of billions of publicly available IP addresses, and it’s used by security experts to analyze network security. SecureX orchestration atomic actions for Shodan include: Basic Search. Access the workflows here.
Earlier this year we announced SecureX Device Insights which provides comprehensive endpoint inventory in a single unified view. Endpoint searching and reporting allows you to assess device security configuration on employee-owned, contractor-owned, company owned, and IoT/OT devices—without risking business disruption. With Device Insights you can
Device insights supports the following third-party sources in its initial release: Jamf Pro, Microsoft Intune, Ivanti MobileIron and VMware Workspace ONE (formerly AirWatch).
Bitglass’ Next-Gen CASB provides data protection, threat protection, access management, and visibility, while Duo offers identity verification options like SSO and MFA. The Duo and Bitglass integration provide a synergistic solution that funnels traffic through Duo’s SSO and verifies users via its MFA so Bitglass can deliver real-time data loss prevention and granular adaptive access control. Because of Bitglass’ agentless architecture, the joint solution can secure any app, any device, anywhere. Read more about the integration here. A joint solution brief is also available here.
Cmd helps companies authenticate and manage user security in Linux production environments without slowing down teams — you don’t need to individually configure identities and devices. Cmd integrates with Duo to put 2FA checkpoints into Linux-based data centers and cloud infrastructure. The combination of Cmd and Duo enables development teams to run at the modern, agile pace they are accustomed to without any security-induced slowdowns. Read more here.
Darktrace is an AI-native platform that delivers self-learning cyber defense and AI investigations and seamlessly integrates with other tools via an open and extensible architecture. Darktrace’s Security Module for Duo provides coverage over access, user sessions and platform administration within the Duo platform. Read more here.
Dashlane is a password manager that now supports Duo using Duo SSO. The integration lets IT Administrators easily deploy Duo + Dashlane and set up access policies. End users can easily access Dashlane and their passwords with SSO from Duo. Read more here.
HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates. Add another layer of protection and protect access to secrets within HashiCorp Vault with Duo Security MFA. Read more here. A recording of the Cisco Duo + HashiCorp webinar is available to view here.
Oort discovers vulnerabilities across an entire user population (or a segment of it). Trigger notifications related to behavioral anomalies or best practices, or policies not being followed. Oort integrates with Duo for identity analytics and threat detection to provide a complete picture of the user behavior and highlight any anomalous activity or identify risks. Read more here.
Perimeter 81 simplifies cyber and network security for the hybrid workforce, ensuring secure access to local networks, applications, and cloud infrastructure. Their integration with Duo provides protection for administrators and end-users who need to log in to Perimeter 81. Read more here.
Specops Software, a leading provider of password management and authentication solutions, protects businesses by securing user authentication across high-risk tasks including account unlocks and password recovery via self-service or the IT service desk. Organizations can extend Duo authentication to secure user verification across these use cases. Read about the integration here. A blog on the integration is also available here.
Sectona is a Privileged Access Management company that delivers integrated privilege management components for securing dynamic remote workforce access across on-premises or cloud workloads, endpoints, and machine-to-machine communication. Duo’s secure access multi-factor authentication can be used to ensure that each user authenticates using multiple methods (factors) while accessing Sectona Privileged Access Management. Read more here.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Image: Shutterstock.
A cybersecurity firm says it has intercepted a large, unique stolen data set containing the names, addresses, email addresses, phone numbers, Social Security Numbers and dates of birth on nearly 23 million Americans. The firm’s analysis of the data suggests it corresponds to current and former customers of AT&T. The telecommunications giant stopped short of saying the data wasn’t theirs, but it maintains the records do not appear to have come from its systems and may be tied to a previous data incident at another company.
Milwaukee-based cybersecurity consultancy Hold Security said it intercepted a 1.6 gigabyte compressed file on a popular dark web file-sharing site. The largest item in the archive is a 3.6 gigabyte file called “dbfull,” and it contains 28.5 million records, including 22.8 million unique email addresses and 23 million unique SSNs. There are no passwords in the database.
Hold Security founder Alex Holden said a number of patterns in the data suggest it relates to AT&T customers. For starters, email addresses ending in “att.net” accounted for 13.7 percent of all addresses in the database, with addresses from SBCGLobal.net and Bellsouth.net — both AT&T companies — making up another seven percent. In contrast, Gmail users made up more than 30 percent of the data set, with Yahoo addresses accounting for 24 percent. More than 10,000 entries in the database list “none@att.com” in the email field.
Hold Security found these email domains account for 87% of all domains in the data set. Nearly 21% belonged to AT&T customers.
Holden’s team also examined the number of email records that included an alias in the username portion of the email, and found 293 email addresses with plus addressing. Of those, 232 included an alias that indicated the customer had signed up at some AT&T property; 190 of the aliased email addresses were “+att@”; 42 were “+uverse@,” an oddly specific reference to an AT&T entity that included broadband Internet. In September 2016, AT&T rebranded U-verse as AT&T Internet.
According to its website, AT&T Internet is offered in 21 states, including Alabama, Arkansas, California, Florida, Georgia, Indiana, Kansas, Kentucky, Louisiana, Michigan, Missouri, Nevada, North Carolina, Ohio, Oklahoma, Tennessee, Texas and Wisconsin. Nearly all of the records in the database that contain a state designation corresponded to those 21 states; all other states made up just 1.64 percent of the records, Hold Security found.
Image: Hold Security.
The vast majority of records in this database belong to consumers, but almost 13,000 of the entries are for corporate entities. Holden said 387 of those corporate names started with “ATT,” with various entries like “ATT PVT XLOW” appearing 81 times. And most of the addresses for these entities are AT&T corporate offices.
How old is this data? One clue may be in the dates of birth exposed in this database. There are very few records in this file with dates of birth after 2000.
“Based on these statistics, we see that the last significant number of subscribers born in March of 2000,” Holden told KrebsOnSecurity, noting that AT&T requires new account holders to be 18 years of age or older. “Therefore, it makes sense that the dataset was likely created close to March of 2018.”
There was also this anomaly: Holden said one of his analysts is an AT&T customer with a 13-letter last name, and that her AT&T bill has always had the same unique misspelling of her surname (they added yet another letter). He said the analyst’s name is identically misspelled in this database.
KrebsOnSecurity shared the large data set with AT&T, as well as Hold Security’s analysis of it. AT&T ultimately declined to say whether all of the people in the database are or were at some point AT&T customers. The company said the data appears to be several years old, and that “it’s not immediately possible to determine the percentage that may be customers.”
“This information does not appear to have come from our systems,” AT&T said in a written statement. “It may be tied to a previous data incident at another company. It is unfortunate that data can continue to surface over several years on the dark web. However, customers often receive notices after such incidents, and advice for ID theft is consistent and can be found online.”
The company declined to elaborate on what they meant by “a previous data incident at another company.”
But it seems likely that this database is related to one that went up for sale on a hacker forum on August 19, 2021. That auction ran with the title “AT&T Database +70M (SSN/DOB),” and was offered by ShinyHunters, a well-known threat actor with a long history of compromising websites and developer repositories to steal credentials or API keys.
Image: BleepingComputer
ShinyHunters established the starting price for the auction at $200,000, but set the “flash” or “buy it now” price at $1 million. The auction also included a small sampling of the stolen information, but that sample is no longer available. The hacker forum where the ShinyHunters sales thread existed was seized by the FBI in April, and its alleged administrator arrested.
But cached copies of the auction, as recorded by cyber intelligence firm Intel 471, show ShinyHunters received bids of up to $230,000 for the entire database before they suspended the sale.
“This thread has been deleted several times,” ShinyHunters wrote in their auction discussion on Sept. 6, 2021. “Therefore, the auction is suspended. AT&T will be available on WHM as soon as they accept new vendors.”
The WHM initialism was a reference to the White House Market, a dark web marketplace that shut down in October 2021.
“In many cases, when a database is not sold, ShinyHunters will release it for free on hacker forums,” wrote BleepingComputer’s Lawrence Abrams, who broke the news of the auction last year and confronted AT&T about the hackers’ claims.
AT&T gave Abrams a similar statement, saying the data didn’t come from their systems.
“When asked whether the data may have come from a third-party partner, AT&T chose not to speculate,” Abrams wrote. “‘Given this information did not come from us, we can’t speculate on where it came from or whether it is valid,'” AT&T told BleepingComputer.
Asked to respond to AT&T’s denial, ShinyHunters told BleepingComputer at the time, “I don’t care if they don’t admit. I’m just selling.”
On June 1, 2022, a 21-year-old Frenchman was arrested in Morocco for allegedly being a member of ShinyHunters. Databreaches.net reports the defendant was arrested on an Interpol “Red Notice” at the request of a U.S. federal prosecutor from Washington state.
Databreaches.net suggests the warrant could be tied to a ShinyHunters theft in May 2020, when the group announced they had exfiltrated 500 GB of Microsoft’s source code from Microsoft’s private GitHub repositories.
“Researchers assess that Shiny Hunters gained access to roughly 1,200 private repositories around March 28, 2020, which have since been secured,” reads a May 2020 alert posted by the New Jersey Cybersecurity & Communications Integration Cell, a component within the New Jersey Office of Homeland Security and Preparedness.
“Though the breach was largely dismissed as insignificant, some images of the directory listing appear to contain source code for Azure, Office, and some Windows runtimes, and concerns have been raised regarding access to private API keys or passwords that may have been mistakenly included in some private repositories,” the alert continues. “Additionally, Shiny Hunters is flooding dark web marketplaces with breached databases.”
Last month, T-Mobile agreed to pay $350 million to settle a consolidated class action lawsuit over a breach in 2021 that affected 40 million current and former customers. The breach came to light on Aug. 16, 2021, when someone starting selling tens of millions of SSN/DOB records from T-Mobile on the same hacker forum where the ShinyHunters would post their auction for the claimed AT&T database just three days later.
T-Mobile has not disclosed many details about the “how” of last year’s breach, but it said the intruder(s) “leveraged their knowledge of technical systems, along with specialized tools and capabilities, to gain access to our testing environments and then used brute force attacks and other methods to make their way into other IT servers that included customer data.”
A sales thread tied to the stolen T-Mobile customer data.
Whether using the internet for play or work, you want to spend your time online enjoying the peace of mind that comes with having a secure network.
You don’t want to contend with someone taking your personal data — whether it’s credit card information, passwords, or bank account details — via malware or a data breach on your Android, Windows, or Apple iOS device.
Fortunately, with some sensible precautions and simple steps, you can use your connected devices productively without worrying about cybercriminals and malicious software. This article explains how to stop hackers from getting access to your sensitive data.
You can take steps to protect your different computing and mobile devices and operating systems. These steps can be divided into technological solutions and the right awareness and information to provide a comforting measure of self-protection.
It’s like learning karate for self-defense, giving you confidence as you negotiate the wider world (and hoping that you never have to use it).
When it comes to identity protection software, McAfee provides a proven solution with our identity protection and privacy services. The protection includes alerts if your sensitive information is found on the dark web (up to 10 months sooner than other providers), personal data cleanup from sites gathering and selling your information, and an unlimited virtual public network (VPN) service that protects your privacy as you use public Wi-Fi networks.
You’ll also get up to $1 million in identity theft coverage and hands-on restoration support to help you reclaim your identity.
Simple, obvious passwords and passcodes (like your street address, your birthday, your kids’ or pets’ names, or “1234” or “abcd”) are easy for cybercriminals to crack, giving them unwanted access to your private data.
The stronger your password, the better your protection. Some best password practices include:
It’s important not to be a standing target. Just as you should use different passwords for everything, you should regularly change your passwords. You should do this a few times a year (although some cyber experts say this might not be necessary if you have a long and very complicated password).
If you have a number of passwords that you update often, it might be worth getting a password manager like McAfee True Key to keep track of them. Not only will you not be faced with remembering all your different passwords or writing them down (also a no-no), but it can also help you create and store unique passwords.
The software uses the strongest encryption algorithms available to protect your passwords, scrambling them so no one else can access them. It’ll also suggest new passwords and automatically log you into your online accounts with just one master password.
Another important line of defense is multi-factor authentication (sometimes known as two-factor authentication). This system uses a password and a second piece of verification — often an SMS message sent to your Android device or iPhone — to authenticate your identity.
This provides hard-to-beat protection even if a hacker has your password. Besides receiving SMS codes, there are also code-generating apps and physical security keys.
Thinking before you click on an email or text is a very important defense against phishing scams. Your bank won’t send you an email or text notifying you that there’s been suspicious activity on your account.
Does getting a large refund from your phone company sound too good to be true? It is. Similarly, the Internal Revenue Service (IRS) won’t text to tell you that you owe them money, and princes aren’t going to give you a fortune out of the blue.
Internet users beware: If you’re not absolutely certain that the text message you received is from a legitimate and trusted source, delete it. You can always contact the business or person directly to confirm that the message is legit.
Any operating system or app you use is open to malicious cyberattacks. This is why you should keep all your software up to date with the latest versions. Software developers are continually fixing holes in their products and offering cybersecurity patches to make them as safe and hacker-proof as possible.
Make sure your software, firmware, and security settings are up to date on your home’s Wi-Fi router, as well. You can often change your settings to allow for automatic updates.
Sure, who doesn’t like to go to a cafe, library, or hotel lobby to use the free Wi-Fi? But security is often weak in these public networks. If you open your online banking account or access personal information, you may unwittingly be giving a personal invitation to eavesdropping cybercriminals.
This is where the bank-grade level of protection of McAfee Secure VPN comes in, which automatically turns on when you need it and keeps you safe on public Wi-Fi networks.
Even if your device does get hacked, you can protect vital information on your Windows or macOS system with an encryption program like BitLocker or FileVault. You can protect any hard drive you use, including portable ones and USB keys.
It’s also a good idea to only shop at encrypted websites marked with the prefix “HTTPS” in their URLs.
One of the best ways to surf the web in comfort while keeping hackers at bay is with the comprehensive solutions provided by McAfee Total Protection.
Your protection includes proactive measures (meaning we’ll guide you to the best choices for prevention), early detection, and expert identity theft support.
This means you’ll get identity monitoring, up to $1 million in identity theft coverage, lost wallet protection, premium antivirus software, a secure VPN, and personal data removal. In particular, our Personal Data Cleanup service will help find and remove your personal information from data broker websites and people search sites.
With McAfee, you don’t have to be afraid of hackers. Let us deal with them.
The post How to Stay One Step Ahead of Hackers appeared first on McAfee Blog.
FreshRSS 