Login
AI is transformative, driving huge productivity gains. The engine of AI — the data center — will grow substantially, maybe an order of magnitude or more over the coming years.
The industry went thr… Read more on Cisco Blogs
In an ever-evolving digital landscape, cybersecurity has become the cornerstone of organizational success. With the proliferation of sophisticated cyber threats, businesses must adopt a multi-layered… Read more on Cisco Blogs
Let’s say that, during the middle of a busy day, you receive what looks like a work-related email with a QR code. The email claims to come from a coworker, requesting your help in reviewing a d… Read more on Cisco Blogs
There has been an exponential increase in breaches within enterprises despite the carefully constructed and controlled perimeters that exist around applications and data. Once an attacker can access… Read more on Cisco Blogs
Related tags
Join the guided tour outside the Security Operations Center, where we’ll discuss real time network traffic of the RSA Conference, as seen in the NetWitness platform. Engineers will be using Cisco S… Read more on Cisco Blogs
Core Infrastructure and Threat Hunting. 2
Cisco Secure Access Enables ZTNA for SOC… Read more on Cisco Blogs
By shifting from point-solutions to a cybersecurity platform approach, IT and security teams significantly improve their efficiency and security outcomes. Security Service Edge (SSE) projects are… Read more on Cisco Blogs
Cisco is honored to be this year’s winner of the Best Next Generation Firewall Award in the SE Labs 2023 Annual Report. This industry recognition validates Cisco’s continuous push towards harmonizing network, workload, and application security across hybrid and multicloud environments. I’m incredibly proud of the Cisco Secure Firewall team and am thankful for our amazing customers who continue to trust Cisco and develop their network security around our capabilities.
SE Labs, a cybersecurity testing and evaluation firm, provides impartial and independent assessments of various cybersecurity products and solutions. In their 2023 Annual Report, SE Labs states:
“Our Annual Security Awards recognizes security vendors that notonly do well in our tests, but perform well in the real world withreal customers. These awards are the only in the industry thatrecognize strong lab work combined with practical success.”
SE Labs performs tests on behalf of customers seeking independent proof-of-value assistance, as well as security vendors. At Cisco, we use third-party evaluations from multiple sources, including SE Labs, to augment our internal testing and to drive product improvement.
Winners were determined after months of in-depth testing, based on a combination of continual public testing, private assessments and feedback from corporate clients who use SE Labs to help choose security products and services. The award further validates that our customers can expect superior threat protection and performance with Cisco Secure Firewall.
SE Labs’ reports use the MITRE ATT&CK framework, employing both common “commodity” malware samples and sophisticated, targeted attacks. Their network security testing uses full attack chains to assess the detection and protection abilities of network devices and combinations of network and endpoint solutions. SE Labs publishes its testing methodologies and is BS EN ISO 9001: 2015 certified for The Provision of IT Security Product Testing.
As a worldwide leader in networking and security, Cisco is better positioned than any other security vendor to incorporate effective firewall controls into our customers’ infrastructure — anywhere data and applications reside. We offer a comprehensive threat defense with industry-leading Snort 3 IPS to protect users, applications, and data from continuously evolving threats. Our solutions also leverage machine learning and advanced threat intelligence from Cisco Talos, one of the world’s largest commercial threat intelligence teams.
In the constantly evolving world of cybersecurity, it is important to have access to the latest and most advanced technologies to stay ahead of threats. Whether you are an enterprise, government, healthcare, or a service provider organization, Cisco Secure Firewall provides top-ranked security.
When you invest in Cisco Secure Firewall, you are investing in award-winning threat defense with capabilities that are built for the real world. Learn more about SE Labs 2023 Annual Report, Cisco Secure Firewall and how you can refresh your firewall.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
For Cisco engineers working on Duo, having a remote-first workplace has helped them reach life goals, connect with colleagues around the world, and be intentional communicators. We understand that working remotely can be an adjustment — that’s why we’ve compiled the 10 parts of remote work that surprised our team members most and their advice for navigating the nuances. If you’re interested in being part of a remote-first workplace, check out our open positions.
Senior Engineering Leader David Rines has worked remotely for the past seven years. He’s found that Cisco’s approach to distributed teams has “enabled us to pick up the right talent, and not necessarily local talent. We are moving towards a global, follow the sun environment,” he said.
One of the aspects Rines appreciates most of this structure is getting “a widely varied set of perspectives and experiences that help build a more reliable, more robust product, which is why we’re here.”
Another benefit to having colleagues across the globe is the sharing of recipes, a perk Senior Site Reliability Engineer Bernard Ting particularly enjoys. Proactively communicating with colleagues virtually “helps you to form bonds with people from other teams. You can always learn something new about cultures elsewhere. I talk to people about food and so I’m always gathering recipes from people from all over the world,” Ting shared.
While some may fear that working remotely could lead to feelings of isolation and loneliness, a different camaraderie can flourish in the structure of our distributed teams. With colleagues across time zones, “there’s always someone there who you can reach out to help solve your problem,” Rines said.
Collaboration hours are another way Site Reliability Engineering Manager Jaya Sistla has cultivated virtual community and problem-solving. These hours are blocked off for team members to talk about what they’re working on. “The main thing is being able to ask for help so you don’t go into the rabbit hole debugging things,” Sistla said.
Ting points out that working in a distributed model allows you to really engage in virtual events and conversations. Given that the team mainly communicates through online chat, Ting has found that “forces you to see everyone as equally approachable, which has made me more comfortable reaching out to people from anywhere in the world.”
For folks sharing an office, collaboration can happen through casual chats over coffee. When facing a challenge, you can ask your neighbor for support. While ideally virtual communication could have a similar cadence and spontaneity, the logistics of remote and distributed work require intentionality and being proactive in connecting with colleagues as people and as co-workers.
When Ting first started working remotely, he felt that every meeting needed to be formal and have a business objective. By sharing his feelings with his manager, he was reassured that “socializing is a very important part of teamwork, because if you don’t have a good relationship with your colleagues you’re not going to be able to have healthy discussions, healthy conflict or be able to critique each other when the situation arises.”
Since that conversation, Ting has been more proactive about catching up with colleagues, which can include sharing a coffee over video chat. Duo’s “coffee roulette” formalizes the process as every month, employees who opt in can be randomly paired up for a quick half-hour chat focused exclusively on socializing. Ting has found being proactive about socializing virtually helpful. “It’s made me more intentional with my time and really treasure the social experience you can get,” he said.
Some folks may be concerned that without a manager observing their efforts and work ethic day in and day out, it may be harder to recognize accomplishments and challenges. Ting found that within his team “when you work on projects and in your one-on-ones with your managers, they’re always very intentional about learning what you’ve been doing and seeing what your progress is like on certain projects. I’ve been asked, ‘How do you think you can improve? What are some of the things you’ve been doing outside of the team work?’”
To cultivate cross-team collaboration and education, there are thoughtfully planned virtual lunch and learns. “We schedule training sessions and common meetings at times that are flexible for everyone. If it has to be repeated, we do it so people can comfortably attend rather than stretching themselves and attending at odd hours,” Sistla said.
For Software Engineer Nick Aspinall, an important and fun part of working remotely is keeping in touch with virtual messaging. One unique perk has been getting to create and customize emojis with team members including a few of himself in “various ridiculous states,” he said.
Connecting with colleagues on themed channels focused on personal and professional interests from coffee to pets “makes it really cool because you can meet people across different teams and still get some of the feeling of rubbing elbows that you get when you’re in the office,” Aspinall said. Participating in these virtual conversations boosts morale while also providing an endless supply of cute animal pics.
Given the multi-faceted nature of our work and the importance of consistent information sharing, having different communication channels and formats to communicate data with varying degrees of complexity is vital. Having information readily accessible, accurate and updated is particularly necessary in a field like cybersecurity.
Senior Software Engineer Mario Lopez finds that the variety of information sources contributes to an easeful remote working experience. For instance, for complex architecture decisions or detailing, Duo’s Wiki is the best source.
Software Engineer Hanna Fernandez has benefited from chat channels dedicated to design and engineering topics to “see what everyone’s up to and what thoughts people have,” she said. Sista pointed out these are great places to ask questions and open up dialogue to solve problems.
Our culture is “video-on,” meaning that it is preferred that during video meetings, as much as possible, attendees have their cameras on. Lopez loves this because “you get a bit of that personal human element.”
“We’re all people behind these screens. You definitely get some of people’s personality through text, but you get it more when you actually see them. It’s infectious when you see someone smiling. You’ve got to smile back,” he shared (while we both smiled).
When Fernandez started at Cisco, she was advised to schedule individual meetings with everyone she would be working with on every team that she joined. That suggestion is one she’s applied even virtually.
“It’s a great strategy because I already know that my team is super talented and very smart, but this way I also get to know them as humans beyond their roles,” Fernandez said. Fernandez also finds it important to check in with co-workers and ask how they’re feeling and how their time off was. “I know a lot of people hate small talk, but it’s not just small talk. I’m genuinely interested in how my co-workers are doing.”
One of Ting’s biggest goals was buying his first house in the countryside outside of London. By working remotely, Ting has flexibility in his location which allowed him to achieve his goal of buying a house and settling down with his partner, while giving their dogs the space they need to be dogs.
When transitioning from fully remote to hybrid, it’s important to recognize that there will be some shifts to get accustomed to. As the structures of remote, distributed and hybrid work evolve, it’s important to stay flexible and notice what’s possible through multiple modalities of team building. Many teams have enjoyed in-person gatherings and connecting through virtual lunches and team games when remote.
Fernandez has had multiple roles with multiple structures at Cisco. As an intern, she was fully in person and shared desk space with other interns who collaborated on full stack engineering. While working in finance IT, Fernandez was hybrid and many of her colleagues were distributed among multiple offices. The pandemic began while she was in a DevOps role, forcing her to maintain boundaries around her work time while working fully remotely. In her current role working on Duo, Fernandez is completely remote but advocates for in-person events if possible, because “humans are social creatures who want to see each other’s faces in real life once in a while.”
For Aspinall, “when we did come back to the office, there was a bit of an adjustment period where you were overstimulated from the office.” He also wanted to ensure team members who were 100% remote were fully included. Now he sees that while half his team is fully remote and the other half is hybrid, “that doesn’t stop anyone from doing anything. All of our meetings feel the same. They’re all seamless.”
If you’re interested in joining our team from wherever you are in the world, check out our open roles.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Related tags
Our core mission in the NOC is network resilience. We also provide integrated security, visibility and automation, a SOC inside the NOC.
In part one, we covered:
In part two, we are going deep with security:
As the needs of Black Hat evolved, so did the Cisco Secure Technologies in the NOC:
The SecureX dashboard made it easy to see the status of each of the connected Cisco Secure technologies.
Since joining the Black Hat NOC in 2016, my goal remains integration and automation. As a NOC team comprised of many technologies and companies, we are pleased that this Black Hat NOC was the most integrated to date, to provide an overall SOC cybersecurity architecture solution.
We have ideas for even more integrations for Black Hat Asia and Black Hat USA 2023. Thank you, Piotr Jarzynka, for designing the integration diagram.
Below are the SecureX threat response integrations for Black Hat Europe, empowering analysts to investigate Indicators of Compromise very quickly, with one search.
The original Black Hat NOC integration for Cisco was NetWitness sending suspicious files to Threat Grid (know Secure Malware Analytics). We expanded that in 2022 with Palo Alto Networks Cortex XSOAR and used it in London, for investigation of malicious payload attack.
NetWitness observed a targeted attack against the Black Hat network. The attack was intended to compromise the network.
NetWitness extracted the payload and sent it to Secure Malware Analytics for detonation.
Reviewing the analysis report, we were able to quickly determine it was the MyDoom worm, which would have been very damaging.
The attack was blocked at the perimeter and the analysts were able to track and enrich the incident in XSOAR.
My first time at Black Hat turned out to be an incredible journey!
Thanks to the cybersecurity partnership between Paris 2024 and Cisco, I was able to integrate into the Cisco Crew, to operate the NOC/SOC as a Threat Hunter on the most dangerous network in the world for this European Edition of Black Hat.
My first day, I helped with deploying the network by installing the wireless Meraki APs on the venue, understanding how they were configured and how they could help analysts to identify and locate any client connected to the network that could have a bad behavior during the event, the idea being to protect the attendees if an attack was to spray on the network.
Following this “physical” deployment, I’ve been able to access the whole Cisco Secure environment including Meraki, Secure Malware Analytics, Umbrella, SecureX and the other Black Hat NOC partners software tools.
SecureX was definitely the product on which I wanted to step up. By having so fantastic professionals around me, we were able to dig in the product, identifying potential use cases to deploy in the orchestration module and expected integrations for Paris 2024.
Time was flying and so were the attendees to the conference, a network without user is fun but can be quite boring as nothing happens, having so many cybersecurity professional at the same place testing different security malwares, attacks and so on led us to very interesting investigations. A paradox at the Black Hat, we do not want to block malicious content as it could be part of exercises or training classes, quite a different mindset as what we, security defenders, are used to! Using the different components, we were able to find some observables/IOCs that we investigate through SecureX, SecureX being connected to all the other components helped us to enrich the observables (IPs, urls, domains…), understanding the criticality of what we identified (such as malware payloads) and even led us to poke the folks in the training classes to let them know that something really wrong was happening on their devices.
Being part of the Black Hat NOC was an incredible experience, I was able to meet fantastic professionals, fully committed on making the event a success for all attendees and exhibitors. It also helped me to better understand how products, that we use or will use within Paris 2024, could be leveraged to our needs and which indicators could be added to our various Dashboards, helping us to identify, instantaneously, that something is happening.
During the last day of Black Hat Europe, our NOC partner, NetWitness saw some files being downloaded on the network. The integration again automatically carved out the file and submitted the Cisco Secure Malware Analytics (SMA) platform. One of those files came back as a trojan, after SMA detonated the file in a sandbox environment. The specific hash is the below SHA-256:
938635a0ceed453dc8ff60eab20c5d168a882bdd41792e5c5056cc960ebef575
The screenshot below shows some of the behaviors that influenced the decision:
The result of seeing these behaviors caused SMA to give it the highest judgement score available to a detonated file:
After this judgement was made, we connected with the Palo Alto Networks team, and they found the IP address associated with the file download.
Once we had this information, we went to the Meraki dashboard and did a search for the IP address. The search returned only one client that has been associated with the address for the entire Black Hat conference.
Knowing that there has only been one client associated with the address made finding the attendee easier. We then needed to know where they were and Meraki had this figured out. After opening the client’s profile, we saw what SSID and access point (AP) they were connected to using the Meraki location map.
We then found the attendee and let them know to have their IT inspect their laptop to make sure it is clean.
Apart from the technical challenges of running a temporary network for N thousand people, the Black Hat event reminded us that success doesn’t happen without teamwork; that leadership isn’t just about keeping the project on track. It is also about looking after the team and that small details in planning, build up and tear down can be just as important, as having all the right tools and fantastically skilled Individuals using them during the event itself.
In the Cisco Secure technology stack, within the Black Hat NOC, we use SecureX Single Sign-on. This reduces the confusion of managing multiple accounts and passwords. It also streamlines the integrations between the Cisco products and our fellow NOC partners. We have an open ecosystem approach to integrations and access in the NOC, so we will provision Cisco Secure accounts for any staff member of the NOC. Logging into each individual console and creating an account is time consuming and can often lead to confusion on which tools to provision and which permission levels are needed.
To automate this process, I developed two workflows: one to create non-admin users for NOC partners and one to create administrator accounts in all the tools for Cisco staff. The workflows create accounts in SecureX, Secure Malware Analytics (Threat Grid), Umbrella DNS and Meraki dashboard, all using SecureX Single Sign-On.
Here is what the workflow looks like for creating non-admin users.
The workflow requires three inputs: first name, last name, and email. Click Run.
The sequence of API calls is as follows:
Once the workflow has completed successfully, the user will receive four emails to create a SecureX Sign-On account and accept the invitations to the various products. These workflows really improved our responsiveness to account provisioning requests and makes it much easier to collaborate with other NOC partners.
Over the previous Black Hat events, we have been utilizing Meraki scanning data to get location data for individual clients, as they roamed conference. In the initial blog post (Black Hat Asia 2022), we created a Docker container to accept the data from the Meraki Scanning API and save it for future analysis. At Black Hat USA 2022, we wrote about how to use Python Folium to use the flat text files to generate chronological heatmaps that illustrated the density of clients throughout the conference.
This time around, we’ve stepped it up again by integrating Umbrella DNS Security events and adding the ability to track clients across the heatmap using their local IP address.
To improve the portability of our data and the efficiency of our code, we began by moving from flat JSON files to a proper database. We chose SQLite this time around, though going forward we will likely use Mongo.
Both can be queried directly into Python Pandas dataframes which is what will give us the optimal performance we are looking for. We have a dedicated Docker container (Meraki-Receiver) that will validate the incoming data stream from the Meraki dashboard and insert the values into the database.
The database is stored on a Docker volume that can be mounted by our second container, the Meraki-Mapper. Though this container’s primary purpose is building the heatmaps, it also performs the task of retrieving and correlating Umbrella DNS security events. That is, any DNS query from the Black Hat network that matches one of several predefined security categories. Umbrella’s APIs were recently improved to add OAuth and simplify the URI scheme for each endpoint. After retrieving a token, we can get all security events in the time frame of the current heatmap with one call.
What we want to do with these events is to create Folium Markers. These are static “pins” that will sit on the map to indicate where the DNS query originated from. Clicking on a marker will popup more information about the query and the client who sent it.
Thanks to the Umbrella Virtual Appliances in the Black Hat network, we have the internal IP address of the client who sent the DNS query. We also have the internal IP address in the Meraki scanning data, along with the latitude and longitude. After converting the database query into a Pandas dataframe, our logic takes the IP address from the DNS query and finds all instances in the database of location data for that IP within a 5-minute window (the resolution of our heatmap).
What we end up with is a list of dictionaries representing the markers we want to add to the map. Using Bootstrap, we can format the popup for each event to make it look a bit more polished. Folium’s Popup plugin allows for an iFrame for each marker popup.
The result is a moving heatmap covering an entire day on a given conference floor, complete with markers indicating security events (the red pushpin icon).
Clicking on the pushpin shows the details of the query, allowing us in the NOC to see the exact location of the client when they sent it.
To further improve this service during the next conference, we plan to implement a web page where NOC staff can submit an IP address and immediately get map tracking that client through the conference floor. This should give us an even more efficient way to find and notify folks who are either behaving maliciously or appear to be infected.
For years we have been tracking the DNS stats at the Blackhat conferences. The post-pandemic 2022 numbers look like we never skipped a beat after the dip in DNS queries from 2021, seen in the bar graph below. This year’s attendance saw well over 11 million total DNS queries.
The Activity volume view from Umbrella gives a top-level level glance of activity by category, which we can drill into for deeper threat hunting. On trend with the previous Black Hat Europe events, the top Security categories were Dynamic DNS and Newly Seen Domains. However, it’s worth noting a proportionally larger increase in the cryptomining and phishing categories from 9 to 17 and 28 to 73, respectively, compared to last year.
These years, Black Hat saw over 4,100 apps connect to the network, which is nearly double of what was seen last year. However, still not topping over 6,100 apps seen at Black Hat USA early this year.
Should the need arise, we can block any application, such as Mail.ru above.
Black Hat Europe 2022 was the best planned and executed NOC in my experience, with the most integrations and visibility. This allowed us the time to deal with problems, which will always arise.
We are very proud of the collaboration of the team and the NOC partners.
Black Hat Asia will be in May 2023, at the Marina Bay Sands, Singapore…hope to see you there!
Thank you to the Cisco NOC team:
Also, to our NOC partners NetWitness (especially David Glover, Iain Davidson, Alessandro Contini and Alessandro Zatti), Palo Alto Networks (especially James Holland, Matt Ford, Matt Smith and Mathew Chase), Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In this blog about the design, deployment and automation of the Black Hat network, we have the following sections:
Cisco is honored to be a Premium Partner of the Black Hat NOC, and is the Official Network Platform, Mobile Device Management, Malware Analysis and DNS (Domain Name Service) Provider of Black Hat.
2022 was Cisco’s sixth year as a NOC partner for Black Hat Europe. However, it was our first time building the network for Black Hat Europe. We used experiences of Black Hat Asia 2022 and Black Hat USA 2022 to refine the planning for network topology design and equipment. Below are our fellow NOC partners providing hardware, to build and secure the network, for our joint customer: Black Hat.
We are grateful to share that Black Hat Europe 2022 was the smoothest experience we’ve had in the years at Black Hat. This is thanks to the 15 Cisco Meraki and Cisco Secure engineers on site (plus virtually supporting engineers) to build, operate and secure the network; and great NOC leadership and collaborative partners.
To plan, configure, deploy (in two days), maintain resilience, and recover (in four hours) an enterprise class network, took a lot of coordination. We appreciate the Black Hat NOC leadership, Informa and the NOC partners; meeting each week to discuss the best design, staffing, gear selection and deployment, to meet the unique needs of the conference. Check out the “Meraki Unboxed” podcast – Episode 94: Learnings from the Black Hat Europe 2022 Cybersecurity Event
We must allow real malware on the Black Hat network: for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees, and prevent bad actors from using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites.
In addition to the weekly meetings with Black Hat and the other partners, the Cisco Meraki engineering team of Sandro Fasser, Rossi Rosario Burgos, Otis Ioannou, Asmae Boutkhil, Jeffry Handal and I met every Friday for two months. We also discussed the challenges in a Webex space with other engineers who worked on past Black Hat events.
The mission:
Division of labor is essential to reduce mistakes and stay laser focused on security scope. Otis took the lead working on network topology design with Partners. Asmae handled the port assignments for the switches. Rossi ensured every AP and Switch was tracked, and the MAC addresses were provided to Palo Alto Networks for DCHP assignments. Otis and Rossi spent two days in the server room with the NOC partners, ensuring every switch was operating and configured correctly. Rossi also deployed and configured a remote Registration switch for Black Hat.
In the weeks before deployment, our virtual Meraki team member, Aleksandar Dimitrov Vladimirov, and I focused on planning and creating a virtual Wi-Fi site survey. Multiple requirements and restrictions had to be taken into consideration. The report was based on the ExCel centre floor plans, the space allocation requirements from Black Hat and the number of APs we had available to us. Although challenging to create, with some uncertainties and often changing requirements due to the number of stakeholders involved, the surveys AP placement for best coverage ended up being pivotal at the event.
Below is the Signal Strength plan for the Expo Hall Floor on the 5 GHz band. The original plan to go with a dual-Band deployment was adjusted onsite and the 2.4 GHz band was disabled to enhance performance and throughput. This was a decision made during the network setup, in coordination with the NOC Leadership and based on experience from past conferences.
Upon arrival at the ExCel Centre, we conducted a walkthrough of the space that most of us had only seen as a floor plan and on some photos. Thanks to good planning, we could start deploying the 100+ APs immediately, with only a small number of changes to optimize the deployment on-site. As the APs had been pre-staged and added to the Meraki dashboard, including their location on the floor maps, the main work was placing and cabling them physically. During operation, the floor plans in the Meraki Dashboard were a visual help to easily spot a problem and navigate the team on the ground to the right spot, if something had to be adjusted.
As the sponsors and attendees filled each space, in the Meraki dashboard, we were able to see in real-time the number of clients connected to each AP, currently and over the time of the conference. This enabled quick reaction if challenges were identified, or APs could be redeployed to other zones. Below is the ExCel Centre Capital Hall and London Suites, Level 0. We could switch between the four levels with a single click on the Floor Plans, and drill into any AP, as needed.
The Location heatmaps also provided essential visibility into conference traffic, both on the network and footfalls of attendees. Physical security is also an important aspect of cybersecurity; we need to know how devices move in space, know where valuable assets are located and monitor their safety.
Below is the Business Hall at lunchtime, on the opening day of the conference. You can see no live APs in the bottom right corner of the Location heatmap. This is an example of adapting the plan to reality onsite. In past Black Hat Europe conferences, the Lobby in that area was the main entrance. Construction in 2022 closed this entrance. So, those APs were reallocated to the Level 1 Lobby, where attendees would naturally flow from Registration.
The floor plans and heatmaps also helped with the Training, Briefings and Keynote network resilience. Capacity was easy to add temporarily, and we were able to remove it and relocate it after a space emptied.
During our time in the NOC, we had the chance to work with other vendor engineers and some use cases that came up led to interesting collaborations. One specific use case was that we wanted to block wireless clients, that show some malicious or bad behavior, automatically after they have been identified by one of the SOC analysts on the different security platforms, in addition we wanted to show them a friendly warning page that guides them to the SOC for a friendly conversation.
The solution was a script that can be triggered thru the interfaces of the other security products and attaches a group policy thru the Meraki Dashboard, including a quarantine VLAN and a splash page, via the Meraki APIs. This integration was just one of the many collaboration bits that we worked on.
During the first day of training, in the Meraki dashboard Air Marshal, I observed packet flood attacks, against we were able to adapt and remain resilient.
I also observed an AP spoofing and broadcast de-authentication attack. I was able to quickly identify the location of the attack, which was at the Lobby outside the Business Hall. Should the attacks continue, physical security had the information to intervene. We also had the ability to track the MAC address throughout the venue, as discussed in Christian Clasen’s section in part two.
From our experiences at Black Hat USA 2022, we had encrypted frames enabled, blunting the attack.
The Meraki dashboards made it very easy to monitor the health of the network APs and Switches, with the ability to aggregate data, and quickly pivot into any switch, AP or clients.
Through the phases of the conference, from two days of pre-conference setup, to focused and intense training the first two days, and transition to the briefings and Business Hall, we were able to visualize the network traffic.
In addition, we could see the number of attendees who passed through the covered area of the conference, with or without connecting to the network. Christian Clasen takes this available data to a new level in Part 2 of the blog.
As the person with core responsibilities for the switch configuration and uptime, the Meraki dashboard made it very simple to quickly change the network topology, according to the needs of the Black Hat customer.
If you refer back to Black Hat USA 2022, you’d have seen that we had over 1,000 iOS devices to deploy, with which we had several difficulties. For context, the company that leases the devices to Black Hat doesn’t use a Mobile Device Management (MDM) platform for any of their other shows…Black Hat is the only one that does. So, instead of using a mass deployment technology, like Apple’s Automated Device Enrollment, the iOS devices are “prepared” using Apple Configurator. This includes uploading a Wi-Fi profile to the devices as part of that process. In Las Vegas, this Wi-Fi profile wasn’t set to auto join the Wi-Fi, resulting in the need to manually change this on 1,000 devices. Furthermore, 200 devices weren’t reset or prepared, so we had those to reimage as well.
Black Hat Europe 2022 was different. We took the lessons from US and coordinated with the contractor to prepare the devices. Now, if you’ve ever used Apple Configurator, there’s several steps needed to prepare a device. However, all of these can be actions can be combined into a Blueprint:
Instead of there being several steps to prepare a device, there is now just one! Applying the Blueprint!
For Black Hat Europe, this included:
There’s lots of other things that can be achieved as well, but this results in the time taken to enroll and set up a device to around 30 seconds. Since devices can be set up in parallel (you’re only limited by the number of USB cables / ports you have), this really streamlines the enrollment and set up process.
Now, for the future, whilst you can’t Export these blueprints, they are transportable. If you open Terminal on a Mac and type:
cd /Users/<YOUR USER NAME>/Library/Group Containers/K36BKF7T3D.group.com.apple.configurator/Library/Application Support/com.apple.configurator/Blueprints
You’ll see a file / package called something.blueprint This can be zipped up and emailed to some else so, they can then use the exact same Blueprint! You may need to reboot your computer for the Blueprint to appear in Apple Configurator.
As mentioned, the registration / lead capture / session scanning devices are provided by the contractor. Obviously, these are all catalogued and have a unique device code / QR code on the back of them. However, during setup, any device name provisioned on the device gets lost.
So, there’s three things we do to know, without having to resort to using the unwieldy serial number, what devices is what.
In the footnote, you’ll see Device Name and Device Serial in blue. This denotes that the values are actually dynamic and change per device. They include:
On the Lock Screen, it’s now possible to see the device’s name and serial number, without having to flip the device over (A problem for the registration devices which are locked in a secure case) or open systems preferences.
We also had integration with SecureX device insights, to see the security status of each iOS device.
With the ability to quickly check on device health from the SecureX dashboard.
This goes without saying, but the iOS devices (Registration, Lead Capture and Session Scanning) do have access to personal information. To ensure the security of the data, devices are wiped at the end of the conference. This is incredibly satisfying, hitting the Erase Devices button in Meraki Systems Manager, and watching the 100+ devices reset!
Deploying a network like Black Hat takes a lot of work, and repetitive configuration. Much of this has been covered in previous blogs. However, to make things easier for this event, instead of the 60 training SSIDs we had in Black Hat US 2022, the Meraki team discussed the benefits of moving to iPSKs with Black Hat NOC Leadership, which accepted the plan.
For context, instead of having a single pre shared key for an SSID, iPSK functionality allows you to have 1000+. Each of these iPSKs can be assigned its own group policy / VLAN. So, we created a script:
This only involves five API calls:
The bulk of the script is error handling (The SSID or network doesn’t exist, for example) and logic!
The result was one SSID for all of training: BHTraining, and each classroom had their own password. This reduced the training SSIDs from over a dozen and helped clear the airwaves.
Thank you to the Cisco NOC team:
Also, to our NOC partners NetWitness (especially David Glover, Iain Davidson, Alessandro Contini and Alessandro Zatti), Palo Alto Networks (especially James Holland, Matt Ford, Matt Smith and Mathew Chase), Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In our last blog, we gave a rundown of what the Telecommunications (Security) Act (TSA) is, why it’s been introduced, who it affects, when it starts, and how firms can prepare. Here, we take a closer look into the themes introduced by the Act, explore how the telecoms industry can explore zero trust to further improve its security posture, and outline the benefits that can be gained when complying.
When the Telecoms Security Act (TSA) was introduced, it was labelled as ‘one of the strongest telecoms security regimes in the world, a rise in standards across the board, set by the government rather than the industry’ by Matt Warman, former Minister of State at the Department for Digital, Culture, Media, and Sport. The industry is certainly feeling the impending impact of the act – with one industry pundit at an event we ran recently describing it as a ‘multi-generational change’ for the sector.
One of the headline grabbers stemming from the Act are the associated fines. With the new powers granted to it by the Act, Ofcom now has the responsibility to oversee operators’ security policies and impose fines of up to 10 percent of turnover or £100,000 a day in case operators don’t comply or the blanket ban of telecoms vendors such as Huawei. Sounds like the typical ‘stick’-based costly compliance messaging that no-one particularly wants to hear, right? But what if the TSA had some ‘carrot’-based business benefits that are much less discussed?
The TSA introduces a new security framework for the UK telecoms sector to ensure that public telecommunications providers operate secure and resilient networks and services and manage their supply chains appropriately. ny of the themes introduced in the code of practice can be aligned with the themes in a zero trust security model, which are also a focus for CISOs.
Zero trust security is a concept (also known as ‘never trust, always verify’) which establishes trust in users and devices through authentication and continuous monitoring of each access attempt, with custom security policies that protect every application. At Duo, our approach to zero trust is:
A crucial point to note here: much like a solution that claims to help with all aspects of the TSA, telecom providers should be wary of any vendor who claims to have a zero-trust product. Both are far much bigger than any ‘silver bullet’ solution purports to offer. But there is a good reason a zero-trust framework has been mandated by the US White House for all federal agencies, and recommended by the Australian Cyber Security Centre (ACSC) and the UK’s National Cyber Security Centre (NCSC).
As well as helping to mitigate the significant cyber risks presented to the telecoms industry, a zero-trust strategy provides many business benefits. Our recent Guide to Zero Trust Maturity shows that:
A robust zero-trust security program includes phishing-resistant multi factor authentication (MFA), access controls for devices and applications, risk-signalling, dynamic authentication, firewalls, analytics, web monitoring and more. As I said previously there is no one answer to zero trust, or indeed the TSA, but getting the basics right like strong MFA, single sign on (SSO) and device trust are an easy and effective way to get started.
The TSA will be a huge undertaking for industry, but it is important to focus on the benefits such a wide-reaching set of regulatory rules will inevitably result in. As another guest from our recent event put it: ‘the TSA is full of the latest and modern best practice around security, so the aim really is to raise the tide and all ships, which can only be a good thing.’
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Watch ThreatWise TV: Explorations in the spam folder
The spam folder: that dark and disregarded corner of every email account, full of too-good-to-be-true offers, unexpected shipments, and supposedly free giveaways.
You’re right to ignore this folder; few good things come from exploring it. But every once in a while one of these misleading, and sometimes malicious, emails manages to evade the filters that normally siphon them off, landing them in your inbox instead.
Fortunately, it’s easy enough to spot these emails if you know what to look for. We’ve investigated this folder once before, showcasing a variety of scams. With the holiday season in full swing, we thought this would be a good time to revisit how scammers are trying to trick unsuspecting users.
The holiday season is traditionally a time when this type of activity increases, and this year is no different. According to research published by credit reporting agency TransUnion, the average daily number of suspected digital fraud attempts was up 82 percent globally between Thanksgiving and Cyber Monday (Nov 24–Nov 28) compared to the rest of the year (Jan 1–Nov 23) and 127 percent higher for transactions originating in the US.
This level of activity makes it all the more important to be aware of these scams. With that in mind, let’s dive into the spam folder to get a picture of the types of campaigns currently circulating.
While much of the spam circulating is innocuous, many emails are phishing attempts, and some are indeed malicious. To explore these scams, we used a dedicated computer, segmented from the rest of the network, and leveraged Cisco Secure Malware Analytics to safely open the emails before clicking on links or opening attachments. The point being, we do not recommend doing this at home.
By far, the largest category of spam we saw were surveys scams. According to these emails, if you fill out a simple survey you’ll receive “exclusive offers” such as gift cards, smartphones, smart watches, power drills, or even pots and pans.
There are even some campaigns that specifically target the holiday shopping season.
Clicking the links in these emails takes the recipient to sites where they are asked to fill out a survey.
These pages often include fake testimonials that say how easy the survey is and what they did with their free gift.
The surveys are straightforward, comprising 10-20 simple questions that cover demographic information and shopping habits.
After the survey is completed, these sites offer the choice of a handful of rewards. All the recipient must do is pay for shipping. They are then brought to a page where they can fill out shipping and payment information, and the reward is supposedly shipped.
However, the attempts to make payment often appear to fail, or the recipient is informed that the prize is no longer available.
An unsuspecting user may simply give up at this point, disappointed that they won’t be getting their free gift. What they may not be aware of, is that they have just given their credit card details away in a phishing scam.
In their 2021 Internet Crime Report, the Internet Crime Complaint Center (IC3) said that Non-Payment / Non-Delivery scams such as these led to more than $337 million in losses, up from $265 million in 2020. Credit card fraud amounted to $172 million in 2021 and has been climbing continuously at a conservative rate of 15-20 percent since 2019.
According to Cisco Umbrella, many of the sites asking for credit card details are known phishing sites, or worse, host malware.
Another topic that we covered the last time we explored these types of scams was package delivery spam. These continue to circulate today. There are a variety of shipping companies impersonated in these campaigns, and some generic ones as well.
Many of these campaigns claim that a package could not be delivered. If the recipient clicks on a link in an email, they’re brought to a web page that explains that there are outstanding delivery fees that need to be paid.
The recipient is further enticed by suggestions that the package contains a big-ticket item, such as an iPhone or iPad Pro. All the recipient is required to do is enter their credit card details to cover the shipping.
While no outright malicious activity was detected while examining these emails in Secure Malware Analytics, several suspicious behaviors were flagged. Chances are the bad actors behind these campaigns are phishing for credit card details.
Sometimes the simplest approaches can work just as well as the flashiest. This certainly holds true with spam campaigns, given the prominence of plain-text messages.
The topics covered in such emails run the gamut, including medical cures, 419 scams, romance and dating, pharmaceuticals, weight loss, and many of the scam types we’ve already covered. Many of these link to phishing sites, though some attempt to establish a dialog with the recipient, tricking them into sending the scammers money.
The IC3 report says that victims of confidence fraud and romance scams lost $956 million collectively, which is up from $600 million in 2020. Healthcare fraud, such as the miracle pills and prescriptions scams, resulted in $7 million in losses in 2021, but nearly $30 million in 2020. While these types of scams seem generic and easily spotted, they still work, and so it’s important to be aware and avoid them.
Many emails hitting the spam box attempt to trick users of various services into believing that there is a problem with their account. The problems cover all sorts of services, including streaming platforms, email providers, antivirus subscriptions, and even public records.
If the links are clicked, the recipient is presented with landing pages that mimic the respective services. Any details that are entered will likely be phished, leading to account takeover and/or access to personal records. However, some domains encountered in these cases may do more than just steal information, they could deliver malware too.
Another frequently encountered scam surrounds billing. Many of these appear to be unexpected bills for services the recipient never purchased.
These emails include attachments that are designed to look like official invoices. Interestingly, most of the attachments that we looked at this time were harmless. The goal is to get the recipient to call what appears to be a toll-free number.
While we haven’t called any of these numbers, the experience usually unfolds like a standard customer service call. In the end the “agents” simply claim the charges—which never existed in the first place—have been removed. Meanwhile the scammers steal any personal or financial information provided during the call.
While most billing scams we encountered played out as described above, a few did indeed contain malware.
In this example, the email appears to come from an internet service provider, informing us that our monthly bill is ready.
An invoice appears to be attached, stored within a .zip file. If the recipient opens it and double clicks the file within, a command prompt appears.
This may seem unusual to the recipient, especially since no invoice appears, but by this point it’s too late. The file contains a script that launches PowerShell and attempts to download a remote file.
While the remote file was no longer available at the time of analysis, there is a high likelihood it was malicious. But even though we were unable to determine its contents, Secure Malware Analytics flagged the script execution as malicious.
Knowing about prevalent scams, especially during the holiday season, is a first step in guarding against them. Granted the bad actors who distribute these spam campaigns do everything they can to make their scams look legitimate.
Fortunately, there are several things that you can do to identify scams and defend against them:
But even the best of us can be fooled, and when overseeing a large operation it’s more a matter of when, rather than if, someone clicks on the wrong link. There are elements of the Cisco Secure portfolio that can help for when the inevitable happens.
Cisco Secure Malware Analytics is the malware analysis and malware threat intelligence engine behind all products across the Cisco Security Architecture. The system delivers enhanced, in-depth, advanced malware analysis and context-rich intelligence to help better understand and fight malware within your environments. Secure Malware Analytics is available as a standalone solution, as a component in other Cisco Security solutions, and through software-as-a-service (SaaS) in the cloud, on-premises, and hybrid delivery models.
Cisco Secure Email protects against fraudulent senders, malware, phishing links, and spam. Its advanced threat detection capabilities can uncover known, emerging, and targeted threats. In addition, it defends against phishing by using advance machine learning techniques, real time behavior analytics, relationship modeling, and telemetry that protects against identity deception–based threats.
Cisco Umbrella unifies multiple security functions in a single cloud service to secure internet access. By enforcing security at the DNS layer, Umbrella blocks requests to malware before a connection is even established—before they reach your network or endpoints. In addition, the secure web gateway logs and inspects all web traffic for greater transparency, control, and protection, while the cloud-delivered firewall helps to block unwanted traffic.
Cisco Secure Endpoint is a single-agent solution that provides comprehensive protection, detection, response, and user access coverage to defend against threats to your endpoints. The SecureX platform is built into Secure Endpoint, as are Extended Detection and Response (XDR) capabilities. With the introduction of Cisco Secure MDR for Endpoint, we have combined Secure Endpoint’s superior capabilities with security operations to create a comprehensive endpoint security solution that dramatically decreases the mean time to detect and respond to threats while offering the highest level of always-on endpoint protection.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Ampol has been Australia’s leading transport fuel company since 1900. What began over 125 years ago is now an organization that powers a country, operating 1,500 retail stores and stations across ANZ, plus 89 depots for refining and importing fuels and lubricants, and 8,200 employees throughout Australia, New Zealand, the United States, and Singapore. And while Ampol’s history goes back a century, they are a modern organization, using internet of things (IoT) technology across operational and retail locations, with sensors on everything from electric vehicle charging units to fuel tank gauges to transportation trucks to refrigeration units inside retail stores.
As a critical energy provider to a country of over 25 million people, Ampol’s security needed to match its evolving infrastructure. As Satish Chowdhary, Network Enterprise Architect, said, “At Ampol, we have implemented sensor technology across our network: from gauges in the fuel tanks to monitor fuel quality and quantity to sensors that monitor the temperature in various refrigerators across our retail sites to ensure goods stay chilled. It’s critical to manage these devices effectively and securely, and that’s where Cisco comes in…With IoT, a major security risk is posed by dodgy legacy devices left unpatched and vulnerable within your network. Cisco’s TrustSec and VLAN segregation automatically isolate vulnerable devices, not exposing the rest of the network to risks from untrusted devices.”
In addition to securing the IoT that let’s Ampol monitor and manage its critical operations, Cisco was able to create a comprehensive security environment that solved for their three strategic goals.
“Three key components of our cyber-resilient strategy were isolation, orchestration, and rapid recovery. Cisco SecureX nailed all three providing us a single interface to see all security events, and malicious files, thus expediting how fast we can isolate events and recover,” Chowdhary explained. “Before using Cisco Secure, security was a hindrance, not an enabler for our IT team, employees, and even customers,” he added.
In fact, Cisco Secure helped Ampol improve their security posture so much that they were able to quickly pivot during the early days of the pandemic.
“When Covid triggered supply challenges during lockdowns, people not being able to access groceries turned to their local service station convenience stores to get what they needed. For Ampol, maintaining that supply continuity was critical, not just for our business, but for the customers who were relying on us to get their supplies. And all of this was done when many employees were now having to work remotely… This was possible only because we could maintain our revamped locations, staff, clients, and business partners safe on our network – while still maintaining speed and efficiency. Cisco Secure was the ticket to Ampol’s resilience in the face of major change,” Chowdhary said.
In addition to enabling flexibility against supply chain fluctuations, Ampol is readily protected against threats, cyberattacks, and other vulnerabilities. Their Cisco security solution included:
“The major force for our Cisco Secure investment was simplification by integrating the entire Security portfolio…If we ever happen to have a cyber-attack, we can quickly find it and contain it,” Chowdhary said, adding, “The greatest outcome of using Cisco Secure is simplicity at its core. We achieved great efficiency integration, better visibility, and context that’s not hidden across five, ten, or fifteen consoles, and ultimately, greater security outcomes.”
To find out how else Cisco Secure is helping protect Ampol against sophisticated threats and other challenges, read the full Ampol case study.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
After a three-year hiatus, Cisco Live 2022 Melbourne is back! Personally, it will be a special experience, attending with my team and leaders for the first time as the Asia Pacific Cybersecurity Lead.
I will be speaking on the “Top Priorities for IT and Security Leaders in 2023” on 7 December at the Cisco Secure Insights Live, covering key priorities for security leaders as we enter the new year. Please join me at the Security Experience Hub at the World of Solutions. You can also participate virtually.
Today’s businesses require a strong culture of security and resilience that is pervasive throughout the organization to withstand uncertainty and emerge stronger. Hear from our Cisco security leaders on powering resilience across the enterprise in the following presentations:
Dive into 80+ security sessions by experts to uncover best practices to address key challenges, and maximize your technology investments.
Explore the Security Experience Hub and Demo Stand at the World of Solutions Zone for exciting security activities:
For those joining us online, we have the Cisco Secure Insights Live broadcast on all things security, and Cisco Live broadcast covering keynote presentations by industry leaders.
I’m excited for you to see how we can help you achieve security resilience and look forward to meeting you at Cisco Live 2022 Melbourne. I’m ALL IN, are you?
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
SaaS Security Marketing Manager Laura O’Melia has always been interested in living and working internationally. After living in Austin, Texas for twenty years, O’Melia was ready for a new adventure and decided to move to Sydney, Australia with the support and encouragement of her manager and Cisco. The pandemic delayed her plans, but now that O’Melia’s settling into life and work in Australia, she shared how she made the move to work from anywhere and how you can, too.
What do you do?
O’Melia: I am on the Security Marketing team and focus on driving demand for our Zero Trust solution in the Asia-Pacific, Japan and China (APJC) region. I work closely with the Sales teams to do activities that will generate pipeline and educate prospects on our security solutions. I spend time finding new leads and trying new ways to engage with our top prospects while having fun along the way.
What do you like most about working at Cisco?
O’Melia: What I love most about working at Cisco is the amount of positive contributions we get to have on the world, from solving some of the world’s biggest problems around cybersecurity to giving money and resources to others in need. I also love the feeling of empowerment to create my own work/life balance as Cisco allows me the opportunity to have a flexible schedule.
What has been your career journey within Cisco?
O’Melia: I started at Duo Security in 2017. While working in Field Marketing, I was able to gain experience across many different teams. For example, I worked closely with a region in the U.S. as well as the Managed Service Provider team, which is a global team with a completely different business model. The needs differ greatly, from how we report and track metrics to the messaging and offers from one team to the next. I am now working in a very different market that is much larger and includes many more languages, so that brings a new level of understanding to how we show up in the market to achieve business goals.
“Stepping outside of my comfort zone is one of my favorite things to do.”
– Laura O’Melia
What prompted you to relocate from Austin, Texas to Sydney, Australia?
O’Melia: Austin is great and was my home for 20 years, but I still wanted to gain international work experience to learn what it would be like somewhere else and compare it to what I know.
Stepping outside of my comfort zone is one of my favorite things to do, so when I heard Duo was expanding internationally and there was an opportunity in Australia, I was immediately interested. Everyone I know that has visited Australia always has absolutely wonderful things to say, so without ever having visited I agreed to take a long-term international two-year assignment.
How has Cisco supported your relocation?
O’Melia: I worked closely with my manager on the process from start to finish. We had the support of Cisco’s Mobility Services team, a group of Cisco employees that help with relocation services. We worked with immigration to obtain my work visa. I was planning to relocate in March 2020 but as we know, the borders were closed and visas were not being processed for nearly two years. I was already in-role, so continued to support the APJC team from Austin.
When the time came, Cisco had a team of experts that I worked with to pack and ship my belongings and help get set up with an overseas bank. I also worked with a realtor to help me find a place to live, and the team even assists with my U.S. and Australian tax returns while I am away.
How has your work changed since relocating?
O’Melia: My role has expanded from doing lead-gen events for Duo in Australia and New Zealand to now being responsible for driving demand across the APJC region through digital campaigns and other marketing channels. I still strive to provide qualified leads to Sales and educate the market on our offerings. My goal is to help get Cisco Secure solutions into more doors to ultimately give users a better experience and stop the bad actors from doing harm.
What advice do you have for others who want to work from anywhere?
O’Melia: If you get the opportunity, take it. Everyone has their own path, but if you feel your career could benefit, even slightly, from the experience you will gain moving to another country and figuring things out far from what you know today, why not give it a try? You can learn so much from meeting and working with people that have a very different experience than you might know.
Ready for an adventure? If you want to solve global challenges through cybersecurity with the potential to work anywhere, check out our open roles.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In November 2020, the Telecommunications (Security) Bill was formally introduced to the UK’s House of Commons by the department for Digital, Culture, Media & Sport. Now, after several readings, debates, committee hearings, and periods of consultation, the Telecommunications (Security) Act is quickly becoming reality for providers of public telecoms networks and services in the UK, going live on 1 October 2022. Here, we outline what exactly the requirements mean for these firms, and what they can do to prepare.
The Act outlines new legal duties on telecoms firms to increase the security of the entire UK network and introduces new regulatory powers to the UK Telecoms regulator OFCOM to regulate Public Telecommunications Providers in the area of cyber security. It place obligations on operators to put in place more measures around the security of their supply chains, which includes the security of the products they procure. The Act grants powers to the Secretary of State to introduce a so-called Code of Practice. It is this Code of Practice which contains the bulk of the technical requirements that operators must comply with. Those not in compliance face large fines (up to 10% of company turnover for one year).
Following the UK Telecoms Supply Chain review in 2018, the government identified three areas of concern that needed addressing:
Following the review, little did we know a major resilience test for the telecoms industry was about to face significant challenges brought on by the Covid-19 pandemic. Data released by Openreach – the UK’s largest broadband network, used by customers of BT, Plusnet, Sky, TalkTalk, Vodafone and Zen – showed that broadband usage more than doubled in 2020 with 50,000 Petabytes (PB) of data being consumed across the country, compared to around 22,000 in 2019.
There is no question the security resilience of the UK telecoms sector is becoming ever more crucial — especially as the government intends to bring gigabit capable broadband to every home and business across the UK by 2025. As outlined in the National Cyber Security Centre’s Security analysis for the UK telecoms sector, ‘As technologies grow and evolve, we must have a security framework that is fit for purpose and ensures the UK’s Critical National Telecoms Infrastructure remains online and secure both now and in the future’.
The legislation will apply to public telecoms providers (including large companies such as BT and Vodafone and smaller companies that offer telecoms networks or services to the public). More specifically to quote the Act itself:
As the requirements are long and varied and so the timelines to comply have been broken down to help organisations comply. The current Code of Practice expects Tier 1 providers to implement ‘the most straightforward and least resource intensive measures’ by 31 March 2024, and the more complex and resource intensive measures by 31 March 2025.
Tier 2 firms have been given an extra two years on top of the dates outlined above to reflect the relative sizes of providers. Tier 3 providers aren’t in scope of the regulatory changes currently but are strongly encouraged to use the Code of Practice as best practice. The Code of Practice also expects that these firms ‘must continue to take appropriate and proportionate measures to comply with their new duties under the Act and the regulations’.
The TSA introduces a range of new requirements for those in the telecoms industry to understand and follow. These will require a multi-year programme for affected organisations. An area of high focus for example will be on Third Party controls and managing the relationship with them.
However there are more common security requirements as well. From our work with many companies across many different industries, we know that establishing that users accessing corporate systems, data and applications are who they say they are is a key aspect of reducing risk by limiting the possibility of attacks coming in through the front door. This is a very real risk highlighted in Verizon’s 2022 Data Breaches Investigations Report, which states that around 82% of data breaches involved a human element, including incidents in which employees expose information directly or making a mistake that enables cyber criminals to access the organisation’s systems.
Therefore, one area to start to try and protect the organisation and take a step on the way to compliance is to build up authentication and secure access to systems, data and applications. However even this can take time to implement over large complex environments. It means gaining an understanding of all devices and ensuring there is a solid profile around them, so they can be reported on, attacks can be blocked and prevented, and access to applications can be controlled as needed.
We will be creating more information around the Act as we move closer to the deadlines, including part two of this blog where we will take a deeper dive into themes introduced by the bill, how it compare with other industries’ and jurisdictions’ cyber security initiatives, and explore what else the telecoms industry can do to improve its security posture.
We are also running an event in London on 13 October: ‘Are you ready for TSA?’ which will include peer discussions where participation is welcome on the TSA. If you are interested in attending, please register here.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Written by Martin Lee and Richard Archdeacon.
Lloyds of London have recently published a Market Bulletin1 addressing the wording of cyber insurance policies to exclude losses arising from:
“state backed cyber-attacks that (a) significantly impair the ability of a state to function or (b) that significantly impair the security capabilities of a state.”
The concern raised is that this sort of attack will produce losses that the market cannot absorb. Most insurance policies already include provisions that exclude the consequences of armed conflict. Applying these to potential cyber warfare is a logical step.
The bulletin includes the tenet to:
“set out a robust basis by which the parties agree on how any state backed cyber- attack will be attributed to one or more states.”
What should the CISO be thinking of when reviewing such an exclusion clause, how can we clearly define this key term and what issues may arise?
Attribution is the science of identifying the perpetrator of a crime. In cyber attacks, this is arrived at by comparing the evidence gathered from an attack with evidence gathered from previous attacks that have been attributed to known perpetrators to identify similarities.
In practice, statements of attributions are carefully phrased. Rarely is evidence clear-cut. Frequently attribution is labelled as being ‘consistent with’ a threat actor, or wrapped in words of estimative probability such as ‘highly likely’, ‘probably’, ‘possibly’ etc.
The malicious actors who conduct cyber attacks are referred to as threat actors. The cyber research community identifies and keeps track of the actions of these threat actors, publishing compendia of known actors such as those made available by MITRE2 or Malpedia3.
Rarely do threat actors identify their true identities, they may actively try to confuse or frustrate attribution. Many of the named groups may be synonyms of other groups, equally many of the chains of evidence used to attribute groups may be incorrect. The compendia of threat actors should not be considered as reaching the evidence threshold of “beyond reasonable doubt”.
Some identified threat actor groups are assumed to be criminal gangs due to the nature of their activity. Others appear to be conducting attacks solely to further the geopolitical aims of a nation state and are assumed as being state sponsored or state backed. Some of these groups have been able to be associated with specific national intelligence agencies or state apparatus.
The following are four practical factors to consider when setting out a robust basis for attribution of attacks in a contractual basis.
Step 1 – Collect forensic evidence.
No attribution of an attack can be made without forensic evidence. CISOs should ensure that they are able to gather forensic evidence from attacks to identify as much information as possible regarding how an attack was carried out, and the infrastructure used by the attacker. This requires a basic level of security telemetry gathering with the ability to secure and query this data.
This forensic capability, how evidence will be gathered and preserved, should be agreed with the insurer. However, both parties must bear in mind that attackers may destroy or tamper with evidence, and in the urgency of halting an attack, forensic evidence may be compromised or omitted.
The CISO should be prepared to discuss internally with senior executives the possibly competing priorities of stopping an attack versus collecting good forensic evidence.
Step 2 – Define how attribution will be made.
The attribution of a specific attack must be made by comparing evidence gathered from the attack with that of previous attacks. CISOs should agree the process by which forensic artifacts are used to attribute attacks and the degree of certitude necessary to declare an attack as having been carried out by a specific group.
The set of organisations trusted to assert attribution should be agreed. Attribution made by national bodies such as NCSC, CISA or ENISA may be assumed to be reliable, as may those made by major security vendors (such as Cisco) with expertise and resources that a CISO will never have inhouse. However, anyone can suggest attribution. CISOs should be certain to insist on the exclusion of assertions that have not been confirmed by a trusted entity.
This raises the question as to whether a trusted organisation would be prepared to support their attribution in a scenario where they would have to expose their intelligence sources and methodologies to examination. Attribution may be based on classified intelligence, or made according to ‘fair efforts’ that fall below the legal threshold of “on the balance of probabilities.”
Step 3 – Consider the volatility of attribution.
The gathering of evidence and intelligence is a continuing process. Information previously assumed to be fact may be subsequently identified as incorrect or a purposeful red herring. New evidence may be identified months or years after an attack that changes the estimated attribution of prior attacks.
CISOs must determine a period after which the attribution of attack (if made) will not be changed even if subsequent evidence is uncovered.
Step 4 – Define the nature of state backing.
CISOs should agree what constitutes state backing. Ideally CISOs should agree with their insurers the set of threat actor groups (and their synonyms) which are considered to be ‘state backed’.
State involvement in cyber attacks is a spectrum of activity. Criminal threat actors may be under various degrees of state tolerance or encouragement without being fully backed by a nation state. Some criminal groups may be under partial state direction, acting in a manner akin to privateers. Some state backed actors may indulge in criminal style attacks to boost their coffers.
In any case, criminal and state sponsored actors can easily be confused. They may choose to use the same tools or apply the same techniques to conduct their activities. Non-state threat actors may come into possession of state developed tools which may have been stolen or traded without permission.
Some threat actors may actively resort to influence attribution, either through choice of tooling, or through sock puppet accounts attesting attribution, to increase pressure on CISOs to pay ransoms by influencing if insurance is paid out or not.
The decision line where an attack can be referred to a ‘state backed’ is a fine one that requires consideration and agreement.
Changes bring opportunities, the need for this robust process may cause complications for CISOs. But it is an opportunity for CISOs to review the details of cyber insurance contracts and to hammer out the details of how issues of attribution will be determined.
Lloyd’s Market Association provide sample clauses for insurers4, we intend to consider these in a subsequent blog.
One thing is certain, there will be many opportunities for the legal profession.
The information provided here does not, and is not intended to, constitute legal advice. When negotiating a specific matter, readers should confer with their own legal adviser to obtain advice appropriate for a specific insurance contract issue.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In part one of our Black Hat USA 2022 NOC blog, we discussed building the network with Meraki:
In this part two, we will discuss:
Cisco is a Premium Partner of the Black Hat NOC, and is the Official Wired & Wireless Network Equipment, Mobile Device Management, DNS (Domain Name Service) and Malware Analysis Provider of Black Hat.
Black Hat USA is my favorite part of my professional life each year. We had an incredible staff of 20 Cisco engineers to build and secure the network. Also, for the first time, we had two Talos Threat Hunters from the Talos Incident Response (TIR) team, providing unique perspectives and skills to the attacks on the network. I really appreciated the close collaboration with the Palo Alto Networks and NetWitness team members. We created new integrations and the NOC continued to serve as an incubator for innovation.
We must allow real malware on the network for training, demonstrations, and briefing sessions; while protecting the attendees from attack within the network from their fellow attendees and prevent bad actors using the network to attack the Internet. It is a critical balance to ensure everyone has a safe experience, while still being able to learn from real world malware, vulnerabilities, and malicious websites. So, context is what really matters when investigating a potential attack and bringing so many technologies together in SecureX really accelerated investigation and response (when needed).
All the Black Hat network traffic was supported by Meraki switches and wireless access points, using the latest Meraki gear donated by Cisco. Our Meraki team was able to block people from the Black Hat network, when an investigation showed they did something in violation of the attendee Code of Conduct, upon review and approval by the Black Hat NOC leadership.
Cisco Secure provided all the domain name service (DNS) requests on the Black Hat network through Umbrella, whenever attendees wanted to connect to a website. If there is a specific DNS attack that threatened the conference, we supported Black Hat in blocking it to protect the network. However, by default, we allow and monitor DNS requests to malware, command and control, phishing, crypto mining, and other dangerous domains, which would be blocked in a production environment. That balance of allowing cybersecurity training and demos to occur, but ready to block when needed.
In addition to the Meraki networking gear, Cisco Secure also shipped an Umbrella DNS virtual appliance to Black Hat USA, for internal network visibility with redundancy. The Intel NUC containing the virtual appliance also contained the bridge to the NetWitness on-premises SIEM, custom developed by Ian Redden.
We also deployed the following cloud-based security software:
We analyzed files that were downloaded on the network, checking them for malicious behavior. When malware is downloaded, we confirm it is for a training, briefing or demonstration, and not the start of an attack on attendees.
During an investigation, we used SecureX to visualize the threat intelligence and related artifacts, correlating data. In the example below, an attacker was attempting remote code execution on the Registration Servers, alerted by the Palo Alto team, investigated by the NOC threat hunters, and blocked by order of the NOC leadership upon the results of the investigation.
Cisco Secure Threat Intelligence (correlated through SecureX)
Donated Partner Threat Intelligence (correlated through SecureX)
Open-Source Threat Intelligence (correlated through SecureX)
Continued Integrations from past Black Hat events
New Integrations Created at Black Hat USA 2022
The NOC partners, especially NetWitness and Palo Alto Networks, were so collaborative and we left Vegas with more ideas for future integration development
Creating Custom Meraki Dashboard Tiles for SecureX, by Matt Vander Horst
One of the biggest benefits of Cisco SecureX is its open architecture. Anyone can build integrations for SecureX if they can develop an API with the right endpoints that speak the right language. In the case of SecureX, the language is the Cisco Threat Intelligence Model (CTIM). As mentioned above, Cisco Meraki powered Black Hat USA 2022 by providing wired and wireless networking for the entire conference. This meant a lot of equipment and users to keep track of. To avoid having to switch between two different dashboards in the NOC, we decided to build a SecureX integration that would provide Meraki dashboard tiles directly into our single pane of glass: SecureX.
Building an integration for SecureX is simple: decide what functionality you want your integration to offer, build an internet-accessible API that offers those functions, and then add the integration to SecureX. At Black Hat, our Meraki integration supported two capabilities: health and dashboard. Here’s a summary of those capabilities and the API endpoints they expect:
| Capability | Description | API Endpoints |
| Health | Enables SecureX to make sure the module is reachable and working properly. | /health |
| Dashboard | Provides a list of available dashboard tiles and, after a tile is added to a dashboard, the tile data itself. | /tiles
/tile-data |
With our capabilities decided, we moved on to building the API for SecureX to talk to. SecureX doesn’t care how you build this API if it has the expected endpoints and speaks the right language. You could build a SecureX-compatible API directly into your product, as a serverless Amazon Web Services (AWS) Lambda, as a Python script with Django, and so on. To enable rapid development at Black Hat, we chose to build our integration API on an existing Ubuntu server in AWS running Apache and PHP.
After building the API framework on our AWS server, we had to decide which dashboard tiles to offer. Here’s what we ended up supporting:
| Tile Name | Description |
| Top Applications | Shows the top 10 applications by flow count |
| Client Statistics | Shows a summary of clients |
| Top SSIDs by Usage in GB | Shows the top 10 SSIDs by data usage in GB |
| Access Point Status | Shows a summary of access points |
Finally, once the API was up and running, we could add the integration to SecureX. To do this, you need to create a module definition and then push it to SecureX using its IROH-INT API. After the module is created, it appears in the Available Integration Modules section of SecureX and can be added. Here’s what our module looked like after being added to the Black Hat SecureX instance:
After adding our new tiles to the SecureX dashboard, SecureX would ask our API for data. The API we built would fetch the data from Meraki’s APIs, format the data from Meraki for SecureX, and then return the formatted data. Here’s the result:
These dashboard tiles gave us useful insights into what was going on in the Meraki network environment alongside our existing dashboard tiles for other products such as Cisco Secure Endpoint, Cisco Umbrella, Cisco Secure Malware Analytics, and so on.
If you want to learn more about building integrations with SecureX, check out these resources:
Talos Threat Hunting, by Jerzy ‘Yuri’ Kramarz and Michael Kelly
Black Hat USA 2022 was our first fully supported event, where we deployed an onsite threat hunting team from Talos Incident Response (TIR). Our colleagues and friends from various business units, connected by SecureX integration, granted us access to all the underlying consoles and API points to support the threat hunting efforts enhanced by Talos Intelligence.
The threat hunting team focused on answering three key hypothesis-driven questions and matched that with data modelling across all of the different technology stacks deployed in Black Hat NOC:
To answer the above hypothesis, our analysis started with understanding of how the network architecture is laid out and what kind of data access is granted to NOC. We quickly realized that our critical partners are key to extending visibility beyond Cisco deployed technologies. Great many thanks go to our friends from NetWitness and Palo Alto Networks for sharing full access to their technologies, to ensure that hunting did not stop on just Cisco kit and contextual intelligence could be gathered across different security products.
Daily threat hunt started with gathering data from Meraki API to identify IP and DNS level requests leaving the devices connected to wireless access points across entire conference. Although Meraki does not directly filter the traffic, we wanted to find signs of malicious activity such as DNS exfiltration attempts or connections to known and malicious domains which were not part of the class teaching. Given the level of access, we were then able to investigate network traffic capture associated with suspicious connections and check for suspected Command and Control (C2) points (there were a few from different threat actors!) or attempts to connect back to malicious DNS or Fast Flux domains which indicated that some of the attendee devices were indeed compromised with malware.
That said, this is to be expected given hostility of the network we were researching and the fact that classroom environments have users who can bring their own devices for hands-on labs. SecureX allowed us to quickly plot this internally to find specific hosts which were connecting and talking with malicious endpoints while also showing a number of additional datapoints which were useful for the investigation and hunting. Below is one such investigation, using SecureX threat response.
While looking at internal traffic, we have also found and plotted quite a few different port-scans running across the internal network. While not stopping these, it was interesting to see different tries and attempts by students to find ports and devices across networks. Good thing that network isolation was in place to prevent that! We blurred out the IP and MAC addresses in the image below.
Here is another example of really nice port scan clusters that were running across both internal and external networks we have found. This time it was the case of multiple hosts scanning each other and looking to discovery ports locally and across many of the Internet-based systems. All of that was part of the class but we had to verify that as it looked quite suspicious from the outset. Again, blurred picture for anonymity.
In a few instances, we also identified remarkably interesting clear-text LDAP traffic leaving the environment and giving a clear indicator of which organization the specific device belonged to simply because of the domain name which was requested in the cleartext. It was quite interesting to see that in 2022, we still have a lot of devices talking clear text protocols such as POP3, LDAP, HTTP or FTP, which are easy to subvert via Man-In-The-Middle type of attacks and can easily disclose the content of important messages such as email or server credentials. Below is an example of the plain text email attachments, visible in NetWitness and Cisco Secure Malware Analytics.
In terms of the external attacks, Log4J exploitation attempts were pretty much a daily occurrence on the infrastructure and applications used for attendee registration along with other typical web-based attacks such as SQL injections or path traversals. Overall, we saw a good number of port scans, floods, probes and all kind of web application exploitation attempts showing up daily, at various peak hours. Fortunately, all of them were successfully identified for context (is this part of a training class or demonstration) and contained (if appropriate) before causing any harm to external systems. Given the fact that we could intercept boundary traffic and investigate specific PCAP dumps, we used all these attacks to identify various command-and-control servers for which we also hunted internally to ensure that no internal system is compromised.
The final piece of the puzzle we looked to address, while threat hunting during Black Hat 2022, was automation to discover interesting investigation avenues. Both of us investigated a possibility of threat hunting using Jupyter playbooks to find outliers that warrant a closer look. We have created and developed a set of scripts which would gather the data from API endpoints and create a data frames which could be modeled for further analysis. This allowed us to quickly gather and filter out systems and connections which were not that interesting. Then, focus on specific hosts we should be checking across different technology stacks such as NetWitness and Palo Alto.
Unmistaken Identity, by Ben Greenbaum
An unusual aspect of the Black Hat NOC and associated security operations activities is that this is an intentionally hostile network. People come to learn new tricks and to conduct what would in any other circumstance be viewed rightfully as malicious, unwanted behavior. So, determining whether this is “acceptable” or “unacceptable” malicious behavior is an added step. Additionally, this is a heavily BYOD environment and while we do not want attendees attacking each other, or our infrastructure, there is a certain amount of suspicious or indicative behavior we may need to overlook to focus on higher priority alerts.
In short, there are broadly speaking 3 levels of security event at Black Hat:
When Umbrella alerted us (via a SecureX orchestration Webex workflow) of DNS requests for a domain involved in “Illegal Activity” it was reminiscent of an event at a previous conference where an attendee was caught using the conference network to download forged vaccination documents.
Using the Cisco Secure Malware Analytics platform’s phishing investigation tools, I loaded and explored the subject domain and found it to be a tool that generates and provides pseudo-randomized fake identities, customizable in various ways to match on demographics. Certainly, something that could be used for nefarious purposes, but is not illegal in and of itself. Physical security and access control is, however, also important at Black Hat, and if this activity was part of an effort to undermine that, then this was still a concern.
This is, however, also the kind of thing that gets taught at Black Hat…
Using the reported internal host IP from Umbrella, Meraki’s connection records, and the Meraki access point map, we were able to narrow the activity down to a specific classroom. Looking up what was being taught in that room, we were able to confirm that the activity was related to the course’s subject matter
Network owners and administrators, especially businesses, typically don’t want their network to be used for crimes. However, here at Black Hat what some would consider “crimes” is just “the curriculum”. This adds a layer of complexity to securing and protecting not just Black Hat, but also Black Hat attendees. In security operations, not every investigation leads to a smoking gun. At Black Hat, even when it does, you may find that the smoking gun was fired in a safe manner at an approved target range. Having the right tools on hand can help you make these determinations quickly and free you up to investigate the next potential threat.
25 Years of Black Hat – Musings from the show (and some DNS stats), by Alejo Calaoagan
Back in Singapore, I wrote about cloud app usage and the potential threat landscape surrounding them. My original plan at Black Hat USA was to dig deeper into this vector to see what interesting tidbits I could find on our attendee network. However, given that this was the 25th anniversary of Black Hat (and my 14th in total between Vegas, Singapore, and London), I’ve decided to pivot to talk about the show itself.
I think it’s safe to say, after two difficult pandemic years, Black Hat is back. Maybe it’s the fact that almost everyone has caught COVID by now (or that a lot of people just stopped caring). I caught it myself at RSA this year back in June, the first of consecutive summer super spread events (Cisco Live Vegas was the following week). Both of those shows were in the 15-18k attendee range, well below their pre-pandemic numbers. Black Hat USA 2022 was estimated at 27,000 attendees.
If I remember correctly, 2019 was in the 25-30K range. Last year in Vegas, there were ~3,000 people at the event, tops. 2021 in London, was even lower…it felt like there were less than 1,000 attendees. Things certainly picked up in Singapore (2-3k attendees), though that event doesn’t typically see attendee numbers as high as the other locations. All in all, while the pandemic certainly isn’t over, Las Vegas gave glimpses of what things were like before the “Rona” took over our lives.
The show floor was certainly back to the norm, with swag flying off the countertops and lines for Nike sneaker and Lego giveaways wrapping around different booths. The smiles on people’s faces as they pitched, sold, hustled, and educated the masses reminded me how much I missed this level of engagement. RSA gave me this feeling as well, before COVID sidelined me midway through the show anyway.
Not everything was quite the same. The Black Hat party scene certainly is not what it used to be. There was no Rapid 7 rager this year or last, or a happy hour event thrown by a security company you’ve never heard of at every bar you walk by on the strip. There were still some good networking events here and there, and there were some awesomely random Vanilla Ice, Sugar Ray, and Smashmouth shows. For those of you familiar with Jeremiah Grossman’s annual Black Hat BJJ throwdown, that’s still, thankfully, a thing. Hopefully, in the coming years, some of that old awesomeness returns….
Enough reminiscing, here are our DNS numbers from the show:
From a sheer traffic perspective, this was the busiest Black Hat ever, with over 50 million DNS requests made…
Digging into these numbers, Umbrella observed over 1.3 million security events, including various types of malware across the attendee network. Our threat hunting team was busy all week!
We’ve also seen an increase in app usage at Black Hat:
In a real-world production environment, Umbrella can block unapproved or high-risk apps via DNS.
The increases in DNS traffic volume and Cloud App usage obviously mirrors Black Hat’s return to the center stage of security conferences, following two years of pandemic uncertainty. I’m hopeful that things will continue to trend in a positive direction leading up to London and, hopefully, we’ll see you all there.
——
Hats off to the entire NOC team. Check out Black Hat Europe in London, 5-8 December 2022!
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team.
SecureX threat response, orchestration, device insights, custom integrations and Malware Analytics: Ian Redden, Aditya Sankar, Ben Greenbaum, Matt Vander Horst and Robert Taylor
Umbrella DNS: Christian Clasen and Alejo Calaoagan
Talos Incident Response Threat Hunters: Jerzy ‘Yuri’ Kramarz and Michael Kelley
Meraki Systems Manager: Paul Fidler (team leader), Paul Hasstedt and Kevin Carter
Meraki Network Engineering: Evan Basta (team leader), Gregory Michel, Richard Fung and CJ Ramsey
Network Design and Wireless Site Survey: Jeffry Handal, Humphrey Cheung, JW McIntire and Romulo Ferreira
Network Build/Tear Down: Dinkar Sharma, Ryan Maclennan, Ron Taylor and Leo Cruz
Critical support in sourcing and delivering the Meraki APs and switches: Lauren Frederick, Eric Goodwin, Isaac Flemate, Scott Pope and Morgan Mann
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially Jason Reverri), Lumen, Gigamon, IronNet, and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Jess Stafford and Steve Oldenbourg).
Read Part 1:
Black Hat USA 2022: Creating Hacker Summer Camp
About Black Hat
For 25 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and USA. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Security tools are only as good as the intelligence and expertise that feeds them. We’re very fortunate to have our security technologies powered by Cisco Talos, one of the largest and most trusted threat intelligence groups in the world. Talos is comprised of highly skilled researchers, analysts, and engineers who provide industry-leading visibility, actionable intelligence, and vulnerability research to protect both our customers and the internet at large.
The Talos team serves as a crucial pillar of our innovation — alerting customers and the public to new threats and mitigation tactics, enabling us to quickly incorporate protection into our products, and stepping in to help organizations with incident response, threat hunting, compromise assessments and more. Talos can also be found securing large-scale events such as the Super Bowl, and working with government and law enforcement organizations across the globe to share intelligence.
With Cisco’s vast customer base and broad portfolio — from routers and switches to email and endpoints — Talos has visibility into worldwide telemetry. Once a threat is seen, whether it’s a phishing URL or an IP address hosting malware, detections are created and indicators of compromise are categorized and blocked across our Cisco Secure portfolio.
Talos also leverages its unique insights to help society as a whole better understand and combat the cyberattacks facing us daily. During the war in Ukraine, the group has taken on the additional task of defending over 30 critical infrastructure providers in the country by directly managing and monitoring their endpoint security.
The reality of security today is that organizations must be constantly ready to detect and contain both known and unknown threats, minimize impact, and keep business going no matter what happens in the cyber realm. In light of hybrid work, evolving network architectures, and increasingly insidious attacks, all organizations must also be prepared to rapidly recover if disaster strikes, and then emerge stronger. We refer to this as security resilience, and Talos plays a critical role in helping our customers achieve it.
For several years, our integrated, cloud-native Cisco SecureX platform has been delivering extended detection and response (XDR) capabilities and more. SecureX allows customers to aggregate, analyze, and act on intelligence from disparate sources for a coordinated response to cyber threats.
Through the SecureX platform, intelligence from Talos is combined with telemetry from our customers’ environments — including many third-party tools — to provide a more complete picture of what’s going on in the network. Additionally, built-in, automated response functionality helps to speed up and streamline mitigation. This way, potential attacks can be identified, prioritized, and remediated before they lead to major impact.
For XDR to be successful, it must not only aggregate data, but also make sense of it. Through combined insights from various resources, SecureX customers obtain the unified visibility and context needed to rapidly prioritize the right threats at the right time. With SecureX, security analysts spend up to 90 percent less time per incident.
One of Australia’s largest universities, Deakin University, needed to improve its outdated security posture and transition from ad hoc processes to a mature program. Its small security team sought an integrated solution to simplify and strengthen threat defense.
With a suite of Cisco security products integrated through SecureX, Deakin University was able to reduce the typical investigation and response time for a major threat down from over a week to just an hour. The university was also able to decrease its response time for malicious emails from an hour to as little as five minutes.
“The most important outcome that we have achieved so far is that security is now a trusted function.”
– Fadi Aljafari, Information Security and Risk Manager, Deakin University
Also in the education space, AzEduNet provides connectivity and online services to 1.5 million students and 150,000 teachers at 4,300 educational institutions in Azerbaijan. “We don’t have enough staff to monitor every entry point into our network and correlate all the information from our security solutions,” says Bahruz Ibrahimov, senior information security engineer at AzEduNet.
The organization therefore implemented Cisco SecureX to accelerate investigations and incident management, maximize operational efficiency with automated workflows, and decrease threat response time. With SecureX, AzEduNet has reduced its security incidents by 80 percent.
“The integration with all our Cisco Secure solutions and with other vendors saves us response and investigation time, as well as saving time for our engineers.”
– Bahruz Ibrahimov, Senior Information Security Engineer, AzEduNet
The sophistication of attackers and sheer number of threats out there today make it extremely challenging for most cybersecurity teams to effectively stay on top of alerts and recognize when something requires their immediate attention. According to a survey by ESG, 81 percent of organizations say their security operations have been affected by the cybersecurity skills shortage.
That’s why Talos employs hundreds of researchers around the globe — and around the clock — to collect and analyze massive amounts of threat data. The group uses the latest in machine learning logic and custom algorithms to distill the data into manageable, actionable intelligence.
“Make no mistake, this is a battle,” said Nick Biasini, head of outreach for Cisco Talos, who oversees a team of global threat hunters. “In order to keep up with the adversaries, you really need a deep technical understanding of how these threats are constructed and how the malware operates to quickly identify how it’s changing and evolving. Offense is easy, defense is hard.”
Earlier this year, we unveiled our strategic vision for the Cisco Security Cloud to deliver end-to-end security across hybrid, multicloud environments. Talos will continue to play a pivotal role in our technology as we execute on this vision. In addition to driving protection in our products, Talos also offers more customized and hands-on expertise to customers when needed.
Cisco Talos Incident Response provides a full suite of proactive and emergency services to help organizations prepare for, respond to, and recover from a breach — 24 hours a day. Additionally, the recently released Talos Intel on Demand service delivers custom research unique to your organization, as well as direct access to Talos security analysts for increased awareness and confidence.
Visit our dedicated Cisco Talos web page to learn more about the group and the resources it offers to help keep global organizations cyber resilient. Then, discover how XDR helps Security Operations Center (SOC) teams hunt for, investigate, and remediate threats.
Watch video: What it means to be a threat hunter
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Chief Information Officers and other technology decision makers continuously seek new and better ways to evaluate and manage their investments in innovation – especially the technologies that may create consequential decisions that impact human rights. As Artificial Intelligence (AI) becomes more prominent in vendor offerings, there is an increasing need to identify, manage, and mitigate the unique risks that AI-based technologies may bring.
Cisco is committed to maintaining a responsible, fair, and reflective approach to the governance, implementation, and use of AI technologies in our solutions. The Cisco Responsible AI initiative maximizes the potential benefits of AI while mitigating bias or inappropriate use of these technologies.
Gartner® Research recently published “Innovation Insight for Bias Detection/Mitigation, Explainable AI and Interpretable AI,” offering guidance on the best ways to incorporate AI-based solutions that facilitates “understanding, trust and performance accountability required by stakeholders.” This newsletter describes Cisco’s approach to Responsible AI governance and features this Gartner report.
At Cisco, we are committed to managing AI development in a way that augments our focus on security, privacy, and human rights. The Cisco Responsible AI initiative and framework governs the application of responsible AI controls in our product development lifecycle, how we manage incidents that arise, engage externally, and its use across Cisco’s solutions, services, and enterprise operations.
Our Responsible AI framework comprises:
We base our Responsible AI initiative on principles consistent with Cisco’s operating practices and directly applicable to the governance of AI innovation. These principles—Transparency, Fairness, Accountability, Privacy, Security, and Reliability—are used to upskill our development teams to map to controls in the Cisco Secure Development Lifecycle and embed Security by Design, Privacy by Design, and Human Rights by Design in our solutions. And our principle-based approach empowers customers to take part in a continuous feedback cycle that informs our development process.
We strive to meet the highest standards of these principles when developing, deploying, and operating AI-based solutions to respect human rights, encourage innovation, and serve Cisco’s purpose to power an inclusive future for all.
Check out Gartner recommendations for integrating AI into an organization’s data systems in this Newsletter and learn more about Cisco’s approach to Responsible Innovation by reading our introduction “Transparency Is Key: Introducing Cisco Responsible AI.”
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
During our threat hunting exercises in recent months, we’ve started to observe a distinguishing pattern of msiexec.exe usage across different endpoints. As we drilled down to individual assets, we found traces of a recently discovered malware called Raspberry Robin. The RedCanary Research Team first coined the name for this malware in their blog post, and Sekoia published a Flash Report about the activity under the name of QNAP Worm. Both articles offer great analysis of the malware’s behavior. Our findings support and enrich prior research on the topic.
Raspberry Robin is a worm that spreads over an external drive. After initial infection, it downloads its payload through msiexec.exe from QNAP cloud accounts, executes its code through rundll32.exe, and establishes a command and control (C2) channel through TOR connections.
Let’s walkthrough the steps of the kill-chain to see how this malware functions.
Raspberry Robin is delivered through infected external disks. Once attached, cmd.exe tries to execute commands from a file within that disk. This file is either a .lnk file or a file with a specific naming pattern. Files with this pattern exhibit a 2 to 5 character name with an usually obscure extension, including .swy, .chk, .ico, .usb, .xml, and .cfg. Also, the attacker uses an excessive amount of whitespace/non printable characters and changing letter case to avoid string matching detection techniques. Example command lines include:
File sample for delivery can be found in this URL:
https://www.virustotal.com/gui/file/04c13e8b168b6f313745be4034db92bf725d47091a6985de9682b21588b8bcae/relations
Next, we observe explorer.exe running with an obscure command line argument, spawned by a previous instance of cmd.exe. This obscure argument seems to take the name of an infected external drive or .lnk file that was previously executed. Some of the samples had values including USB, USB DISK, or USB Drive, while some other samples had more specific names. On every instance of explorer.exe we see that the adversary is changing the letter case to avoid detection:
After delivery and initial execution, cmd.exe spawns msiexec.exe to download the Raspberry Robin payload. It uses -q or /q together with standard installation parameter to operate quietly. Once again, mixed case letters are used to bypass detection:
As you can see above, URLs used for payload download have a specific pattern. Domains use 2 to 4 character names with obscure TLDs including .xyz, .hk, .info, .pw, .cx, .me, and more. URL paths have a single directory with a random string 11 characters long, followed by hostname and the username of the victim. On network telemetry, we also observed the Windows Installer user agent due to the usage of msiexec.exe. To detect Raspberry Robin through its URL pattern, use this regex:
^http[s]{0,1}\:\/\/[a-zA-Z0-9]{2,4}\.[a-zA-Z0-9]{2,6}\:8080\/[a-zA-Z0-9]+\/.*?(?:-|\=|\?).*?$
If we look up the WHOIS information for given domains, we see domain registration dates going as far back as February 2015. We also see an increase on registered domains starting from September 2021, which aligns with initial observations of Raspberry Robin by our peers.
| WHOIS Creation Date | Count |
| 12/9/2015 | 1 |
| … | … |
| 10/8/2020 | 1 |
| 11/14/2020 | 1 |
| 7/3/2021 | 1 |
| 7/26/2021 | 2 |
| 9/11/2021 | 2 |
| 9/23/2021 | 9 |
| 9/24/2021 | 6 |
| 9/26/2021 | 4 |
| 9/27/2021 | 2 |
| 11/9/2021 | 3 |
| 11/10/2021 | 1 |
| 11/18/2021 | 2 |
| 11/21/2021 | 3 |
| 12/11/2021 | 7 |
| 12/31/2021 | 7 |
| 1/17/2022 | 6 |
| 1/30/2022 | 11 |
| 1/31/2022 | 3 |
| 4/17/2022 | 5 |
Table 1: Distribution of domain creation dates over time
Associated domains have SSL certificates with the subject alternative name of q74243532.myqnapcloud.com, which points out the underlying QNAP cloud infra. Also, their URL scan results return login pages to QTS service of QNAP:
Once the payload is downloaded, it is executed through various system binaries. First, rundll32.exe uses the ShellExec_RunDLL function from shell32.dll to leverage system binaries such as msiexec.exe, odbcconf.exe, or control.exe. These binaries are used to execute the payload stored in C:\ProgramData\[3 chars]\
It is followed by the execution of fodhelper.exe, which has the auto elevated bit set to true. It is often leveraged by adversaries in order to bypass User Account Control and execute additional commands with escalated privileges [3]. To monitor suspicious executions of fodhelper.exe, we suggest monitoring its instances without any command line arguments.
Raspberry Robin sets up its C2 channel through the additional execution of system binaries without any command line argument, which is quite unusual. That likely points to process injection given elevated privileges in previous steps of execution. It uses dllhost.exe, rundll32.exe, and regsvr32.exe to set up a TOR connection.
In Cisco Global Threat Alerts available through Cisco Secure Network Analytics and Cisco Secure Endpoint, we track this activity under the Raspberry Robin threat object. Image 3 shows a detection sample of Raspberry Robin:
Raspberry Robin tries to remain undetected through its use of system binaries, mixed letter case, TOR-based C2, and abuse of compromised QNAP accounts. Although we have similar intelligence gaps (how it infects external disks, what are its actions on objective) like our peers, we are continuously observing its activities.
| Type | Stage | IOC |
| Domain | Payload Delivery | k6j[.]pw |
| Domain | Payload Delivery | kjaj[.]top |
| Domain | Payload Delivery | v0[.]cx |
| Domain | Payload Delivery | zk4[.]me |
| Domain | Payload Delivery | zk5[.]co |
| Domain | Payload Delivery | 0dz[.]me |
| Domain | Payload Delivery | 0e[.]si |
| Domain | Payload Delivery | 5qw[.]pw |
| Domain | Payload Delivery | 6w[.]re |
| Domain | Payload Delivery | 6xj[.]xyz |
| Domain | Payload Delivery | aij[.]hk |
| Domain | Payload Delivery | b9[.]pm |
| Domain | Payload Delivery | glnj[.]nl |
| Domain | Payload Delivery | j4r[.]xyz |
| Domain | Payload Delivery | j68[.]info |
| Domain | Payload Delivery | j8[.]si |
| Domain | Payload Delivery | jjl[.]one |
| Domain | Payload Delivery | jzm[.]pw |
| Domain | Payload Delivery | k6c[.]org |
| Domain | Payload Delivery | kj1[.]xyz |
| Domain | Payload Delivery | kr4[.]xyz |
| Domain | Payload Delivery | l9b[.]org |
| Domain | Payload Delivery | lwip[.]re |
| Domain | Payload Delivery | mzjc[.]is |
| Domain | Payload Delivery | nt3[.]xyz |
| Domain | Payload Delivery | qmpo[.]art |
| Domain | Payload Delivery | tiua[.]uk |
| Domain | Payload Delivery | vn6[.]co |
| Domain | Payload Delivery | z7s[.]org |
| Domain | Payload Delivery | k5x[.]xyz |
| Domain | Payload Delivery | 6Y[.]rE |
| Domain | Payload Delivery | doem[.]Re |
| Domain | Payload Delivery | bpyo[.]IN |
| Domain | Payload Delivery | l5k[.]xYZ |
| Domain | Payload Delivery | uQW[.]fUTbOL |
| Domain | Payload Delivery | t7[.]Nz |
| Domain | Payload Delivery | 0t[.]yT |
Few security career paths are linear. For Stephanie Frankel the journey to Cisco Secure was circuitous. The Ann Arbor, Michigan native studied journalism at the University of Michigan before managing communications for the Washington Capitals and NBC Sports. But after several stints at communications agencies, she charted a new path for herself in cybersecurity. Not only has her diverse background served as a strength in her current role as senior manager for strategy and operations, but it’s also informed her management philosophy.
After doing project management and account direction at consulting agencies, Frankel was interested in honing her skills and expertise on the client side. She had heard amazing things about Duo and wanted to stay in Ann Arbor and work for a company with local roots. After interviewing, Frankel realized that “working at Duo was a cool, exciting opportunity with a really awesome group of people.”
Frankel was on the ground running working as a technical project manager in research and development overseeing the Multi-Factor Authentication, applications and mobile engineering teams despite not having worked in information security before.
Duo’s security education allowed Frankel to understand the industry and is something she values for getting more people into the cybersecurity field. At Duo and Cisco Secure, employees come from a variety of backgrounds and some don’t have much (or any) experience with cybersecurity.
Robust educational programs build knowledge about security and specific products which empower new team members to grow and learn. Every team also has a learning and development budget for employees to quench their curiosity and enhance their knowledge through courses, books or other programs Manager of Global Employee Programs Anndrea Boris shared.
“People are open to having conversations and open to ideas and ways to solve those ideas. If you have an idea of how to solve a problem, no matter whether it’s your job or not, people are open and willing to listen to you.” – Stephanie Frankel
Something Frankel also appreciates most is that ideas are valued at Duo and Cisco Secure: “Even in my first job, I would have ideas and go to my boss or our head of engineering and say, ‘Hey, I think this could be a really cool opportunity, and I think it needs this.’ People are open to having conversations and open to ideas and ways to solve those ideas. If you have an idea of how to solve a problem, no matter whether it’s your job or not, people are willing to listen to you.”
After a year, Frankel moved from engineering to marketing to run operations for Duo’s in-house brand team, leading the team through a rebrand. “The team really rallied behind this new brand and it was amazing to see their pride and hard work when sharing it,” she said. With Frankel’s leadership, the team showcased not only the new look and feel of the brand but also the customer research that went into understanding the need for the change.
“Our amazing team knew that for it to catch on internally we needed to help people understand the why. The team put together an amazing training and went around the company to help people understand the security buyer, the industry overall and our differentiators and how we could do all of this within the umbrella of Cisco,” she said.
Recognizing that she most enjoys and feels best suited for a strategic operations role, she had open conversations with her manager. “I told my boss, ‘It’s just not a great fit.’” Her manager was very supportive, and they worked through potential options. “You’ll find a lot of that at Cisco,” she said.
Now as senior manager in the Strategy and Operations Group within Cisco’s Security and Collaboration division, Frankel runs key initiatives for business operations that drive business growth. She is empowered to creatively solve problems and collaborate “with all the stakeholders within each group to move these programs forward, to understand the problems we’re looking to solve, create objectives, a program plan, and continue to track metrics and progress towards those ultimate goals,” she said.
A self-described “over communicator,” Frankel believes that as a leader, “the more you communicate and the more transparent you are, the better.” Frankel loves leading people who are experts in their fields and letting them do what they do best.
On the brand team, for example, she trusted her team’s expertise in producing stories, videos and animations to demystify Cisco’s security products.
“All I needed to do was give them the objective and the goals and they were able to come up with the solutions,” Frankel said.
She fondly remembers the boss at one of her first jobs out of college. In that job Frankel wrote press releases and wanted her boss to fully approve the final versions before sending them to the media. Once her boss told her, “Stephanie, if you keep giving it back to me, I will keep finding things to change. I trust you to know when it is ready to go.” That confidence in her so early in her career “gave me so much confidence in myself,” she said.
Frankel emulates his approach to management by recognizing that each employee has different needs in their lives, in their careers, and in how they like to receive feedback. From that boss Frankel first learned that for every piece of negative feedback, you must give four pieces of positive feedback for “someone to actually hear it because that’s how you balance things out in your mind.”
Frankel believes feedback is crucial for growth. “I don’t see how you can improve or grow without it, no matter what level of your career you’re at. Feedback shouldn’t be taken as negative, as much as it is a way for you to improve,” she said.
One of the most helpful things Frankel learned in a Cisco class for managers was the importance of asking a person if they are in a good place to receive critical feedback. “You might not be in the mindset to accept the feedback and to do something constructive with it,” she said. ”If you’re having a bad day or struggling, you could say, ‘You know what, I’m not going to be able to take it today, but let’s talk tomorrow and I’ll be in a better place to receive it.’’’
Frankel has spent the last year thriving in a role she never anticipated in an industry her college training in journalism didn’t fully prepare her for. The secret, she says, is keeping an open mind to new possibilities and a willingness to take on new challenges, even if you don’t feel 100% ready.
“A lot of it is getting real world experience and learning your way through it and knowing that there’s a lot of opportunities and a lot of people that are willing to teach you,” she said.
To pivot professionally Frankel advises not feeling pigeonholed just because you studied a particular topic or have been in a certain industry for a long time. Take what you can from where you started such as storytelling and communications skills in the case of journalism for Frankel. While trying something new may require taking a different level or type of job “sometimes it’s worth it because you have that opportunity to grow and you might find you’re happier somewhere else,” she said.
When discerning professional steps Frankel recommends having open and honest conversations with yourself and others such as mentors.
“Cisco has so many mentorship programs and so many people that are knowledgeable about a lot of things,” she said. ”Just because your current role isn’t a great fit doesn’t mean that there’s not another good fit within the corporation, or it doesn’t mean that you can’t create your own good fit.”
Did you know that Cisco offers cybersecurity trainings and certifications? Start developing your cybersecurity skills today! And if you’re ready to jump into an exciting new career in security, check out the open roles at Cisco Secure and Duo Security.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Managed services are an essential and fast-growing part of the security market, growing 14% annually. This opportunity presents new challenges MSPs must juggle day to day, including onboarding vendors and driving customer acquisition, all while making sure to provide robust IT solutions for your diverse set of clients. Clients are demanding more security and capabilities for a hybrid workforce, which provides a great opportunity for MSPs like you to grow your business.
We love our MSP community and want to help you deliver great security solutions to your clients. After speaking with many of you to understand how Cisco can help unlock growth for your businesses today, we developed a simplified buying model that delivers faster time to value. Cisco Secure MSP was born.
Secure MSP center was launched in the US market in November 2021 and MSPs across America have been rapidly transacting their business on MSP Center. We are excited to announce we are expanding this direct buying experience to Canadian MSPs in local Canadian Dollars for faster time to value and better ROI for your business.
It is a lightning fast and direct buying experience of SaaS security- No invoicing. Straightforward market pricing. And easy click-to-accept agreements. Cisco Umbrella’s market-leading DNS security is currently available with more SaaS security products coming soon.
It’s a simple three-step process that takes just minutes, from signup to deployment.
Step 1 – You can sign up here and login with your Cisco ID (or create one)
Step 2 – Provide billing and credit card information and sign a click-to-accept agreement
Step 3 – Get access to our world class Cisco Umbrella DNS security offer
From here, you can onboard your clients and start providing the first line of defense through Umbrella DNS Security product instantly. Sign up to deployment takes minutes – not hours or days.
From here, you can onboard your clients and start providing the first line of defense through Umbrella DNS Security product instantly. Sign up to deployment takes minutes – not hours or days.
There are no minimums or upfront fees. Your credit card will be charged on the first of the month and you’ll receive a detailed invoice. This is a simple, no hassle, and post-paid consumption-based model.
Other perks include a dedicated partner account manager alongside our sales engineer, who will help you not only with product deployments but also work with you to grow your business. We also have an MSP specialist team to answer your questions.
Partners currently using Secure MSP Center have had great things to say –
“Wow, this was a much easier process than I thought it would be”
“I’m glad Cisco created a program and process that was this simple”
“I thought this would be more complicated”
“That’s all there is to it?”
So, what are you waiting for? Come and take the first step in simplifying security offers for your clients. Sign up here: cisco.com/go/securemsp.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
Cisco Talos has a long-standing relationship with Ukraine, so when Russia invaded the country earlier this year, things hit close to home. Cisco Talos leaders rallied together to provide cybersecurity threat hunting to vital infrastructure, humanitarian support and goods and services to employees and their families in the region.
Ashlee Benge, Amy Henderson and Sammi Seaman spearheaded initiatives to support and sustain Ukrainian employees and threat hunters working around-the-clock to prevent cyberattacks and remember the human element. Even in the midst of crisis, they’ve facilitated open communication, emphasized mental health and cultivated connection.
Given Ukraine’s unique position on the front lines of cyberwarfare, Cisco Talos has had a very close partnership with Ukraine. The threat intelligence team has worked with several partners in the country from a cyber threat perspective. That long standing connection is part of why Russia’s invasion of Ukraine has been felt so deeply. “Some Ukrainian team members evacuated before the invasion, others did not,” said Amy Henderson, head of strategic planning & communications. “Our teams of threat hunters have been around-the-clock hunting in the data since the invasion. They’re stopping attacks from happening.”
Cisco Talos set up Cisco Secure Endpoint on about thirty partners’ organizations and extended the offering to critical infrastructure organizations in Ukraine such as hospitals, directly monitoring Cisco Secure Endpoint, “because their people are busy doing other things right now. They can’t sit at a screen,” Henderson said.
Lead of Strategic Business Intelligence Ashlee Benge directs the Ukraine Threat Hunting Task Unit which requires empathy, compassion and an awareness of the needs of forty-five threat hunters. Veteran threat hunters with decades of experience have volunteered to contribute to the team while other members of Cisco Talos have also volunteered their skill sets to the work. Benge values the distinct contributions of her team members and describes them as, “quite brilliant and very good at their jobs. Talos does a really good job of hiring good people, and so the worst thing that I could do is get in their way.” Getting in their way looks different for different team members which is why Benge has established trainings and consistent ways to evaluate that the needs of her team are being met.
The nature of such a demanding, on-going situation coupled with the team’s dedication can lead employees to work themselves into the ground. To combat this, leaders maintain weekly check-ins that include asking employees how they’re taking care of themselves and checking for signs of burnout. “When you have rest you’re at peak performance and can problem solve. But when you start burning out and get to be irritable and snappy, you’re not able to problem solve. Just step back. You’ll be in a much better head space,” Henderson advises.
Stepping back has meant rotating projects to level out activity levels and urgency. Leaders have also stepped in to ensure employees take time off and that when they’re away, they’re fully away. “When you’re in such a high intensity environment it takes two to three days just to come off of that. If you’re only taking a day here or day there, you’re not even scratching the surface of coming down. So I’ll suggest maybe you need to take a week and completely recharge,” Henderson says.
Team Lead of Employee Experience Sammi Seaman was heartened by Cisco’s support of Ukrainian employees including helping employees and their families out of cities and into new housing. The humanitarian focus led Seaman to ask “How else can we help? Our colleagues have had to leave their homes and they’re still trying to do work. How do I get them necessities like medicine and shampoo?”
Seaman’s empathy and collaboration within her team and with Cisco Talos leadership led to determining the highest needs including more stable internet and navigating the transport of goods directly to employees and their families through freight mail. Seaman worked with her team to ensure necessary items like medical kits could get directly to people who needed them as quickly as possible. There are also pages available coordinating housing, transportation and other forms of support.
“It’s been interesting to think about people needing medicine for various reasons and that I’m also buying Legos and castles so that the children who have been displaced have toys and things that bring them joy and allow them to be kids in this situation,” Seaman said.
As Seaman prepared more boxes to ship, an employee shared a photograph of his daughter with some of the things Seaman had sent. “I just started crying. It was such a relief.” A relief she wanted to share, leaving the boxes for a moment to connect with other team members around the positive impact of their hard work.
“Despite all of these things that are happening around us that are horrific and awful and things that shouldn’t be happening, there are still things that we can celebrate. We’re still humans who have feelings, relationships, milestones and holidays.” – Sammi Seaman
Remembering children also became important during spring holidays. Through asking employees if they celebrated Easter and if they’d like Easter baskets, she learned that many employees celebrated traditional Orthodox Ukrainian Easter and would appreciate the baskets.
Seaman’s colleague researched what people in Ukraine typically put in their Easter baskets and together they made the baskets, boxed them up and shipped them. “The baskets weren’t a necessity but were nice to remind people that despite all of these things that are happening around us that are horrific and awful and things that shouldn’t be happening, there are still things that we can celebrate. We’re still humans who have feelings, relationships, milestones and holidays.”
Outside of work, Benge competes as an Olympic weightlifter. After months of training, her first national level meet was scheduled to happen early into the war in Ukraine. She considered withdrawing given the 24/7 nature of Cisco Talos’ response. However, “only because of the support of those around me,” Benge decided to compete—while working from her phone in the warm up room between lifts. The physical movement allows Benge to manage her mental health and stress while modeling self-care for the team: “If I can’t be my own best self, then the people around me can hardly be expected to do the same.”
Self-care and mental health are so important to the team that Henderson and Benge recently joined their colleagues, Matt Olney, the director of threat intelligence and interdiction, and Strategic Communications Leader Mitch Neff on a Cisco Secure podcast about mental health. The conversation illuminated the importance of reaching out for help, utilizing support systems such as those provided by Cisco and talking to someone including a therapist.
“Using those types of resources is a valuable thing, particularly when managing very high levels of stress and anxiety that come with cybersecurity. No matter what kind of support it is that we need, it’s important to take that time and recognize that it’s valuable to invest in your own mental health,” Benge stated.
Seaman shared that because it can be hard to ask for help or delegate, when she does, she gives herself a pat on the back. She advises that especially in crisis situations it’s important to remember that while things need to get done, it’s not entirely on you to get those things done. “The leadership at Cisco Talos has really emphasized that you’re not alone. The employee assistance program has been a great resource and I’ve got a therapist that I talk to about these things and make sure that I’m taking care of myself so that I can continue to take care of others.”
The team’s bond and purpose run deep. “We care deeply about everyone that we work with. It’s okay to not be on at all times. It’s okay to feel sad and it’s okay to feel anxious. One of the things that I’ve loved about working with Cisco Talos, especially during these more difficult things, is that everybody’s got your back and they make it a safe space to share those feelings. I truly feel like the people I work with are like my family. We’re curated an environment where we can all talk about what we’re going through.”
To learn more about Cisco Talos, Cisco Secure and Duo Security and how you can apply your empathy, skills and passion to make a difference in cybersecurity, check out open roles.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
This article is part of a series in which we will explore several features, principles, and the building blocks of a security detection engine within an extended detection and response (XDR) solution.
In this second installment, we will look at ways of structuring the presentation of machine-generated alerts, so that each alert offers a cohesive and compelling narrative, as if written by a human analyst, at scale and in realtime.
In cyber security, we are used to two types of stories.
The first story is common for reports written by humans. It contains sections such as “impact,” “reproduction,” and “remediation” to help us understand what is at stake and what we need to fix. For example:
IMPACT: An SSH server which supports password authentication is susceptible to brute-forcing attacks.
REPRODUCTION: Use the `ssh` command in verbose mode (`ssh -v`) to determine supported authentication methods. Look for “keyboard-interactive” and “password” methods.
REMEDIATION: Disable unneeded authentication methods.
The second story comes from machine detections. It is much terser in content and sometimes leaves us scratching our heads. “Malware,” the machine says with little explanation, followed by a horde of gibberish-looking data of network flows, executable traces, and so on.
The challenge is now to get the best of both worlds: to enhance machine-generated alerts with the richness of human-written reports. The following sections explain how this can be approached.
In our example of a report written by a human, the “reproduction” section would help us understand, from a factual perspective, how exactly the conclusions were derived.
On the other hand, the machine-generated horde of data provides evidence in a very nondescript way. We would need to be smart enough to spot or reverse-engineer what algorithm the machine was following on said data. Most security analysts do not wish to do this. Instead, they attempt to seek the first story type. “Surely, someone must have written a blog or something more descriptive about this already,” they would say. Then, they would copy-paste anything that looks like a searchable term – an IP address, domain, SHA checksum – and start searching it, either on a threat intelligence search site or even a general-purpose search engine.
Having such cryptic machine-generated alerts is leading us to our first two issues: first, when the story is incomplete or misunderstood, it may lead the analyst astray. For example, the security event might involve requests to communicate with an IP address, and the analyst would say, “This IP address belongs to my DNS server, so the traffic is legitimate.” However, the detection engine was really saying, “I suspect there is DNS tunnelling activity happening through your DNS server—just look at the volume.”
Second, when an analyst seeks explanations from elsewhere, the main function of an advanced detection engine — finding novel, localized, and targeted attacks — cannot work. Information on attacks is generally available only after they have been discovered and analyzed, not when they happen initially.
A common approach to remedy this situation is to include a short description of the algorithm. “This detector works by maintaining a baseline of when during the day a user is active and then reports any deviations,” a help dialog would say. “Okay, that’s clever,” an analyst would reply. But this is not enough. “Wait, what is the baseline, and how was it violated in this particular security event?” To find the answer, we need to go back to the horde of data.
To mimic the “reproduction” section of the human-written report, our security events are enriched with an annotation—a short summary of the behavior described by the event. Here are a few examples of such annotated events:
In the first and second cases, the story is relatively straightforward: in the horde of data, successful communication with said hostnames was observed. An inference through threat intelligence associates these hostnames to the Sality malware.
The third line informs us that, on a factual basis, only a communication with an IP address was observed. Further chain of inferences is that this IP address was associated by a passive DNS mechanism to a hostname which is in turn associated to the Sality malware.
In the fourth event, we have an observation of full HTTP URL requests, and inference through a pattern matcher associates this URL to the Sality malware. In this case, neither the hostname nor the IP address is important to the detector.
In all these annotated events, an analyst can easily grasp the factual circumstances and what the detection engine infers and thinks about the observations. Note that whether these events describe benign, malicious, relevant, or irrelevant behavior, or whether they lead to true or false positives, is not necessarily the concern. The concern is to be specific about the circumstances of the observed behavior and to be transparent about the inferences.
When we eventually succeed in explaining the security events, we might not be finished with the storytelling yet. The analyst would face another dilemma. They would ask: “What relevance does this event have in my environment? Is it part of an attack, an attack technique perhaps? What should I look for next?”
In the human-written report, the “impact” section provides a translation between the fact-based technical language of “how” and the business language of “what.” In this business language, we talk about threats, risks, attacker objectives, their progress, and so on.
This translation is an important part of the story. In our previous example about DNS tunnelling, we might want to express that “an anomaly in DNS traffic is a sign of an attacker communicating with their command-and-control infrastructure,” or that “it is a sign of exfiltration,” or perhaps both. The connotation is that both techniques are post-infection, and that there is probably already a foothold that the attacker has established. Perhaps other security events point to this, or perhaps it needs to be sought after by the analyst.
When it is not explicit, the analyst needs to mentally perform the translation. Again, an analyst might look up some intelligence in external sources and incorrectly interpret the detection engine’s message. Instead, they might conclude that “an anomaly in DNS traffic is a policy violation, user error, or reconnaissance activity,” leading them astray from pivoting and searching for the endpoint foothold that performs the command-and-control activity.
We take special attention not to mix these two different dictionaries. Rather, we express separately the factual observations versus the conclusions in the form of threats and risks. Inbetween, there are the various chains of inferences. Based on the complexity, the depth of the story varies, but the beginning and the end will always be there: facts versus conclusions.
This is very similar to how an analyst would set up their investigation board to organize what they know about the case. Here is an elaborate example:
In this case, from top to bottom:
In all points, the “what” and “how” languages are distinguished from each other. Finally, the whole story is stitched together into one alert by using the alert fusion algorithm described in the Intelligent alert management blog post.
Have we bridged the storytelling gap between machine-generated and human-generated reports?
Threat detections need to be narrated in sufficient detail, so that our users can understand them. Previously, we relied on the human aspect—we would need to document, provide support, and even reverse-engineer what the detection algorithms said.
The two solutions, distinguishing the “what/how” languages and the annotated events, provide the bandwidth to transmit the details and the expert knowledge directly from the detection algorithms. Our stories are now rich with detail and are built automatically in real time.
The result allows for quick orientation in complex detections and lowers the time to triage. It also helps to correctly convey the message, from our team, through the detection engine, and towards the analyst, lowering the possibility of misinterpretation.
This capability is part of Cisco Global Threat Alerts, currently available within Cisco Secure Network Analytics and Cisco Secure Endpoint, and has been continually improved based on customer feedback. In the future, it will also be available in Cisco SecureX XDR.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
The SASE landscape is full of vendors. So full, in fact, that the entire SASE vendor market grew 37% in just a year between 2020 and 2021. It’s clear that SASE is on the top of everyone’s minds. Why? SASE is the evolution of networking and security – an architecture that converges them into a single, cloud delivered service. This streamlined approach is key to securing and connecting the always-on, work-from-anywhere modern work model.
Traditional, siloed security solutions aren’t equipped to handle all the challenges that come along with a multi-environment IT reality. The accelerated transition to hybrid work has increased complexity in managing security and connectivity. Businesses face increased cloud adoption, bring-your-own-device connectivity, increasingly complex cybersecurity threats, and the constant change. Businesses have struggled to keep up with the gaps in coverage, and tech vendors have hurried in to fill the space with cloud security and networking options. But not all SASE solutions are created equal.
In the rush to compete in the market for the future of networking and security, vendors positioned themselves as SASE without offering a truly integrated approach that’s critical to SASE success. Many vendors offer cloud security solutions with no native or integrated SD-WAN networking capability. Others aren’t backed with robust threat intelligence that enable them to effectively deliver on threat detection and prevention. Some don’t offer the flexibility and scalability that businesses need to adopt cloud-delivered security. Many don’t offer open, integrated management platforms. Plus, most organizations face monumental complexity – the exact opposite of what SASE should deliver – due to using several different vendors for different security functions.
Dell’Oro Group, the trusted source for market information in the telecommunications, enterprise networks, and data center IT infrastructure industries, recognized Cisco as the SASE Market Share Leader in 2021, with 19% of the total market share by revenue.
“Cisco was the SASE market share leader because of the combined strength of their networking (SD-WAN) and security capabilities (including secure web gateway, cloud access security broker, and zero trust network access),” said Mauricio Sanchez, Research Director, Network Security, and SASE & SD-WAN at Dell’Oro Group.
At Cisco, we began our journey by pioneering network connectivity and offering innovative tech solutions; today, we have the most SD-WAN market share and secure 100% of Fortune 100 companies. From that foundation, we’ve been able to build and deliver award-winning cloud security solutions that, when combined with our networking services, create a robust, complete SASE architecture.
Cisco’s SASE approach combines networking, client connectivity, security, and internet and cloud intelligence capabilities and helps organizations:
The benefits of a SASE model are unlocked by working with a single vendor who can bring together best-in-class networking, security, and internet and cloud intelligence—while offering the flexibility and investment protection to transition to the cloud at your pace.
While Cisco provides a comprehensive SASE framework, we know that everyone’s journey to the cloud is different. Organizations, especially now, are shifting and refining their strategies, particularly when it comes to cybersecurity and the increase of both the number and type of threats that businesses see every day. Cisco can help organizations make the most of their existing security and networking investments, while also offering increased and amplified functionality across their security infrastructure. Wherever you are on your journey to SASE, Cisco Secure has the unparalleled experience and reputation that can support you on your next steps.
Check out our SASE demo to find out how Cisco delivers a simple, secure, and scalable approach to SASE.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
In part one of our Black Hat Asia 2022 NOC blog, we discussed building the network with Meraki:
In this part two, we will discuss:
SecureX: Bringing Threat Intelligence Together by Ian Redden
In addition to the Meraki networking gear, Cisco Secure also shipped two Umbrella DNS virtual appliances to Black Hat Asia, for internal network visibility with redundancy, in addition to providing:
Cisco Secure Threat Intelligence (correlated through SecureX)
Donated Partner Threat Intelligence (correlated through SecureX)
alphaMountain.ai threat intelligence
Open-Source Threat Intelligence (correlated through SecureX)
Continued Integrations from past Black Hat events
New Integrations Created at Black Hat Asia 2022
Device type spoofing event by Jonny Noble
Overview
During the conference, a NOC Partner informed us that they received an alert from May 10 concerning an endpoint client that accessed two domains that they saw as malicious:
Client details from Partner:
Based on the user agent, the partner derived that the device type was an Apple iPhone.
SecureX analysis
Umbrella Investigate analysis
Umbrella Investigate positions both domains as low risk, both registered recently in Poland, and both hosted on the same IP:
Despite the low-risk score, the nameservers have high counts of malicious associated domains:
Targeting users in ASA, UK, and Nigeria:
Meraki analysis
Based on the time of the incident, we can trace the device’s location (based on its IP address). This is thanks to the effort we invested in mapping out the exact location of all Meraki APs, which we deployed across the convention center with an overlay of the event map covering the area of the event:
Further analysis and conclusions
The device name (LAPTOP-8MLGXXXXXX) and MAC address seen (f4:XX:XX:XX:XX:XX) both matched across the partner and Meraki, so there was no question that we were analyzing the same device.
Based on the useragent captured by the partner, the device type was an Apple iPhone. However, Meraki was reporting the Device and its OS as “Intel, Android”
A quick look up for the MAC address confirmed that the OUI (organizationally unique identifier) for f42679 was Intel Malaysia, making it unlikely that this was an Apple iPhone.
The description for the training “Web Hacking Black Belt Edition” can be seen here:
https://www.blackhat.com/asia-22/training/schedule/#web-hacking-black-belt-edition–day-25388
It is highly likely that the training content included the use of tools and techniques for spoofing the visibility of useragent or device type.
There is also a high probability that the two domains observed were used as part of the training activity, rather than this being part of a live attack.
It is clear that integrating the various Cisco technologies (Meraki wireless infrastructure, SecureX, Umbrella, Investigate) used in the investigation of this incident, together with the close partnership and collaboration of our NOC partners, positioned us where we needed to be and provided us with the tools we needed to swiftly collect the data, join the dots, make conclusions, and successfully bring the incident to closure.
Self Service with SecureX Orchestration and Slack by Matt Vander Horst
Overview
Since Meraki was a new platform for much of the NOC’s staff, we wanted to make information easier to gather and enable a certain amount of self-service. Since the Black Hat NOC uses Slack for messaging, we decided to create a Slack bot that NOC staff could use to interact with the Meraki infrastructure as well as Palo Alto Panorama using the SecureX Orchestration remote appliance. When users communicate with the bot, webhooks are sent to Cisco SecureX Orchestration to do the work on the back end and send the results back to the user.
Design
Here’s how this integration works:
Workflow #1: Handle Slash Commands
Slash commands are a special type of message built into Slack that allow users to interact with a bot. When a Slack user executes a slash command, the command and its arguments are sent to SecureX Orchestration where a workflow handles the command. The table below shows a summary of the slash commands our bot supported for Black Hat Asia 2022:
Here’s a sample of a portion of the SecureX Orchestration workflow that powers the above commands:
And here’s a sample of firewall logs as returned from the “/pan_traffic_history” command:
Workflow #2: Handle Interactivity
A more advanced form of user interaction comes in the form of Slack blocks. Instead of including a command’s arguments in the command itself, you can execute the command and Slack will present you with a form to complete, like this one for the “/update_vlan” command:
These forms are much more user friendly and allow information to be pre-populated for the user. In the example above, the user can simply select the switch to configure from a drop-down list instead of having to enter its name or serial number. When the user submits one of these forms, a webhook is sent to SecureX Orchestration to execute a workflow. The workflow takes the requested action and sends back a confirmation to the user:
Conclusion
While these two workflows only scratched the surface of what can be done with SecureX Orchestration webhooks and Slack, we now have a foundation that can be easily expanded upon going forward. We can add additional commands, new forms of interactivity, and continue to enable NOC staff to get the information they need and take necessary action. The goal of orchestration is to make life simpler, whether it is by automating our interactions with technology or making those interactions easier for the user.
Future Threat Vectors to Consider – Cloud App Discovery by Alejo Calaoagan
Since 2017 (starting in Black Hat USA – Las Vegas), Cisco Umbrella has provided DNS security to the Black Hat attendee network, added layers of traffic visibility previously not seen. Our efforts have largely been successful, identifying thousands of threats over the years and mitigating them via Umbrella’s blocking capabilities when necessary. This was taken a step further at Black Hat London 2021, where we introduced our Virtual Appliances to provide source IP attribution to the devices making requests.
Here at Black Hat Asia 2022, we’ve been noodling on additional ways to provide advanced protection for future shows, and it starts with Umbrella’s Cloud Application Discovery’s feature, which identified 2,286 unique applications accessed by users on the attendee network across the four-day conference. Looking at a snapshot from a single day of the show, Umbrella captured 572,282 DNS requests from all cloud apps, with over 42,000 posing either high or very high risk.
Digging deeper into the data, we see not only the types of apps being accessed…
…but also see the apps themselves…
…and we can flag apps that look suspicious.
We also include risk downs breaks by category…
…and drill downs on each.
While this data alone won’t provide enough information to take action, including this data in analysis, something we have been doing, may provide a window into new threat vectors that may have previously gone unseen. For example, if we identify a compromised device infected with malware or a device attempting to access things on the network that are restricted, we can dig deeper into the types of cloud apps those devices are using and correlate that data with suspicious request activity, potential uncovering tools we should be blocking in the future.
I can’t say for certain how much this extra data set will help us uncover new threats, but, with Black Hat USA just around the corner, we’ll find out soon.
Using SecureX sign-on to streamline access to the Cisco Stack at Black Hat by Adi Sankar
From five years ago to now, Cisco has tremendously expanded our presence at Black Hat to include a multitude of products. Of course, sign-on was simple when it was just one product (Secure Malware Analytics) and one user to log in. When it came time to add a new technology to the stack it was added separately as a standalone product with its own method of logging in. As the number of products increased, so did the number of Cisco staff at the conference to support these products. This means sharing usernames and passwords became tedious and not to mention insecure, especially with 15 Cisco staff, plus partners, accessing the platforms.
The Cisco Secure stack at Black Hat includes SecureX, Umbrella, Malware Analytics, Secure Endpoint (iOS clarity), and Meraki. All of these technologies support using SAML SSO natively with SecureX sign-on. This means that each of our Cisco staff members can have an individual SecureX sign-on account to log into the various consoles. This results in better role-based access control, better audit logging and an overall better login experience. With SecureX sign-on we can log into all the products only having to type a password one time and approve one Cisco DUO Multi-Factor Authentication (MFA) push.
How does this magic work behind the scenes? It’s actually rather simple to configure SSO for each of the Cisco technologies, since they all support SecureX sign-on natively. First and foremost, you must set up a new SecureX org by creating a SecureX sign-on account, creating a new organization and integrating at least one Cisco technology. In this case I created a new SecureX organization for Black Hat and added the Secure Endpoint module, Umbrella Module, Meraki Systems Manager module and the Secure Malware Analytics module. Then from Administration à Users in SecureX, I sent an invite to the Cisco staffers that would be attending the conference, which contained a link to create their account and join the Blackhat SecureX organization. Next let’s take a look at the individual product configurations.
Meraki:
In the Meraki organization settings enable SecureX sign-on. Then under Organization à Administrators add a new user and specify SecureX sign-on as the authentication method. Meraki even lets you limit users to particular networks and set permission levels for those networks. Accepting the email invitation is easy since the user should already be logged into their SecureX sign-on account. Now, logging into Meraki only requires an email address and no password or additional DUO push.
Umbrella:
Under Admin à Authentication configure SecureX sign-on which requires a test login to ensure you can still login before using SSO for authentication to Umbrella. There is no need to configure MFA in Umbrella since SecureX sign-on comes with built in DUO MFA. Existing users and any new users added in Umbrella under Admin à Accounts will now be using SecureX sign-on to login to Umbrella. Logging into Umbrella is now a seamless launch from the SecureX dashboard or from the SecureX ribbon in any of the other consoles.
Secure Malware Analytics:
A Secure Malware Analytics organization admin can create new users in their Threat Grid tenant. This username is unique to Malware Analytics, but it can be connected to a SecureX sign-on account to take advantage of the seamless login flow. From the email invitation the user will create a password for their Malware Analytics user and accept the EULA. Then in the top right under My Malware Analytics Account, the user has an option to connect their SecureX sign-on account which is a one click process if already signed in with SecureX sign-on. Now when a user navigates to Malware Analytics login page, simply clicking “Login with SecureX Sign-On” will grant them access to the console.
Secure Endpoint:
The Secure Endpoint deployment at Blackhat is limited to IOS clarity through Meraki Systems Manager for the conference IOS devices. Most of the asset information we need about the iPhones/iPads is brought in through the SecureX Device Insights inventory. However, for initial configuration and to view device trajectory it is required to log into Secure Endpoint. A new Secure Endpoint account can be created under Accounts à Users and an invite is sent to corresponding email address. Accepting the invite is a smooth process since the user is already signed in with SecureX sign-on. Privileges for the user in the Endpoint console can be granted from within the user account.
Conclusion:
To sum it all up, SecureX sign-on is the standard for the Cisco stack moving forward. With a new SecureX organization instantiated using SecureX sign-on any new users to the Cisco stack at Black Hat will be using SecureX sign-on. SecureX sign-on has helped our user management be much more secure as we have expanded our presence at Black Hat. SecureX sign-on provides a unified login mechanism for all the products and modernized our login experience at the conference.
Malware Threat Intelligence made easy and available, with Cisco Secure Malware Analytics and SecureX by Ben Greenbaum
I’d gotten used to people’s reactions upon seeing SecureX in use for the first time. A few times at Black Hat, a small audience gathered just to watch us effortlessly correlate data from multiple threat intelligence repositories and several security sensor networks in just a few clicks in a single interface for rapid sequencing of events and an intuitive understanding of security events, situations, causes, and consequences. You’ve already read about a few of these instances above. Here is just one example of SecureX automatically putting together a chronological history of observed network events detected by products from two vendors (Cisco Umbrella and NetWitness) . The participation of NetWitness in this and all of our other investigations was made possible by our open architecture, available APIs and API specifications, and the creation of the NetWitness module described above.
In addition to the traffic and online activities of hundreds of user devices on the network, we were responsible for monitoring a handful of Black Hat-owned devices as well. Secure X Device Insights made it easy to access information about these assets, either en masse or as required during an ongoing investigation. iOS Clarity for Secure Endpoint and Meraki System Manager both contributed to this useful tool which adds business intelligence and asset context to SecureX’s native event and threat intelligence, for more complete and more actionable security intelligence overall.
SecureX is made possible by dozens of integrations, each bringing their own unique information and capabilities. This time though, for me, the star of the SecureX show was our malware analysis engine, Cisco Secure Malware Analytics (CSMA). Shortly before Black Hat Asia, the CSMA team released a new version of their SecureX module. SecureX can now query CSMA’s database of malware behavior and activity, including all relevant indicators and observables, as an automated part of the regular process of any investigation performed in SecureX Threat Response.
This capability is most useful in two scenarios:
1: determining if suspicious domains, IPs and files reported by any other technology had been observed in the analysis of any of the millions of publicly submitted file samples, or our own.
2: rapidly gathering additional context about files submitted to the analysis engine by the integrated products in the Black Hat NOC.
The first was a significant time saver in several investigations. In the example below, we received an alert about connections to a suspicious domain. In that scenario, our first course of action is to investigate the domain and any other observables reported with it (typically the internal and public IPs included in the alert). Due to the new CSMA module, we immediately discovered that the domain had a history of being contacted by a variety of malware samples, from multiple families, and that information, corroborated by automatically gathered reputation information from multiple sources about each of those files, gave us an immediate next direction to investigate as we hunted for evidence of those files being present in network traffic or of any traffic to other C&C resources known to be used by those families. From the first alert to having a robust, data-driven set of related signals to look for, took only minutes, including from SecureX partner Recorded Future, who donated a full threat intelligence license for the Black Hat NOC.
The other scenario, investigating files submitted for analysis, came up less frequently but when it did, the CSMA/SecureX integration was equally impressive. We could rapidly, nearly immediately, look for evidence of any of our analyzed samples in the environment across all other deployed SecureX-compatible technologies. That evidence was no longer limited to searching for the hash itself, but included any of the network resources or dropped payloads associated with the sample as well, easily identifying local targets who had not perhaps seen the exact variant submitted, but who had nonetheless been in contact with that sample’s Command and Control infrastructure or other related artifacts.
And of course, thanks to the presence of the ribbon in the CSMA UI, we could be even more efficient and do this with multiple samples at once.
SecureX greatly increased the efficiency of our small volunteer team, and certainly made it possible for us to investigate more alerts and events, and hunt for more threats, all more thoroughly, than we would have been able to without it. SecureX truly took this team to the next level, by augmenting and operationalizing the tools and the staff that we had at our disposal.
We look forward to seeing you at Black Hat USA in Las Vegas, 6-11 August 2022!
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team: Aditya Sankar, Aldous Yeung, Alejo Calaoagan, Ben Greenbaum, Christian Clasen, Felix H Y Lam, George Dorsey, Humphrey Cheung, Ian Redden, Jeffrey Chua, Jeffry Handal, Jonny Noble, Matt Vander Horst, Paul Fidler and Steven Fan.
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially James Holland), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, James Pope, Steve Fink and Steve Oldenbourg).
About Black Hat
For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
In part one of this issue of our Black Hat Asia NOC blog, you will find:
Cisco Meraki was asked by Black Hat Events to be the Official Wired and Wireless Network Equipment, for Black Hat Asia 2022, in Singapore, 10-13 May 2022; in addition to providing the Mobile Device Management (since Black Hat USA 2021), Malware Analysis (since Black Hat USA 2016), & DNS (since Black Hat USA 2017) for the Network Operations Center. We were proud to collaborate with NOC partners Gigamon, IronNet, MyRepublic, NetWitness and Palo Alto Networks.
To accomplish this undertaking in a few weeks’ time, after the conference had a green light with the new COVID protocols, Cisco Meraki and Cisco Secure leadership gave their full support to send the necessary hardware, software licenses and staff to Singapore. Thirteen Cisco engineers deployed to the Marina Bay Sands Convention Center, from Singapore, Australia, United States and United Kingdom; with two additional remote Cisco engineers from the United States.
From attendee to press to volunteer – coming back to Black Hat as NOC volunteer by Humphrey Cheung
Loops in the networking world are usually considered a bad thing. Spanning tree loops and routing loops happen in an instant and can ruin your whole day, but over the 2nd week in May, I made a different kind of loop. Twenty years ago, I first attended the Black Hat and Defcon conventions – yay Caesars Palace and Alexis Park – a wide-eyed tech newbie who barely knew what WEP hacking, Driftnet image stealing and session hijacking meant. The community was amazing and the friendships and knowledge I gained, springboarded my IT career.
In 2005, I was lucky enough to become a Senior Editor at Tom’s Hardware Guide and attended Black Hat as accredited press from 2005 to 2008. From writing about the latest hardware zero-days to learning how to steal cookies from the master himself, Robert Graham, I can say, without any doubt, Black Hat and Defcon were my favorite events of the year.
Since 2016, I have been a Technical Solutions Architect at Cisco Meraki and have worked on insanely large Meraki installations – some with twenty thousand branches and more than a hundred thousand access points, so setting up the Black Hat network should be a piece of cake right? Heck no, this is unlike any network you’ve experienced!
As an attendee and press, I took the Black Hat network for granted. To take a phrase that we often hear about Cisco Meraki equipment, “it just works”. Back then, while I did see access points and switches around the show, I never really dived into how everything was set up.
A serious challenge was to secure the needed hardware and ship it in time for the conference, given the global supply chain issues. Special recognition to Jeffry Handal for locating the hardware and obtaining the approvals to donate to Black Hat Events. For Black Hat Asia, Cisco Meraki shipped:
Let’s start with availability. iPads and iPhones are scanning QR codes to register attendees. Badge printers need access to the registration system. Training rooms all have their separate wireless networks – after all, Black Hat attendees get a baptism by fire on network defense and attack. To top it all off, hundreds of attendees gulped down terabytes of data through the main conference wireless network.
All this connectivity was provided by Cisco Meraki access points, switches, security appliances, along with integrations into SecureX, Umbrella and other products. We fielded a literal army of engineers to stand up the network in less than two days… just in time for the training sessions on May 10 to 13th and throughout the Black Hat Briefings and Business Hall on May 12 and 13.
Let’s talk security and visibility. For a few days, the Black Hat network is probably one of the most hostile in the world. Attendees learn new exploits, download new tools and are encouraged to test them out. Being able to drill down on attendee connection details and traffic was instrumental on ensuring attendees didn’t get too crazy.
On the wireless front, we made extensive use of our Radio Profiles to reduce interference by tuning power and channel settings. We enabled band steering to get more clients on the 5GHz bands versus 2.4GHz and watched the Location Heatmap like a hawk looking for hotspots and dead areas. Handling the barrage of wireless change requests – enable or disabling this SSID, moving VLANs (Virtual Local Area Networks), enabling tunneling or NAT mode, – was a snap with the Meraki Dashboard.
While the Cisco Meraki Dashboard is extremely powerful, we happily supported exporting of logs and integration in major event collectors, such as the NetWitness SIEM and even the Palo Alto firewall. On Thursday morning, the NOC team found a potentially malicious Macbook Pro performing vulnerability scans against the Black Hat management network. It is a balance, as we must allow trainings and demos connect to malicious websites, download malware and execute. However, there is a Code of Conduct to which all attendees are expected to follow and is posted at Registration with a QR code.
The Cisco Meraki network was exporting syslog and other information to the Palo Alto firewall, and after correlating the data between the Palo Alto Dashboard and Cisco Meraki client details page, we tracked down the laptop to the Business Hall.
We briefed the NOC management, who confirmed the scanning was violation of the Code of Conduct, and the device was blocked in the Meraki Dashboard, with the instruction to come to the NOC.
The device name and location made it very easy to determine to whom it belonged in the conference attendees.
A delegation from the NOC went to the Business Hall, politely waited for the demo to finish at the booth and had a thoughtful conversation with the person about scanning the network.
Coming back to Black Hat as a NOC volunteer was an amazing experience. While it made for long days with little sleep, I really can’t think of a better way to give back to the conference that helped jumpstart my professional career.
Meraki MR, MS, MX and Systems Manager by Paul Fidler
With the invitation extended to Cisco Meraki to provide network access, both from a wired and wireless perspective, there was an opportunity to show the value of the Meraki platform integration capabilities of Access Points (AP), switches, security appliances and mobile device management.
The first amongst this was the use of the Meraki API. We were able to import the list of MAC addresses of the Meraki MRs, to ensure that the APs were named appropriately and tagged, using a single source of truth document shared with the NOC management and partners, with the ability to update en masse at any time.
On the first day of NOC setup, the Cisco team walked around the venue to discuss AP placements with the staff of the Marina Bay Sands. Whilst we had a simple Powerpoint showing approximate AP placements for the conference, it was noted that the venue team had an incredibly detailed floor plan of the venue. This was acquired in PDF and uploaded into the Meraki Dashboard; and with a little fine tuning, aligned perfectly with the Google Map.
Meraki APs were then placed physically in the venue meeting and training rooms, and very roughly on the floor plan. One of the team members then used a printout of the floor plan to mark accurately the placement of the APs. Having the APs named, as mentioned above, made this an easy task (walking around the venue notwithstanding!). This enabled accurate heatmap capability.
The Location Heatmap was a new capability for Black Hat NOC, and the client data visualized in NOC continued to be of great interest to the Black Hat management team, such as which training, briefing and sponsor booths drew the most interest.
The ability to use SSID Availability was incredibly useful. It allowed ALL of the access points to be placed within a single Meraki Network. Not only that, because of the training events happening during the week, as well as TWO dedicated SSIDs for the Registration and lead tracking iOS devices (more of which later), one for initial provisioning (which was later turned off), and one for certificated based authentication, for a very secure connection.
We were able to monitor the number of connected clients, network usage, the persons passing by the network and location analytics, throughout the conference days. We provided visibility access to the Black Hat NOC management and the technology partners (along with full API access), so they could integrate with the network platform.
Meraki alerts are exactly that: the ability to be alerted to something that happens in the Dashboard. Default behavior is to be emailed when something happens. Obviously, emails got lost in the noise, so a web hook was created in SecureX orchestration to be able to consume Meraki alerts and send it to Slack (the messaging platform within the Black Hat NOC), using the native template in the Meraki Dashboard. The first alert to be created was to be alerted if an AP went down. We were to be alerted after five minutes of an AP going down, which is the smallest amount of time available before being alerted.
The bot was ready; however, the APs stayed up the entire time!
Applying the lessons learned at Black Hat Europe 2021, for the initial configuration of the conference iOS devices, we set up the Registration iPads and lead retrieval iPhones with Umbrella, Secure Endpoint and WiFi config. Devices were, as in London, initially configured using Apple Configurator, to both supervise and enroll the devices into a new Meraki Systems Manager instance in the Dashboard.
However, Black Hat Asia 2022 offered us a unique opportunity to show off some of the more integrated functionality.
System Apps were hidden and various restrictions (disallow joining of unknown networks, disallow tethering to computers, etc.) were applied, as well as a standard WPA2 SSID for the devices that the device vendor had set up (we gave them the name of the SSID and Password).
We also stood up a new SSID and turned-on Sentry, which allows you to provision managed devices with, not only the SSID information, but also a dynamically generated certificate. The certificate authority and radius server needed to do this 802.1x is included in the Meraki Dashboard automatically! When the device attempts to authenticate to the network, if it doesn’t have the certificate, it doesn’t get access. This SSID, using SSID availability, was only available to the access points in the Registration area.
Using the Sentry allowed us to easily identify devices in the client list.
One of the alerts generated with SysLog by Meraki, and then viewable and correlated in the NetWitness SIEM, was a ‘De Auth’ event that came from an access point. Whilst we had the IP address of the device, making it easy to find, because the event was a de auth, meaning 802.1x, it narrowed down the devices to JUST the iPads and iPhones used for registration (as all other access points were using WPA2). This was further enhanced by seeing the certificate name used in the de-auth:
Along with the certificate name was the name of the AP: R**
One of the inherent problems with iOS device location is when devices are used indoors, as GPS signals just aren’t strong enough to penetrate modern buildings. However, because the accurate location of the Meraki access points was placed on the floor plan in the Dashboard, and because the Meraki Systems Manager iOS devices were in the same Dashboard organization as the access points, we got to see a much more accurate map of devices compared to Black Hat Europe 2021 in London.
When the conference Registration closed on the last day and the Business Hall Sponsors all returned their iPhones, we were able to remotely wipe all of the devices, removing all attendee data, prior to returning to the device contractor.
Meraki Scanning API Receiver by Christian Clasen
Leveraging the ubiquity of both WiFi and Bluetooth radios in mobile devices and laptops, Cisco Meraki’s wireless access points can detect and provide location analytics to report on user foot traffic behavior. This can be useful in retail scenarios where customers desire location and movement data to better understand the trends of engagement in their physical stores.
Meraki can aggregate real-time data of detected WiFi and Bluetooth devices and triangulate their location rather precisely when the floorplan and AP placement has been diligently designed and documented. At the Black Hat Asia conference, we made sure to properly map the AP locations carefully to ensure the highest accuracy possible.
This scanning data is available for clients whether they are associated with the access points or not. At the conference, we were able to get very detailed heatmaps and time-lapse animations representing the movement of attendees throughout the day. This data is valuable to conference organizers in determining the popularity of certain talks, and the attendance at things like keynote presentations and foot traffic at booths.
This was great for monitoring during the event, but the Dashboard would only provide 24-hours of scanning data, limiting what we could do when it came to long-term data analysis. Fortunately for us, Meraki offers an API service we can use to capture this treasure trove offline for further analysis. We only needed to build a receiver for it.
The Scanning API requires that the customer stand up infrastructure to store the data, and then register with the Meraki cloud using a verification code and secret. It is composed of two endpoints:
Returns the validator string in the response body
[GET] https://yourserver/
This endpoint is called by Meraki to validate the receiving server. It expects to receive a string that matches the validator defined in the Meraki Dashboard for the respective network.
Accepts an observation payload from the Meraki cloud
[POST] https://yourserver/
This endpoint is responsible for receiving the observation data provided by Meraki. The URL path should match that of the [GET] request, used for validation.
The response body will consist of an array of JSON objects containing the observations at an aggregate per network level. The JSON will be determined based on WiFi or BLE device observations as indicated in the type parameter.
What we needed was a simple technology stack that would contain (at minimum) a publicly accessible web server capable of TLS. In the end, the simplest implementation was a web server written using Python Flask, in a Docker container, deployed in AWS, connected through ngrok.
In fewer than 50 lines of Python, we could accept the inbound connection from Meraki and reply with the chosen verification code. We would then listen for the incoming POST data and dump it into a local data store for future analysis. Since this was to be a temporary solution (the duration of the four-day conference), the thought of registering a public domain and configuring TLS certificates wasn’t particularly appealing. An excellent solution for these types of API integrations is ngrok (https://ngrok.com/). And a handy Python wrapper was available for simple integration into the script (https://pyngrok.readthedocs.io/en/latest/index.html).
We wanted to easily re-use this stack next time around, so it only made sense to containerize it in Docker. This way, the whole thing could be stood up at the next conference, with one simple command. The image we ended up with would mount a local volume, so that the ingested data would remain persistent across container restarts.
Ngrok allowed us to create a secure tunnel from the container that could be connected in the cloud to a publicly resolvable domain with a trusted TLS certificate generated for us. Adding that URL to the Meraki Dashboard is all we needed to do start ingesting the massive treasure trove of location data from the Aps – nearly 1GB of JSON over 24 hours.
This “quick and dirty” solution illustrated the importance of interoperability and openness in the technology space when enabling security operations to gather and analyze the data they require to monitor and secure events like Black Hat, and their enterprise networks as well. It served us well during the conference and will certainly be used again going forward.
Check out part two of the blog, Black Hat Asia 2022 Continued: Cisco Secure Integrations, where we will discuss integrating NOC operations and making your Cisco Secure deployment more effective:
Acknowledgements: Special thanks to the Cisco Meraki and Cisco Secure Black Hat NOC team: Aditya Sankar, Aldous Yeung, Alejo Calaoagan, Ben Greenbaum, Christian Clasen, Felix H Y Lam, George Dorsey, Humphrey Cheung, Ian Redden, Jeffrey Chua, Jeffry Handal, Jonny Noble, Matt Vander Horst, Paul Fidler and Steven Fan.
Also, to our NOC partners NetWitness (especially David Glover), Palo Alto Networks (especially James Holland), Gigamon, IronNet (especially Bill Swearington), and the entire Black Hat / Informa Tech staff (especially Grifter ‘Neil Wyler’, Bart Stump, James Pope, Steve Fink and Steve Oldenbourg).
For more than 20 years, Black Hat has provided attendees with the very latest in information security research, development, and trends. These high-profile global events and trainings are driven by the needs of the security community, striving to bring together the best minds in the industry. Black Hat inspires professionals at all career levels, encouraging growth and collaboration among academia, world-class researchers, and leaders in the public and private sectors. Black Hat Briefings and Trainings are held annually in the United States, Europe and Asia. More information is available at: blackhat.com. Black Hat is brought to you by Informa Tech.
We’d love to hear what you think. Ask a Question, Comment Below, and Stay Connected with Cisco Secure on social!
Cisco Secure Social Channels
FreshRSS