BSides Austin 2024: Balancing AI Safety and Real-World Risks

Austin, Texas, coined the phrase "breakfast taco," introducing this savory morning food to the US. Almost every local eatery in Austin serves up a unique variation on this theme, meaning you get a varied experience from every restaurant. It is reminiscent of how all our IT departments use the same general technologies, yet each has a unique configuration. Securing our applications, networks, and users will, therefore, always present a challenge, with as many considerations as there are ways to build tacos. It is a good thing this year that there was a place to go and enjoy some breakfast tacos while connecting with other security professionals to share tips, techniques, and best practices – BSides Austin 2024.

This year's event saw around 500 offensive, defensive, and purple teaming security practitioners gather at the Commons Learning Center on the J.J. Pickle Research Campus at The University of Texas at Austin. There was a CTF and a Locksport Village, but the majority of attendees were there for the fantastic lineup of talks and in-depth workshops. Over 60 presenters shared their knowledge about defending our IT infrastructure, defeating attackers, and safely leveraging AI to speed up our work.

There was far too much than what can be covered here in this article, so here are just a few highlights from this most recent edition of BSides Austin. 

Defending your assets takes wider, systematic thinking

In his session, "The Whole is more Dangerous than the Sum of its Parts," Damon "ch3f" Small, Principal Consultant at Blue Bastion, shared insights from a penetration test completed over the course of weeks by a team of six specialists—a mix of blue teamers, red teamers, architects, and wireless engineers. Their key takeaway was organizations need to take a more systemic view when it comes to security. 

Too often, organizations focus on individual attack vectors while missing the larger, system-wide issues that enable those attacks. By shifting focus from specific vulnerabilities to the processes and weaknesses that allowed them to exist, teams can craft long-term strategies for improvement.

One of the underlying issues he stressed was credential mismanagement. During this engagement, domain credentials, which are user username/password pairs, were extracted from a file server. This led to the team finding over 1,500 more unique plaintext credentials revealed through tools like Mimikatz. He said while they were there, the client wanted them to focus on how well they hardened the perimeter, but issues like this surfaced as the real cause of their insecurity. 

This project culminated in a three-year infosec plan for the client—a clear example of how breaking might be fast, but fixing requires strategic foresight and patience.

A better open-source tool for better log analysis

Imagine setting up a series of security cameras throughout your facility. In order to see the footage from each source, you will need to manually plug into each individual camera, and you only have a short window to do this, otherwise, the recordings erase themselves. While no one would set up cameras like that today, this is exactly how most organizations currently treat their logs from different providers, according to Ela Dogjani, Associate Threat Researcher at Permiso Security, who discussed this in her session "​​CloudTail: Making Heads or Tails of Selectively Retaining Multi-Cloud Logs (w/o a SIEM!)."Ela said traditional SIEMs often fall short in the face of API throttling, overwhelming data, and limited retention periods.

She also enumerated the challenges of cloud logging: 

• Logs lack remote forwarding, leading to manual data retrieval.
• Retention periods are short, with many systems self-deleting after 90 days.
• API calls generate an unmanageable flood of data, further complicated by throttling.

Fortunately, the open-source community has a free solution to this mess: CloudTail, an open-source solution for selective log retention and analysis. Built with multi-account support in mind, users can do config-based filtering and choose whatever data storage options fit their workflows. While there is a lot of promise that CloudTail could simplify cloud log management hand elp teams focus only on the most critical events, it is still early in the life of the project. Right now, there is only support for AWS and Azure, but the framework to expand it is already available.

Speaking the same language is critical for communicating with executives

Gideon Rasmussen, cybersecurity veteran and current Cybersecurity Management Consultant at Virtual CSO, shared "Selling Security to Executives," giving his hard-learned lessons about communicating security needs to a business-focused audience. He covered everything from influencing funding decisions to establishing cybersecurity governance frameworks.

The key is to always use data to drive the conversations and find shared goals, making budgeting easier to absorb across teams. Gideon said it was critical to visualize the path, showing executives a multi-year security roadmap emphasizing continuity and long-term benefits. He said every annual plan should be turned into a 'multi-generational plan' showing the next 3 years. You might not get every part exactly right, but it paves the way for easier long-term conversations.

He stressed the importance of building relationships. Regular updates and cross-departmental collaborations ensure alignment on shared goals. One path he likes to get a team thinking together is getting everyone together for tabletop exercises and adjusting the difficulty level based on group experience levels. For example, at level 1, he will offer multiple-choice answers to scenarios. In advanced team scenarios, he will inject new twists and specific events as they go, keeping them off balance and forcing them to think creatively.

He left us with the advice that we need always to know our audience, accept you are not a salesperson or a politician, and that you always need to be genuine and ethical as we work. 

Can a Magic 8-ball be considered AI? 

Bobby Kuzma, Director of Offensive Cyber Operations at ProCircular, challenged the audience to rethink their assumptions about AI in his session, "Know no evil, speak no evil, do no evil." He emphasized the need for better alignment between AI capabilities and real-world security needs.

Bobby said one of the hardest challenges is getting the data scope correct. There will always be a finite amount of data that a model can ingest, and it can only tell part of the story. For example, even if you take all the recordings from all the security events ever held, transcribe them, and use them to train an LLM, it still could not contain the whole of what it means to be in the security community.

Ultimately, Bobby does think that AI can help us secure our environments, but only if we keep a functional focus. He stressed that models should prioritize detecting real vulnerabilities over speculative threats. We need to focus AI on speeding our detection capabilities rather than discovering novel attack vectors. His session was a timely reminder that AI safety is not just about protecting systems but ensuring AI itself operates safely within defined parameters.

Working together no matter what 

One of the themes that kept appearing throughout BSides Austin 2024 was balancing technical innovation with pragmatic risk management. Your author was able to highlight this sentiment in a talk on the hidden dangers of AI in the developer workflow, making the point that AI is powerful and, when used correctly, is of great help, but there are some inherent risks. 

One way to minimize the risk in AI, and really all other areas, is to connect with other humans. The more everyone gets on the same page about how to work more securely, the further down the road we will all be in our security journey. One of the best places to meet other like-minded folks and discuss all things security is your local BSides and other security events.