Describing DEF CON is akin to the ancient tale of a group of blind elders describing an elephant. One felt the trunk and said it was like a snake, one felt a leg and said it was like a tree trunk, and one felt the head and said it was like a wall. Everyone who traveled to Las Vegas in the summer heat to experience DEF CON 31 will have their own experiences and describe the event in a unique way.
What follows is just a small sliver of what the GitGuardian team experienced at this legendary event.
AppSec Village
DEF CON is comprised of multiple workshops, talks, capture-the-flag, CTF, competitions, and villages. Villages are dedicated, independently run spaces organized around a specific topic, such as lockpicking, red teaming, blue teaming, cloud technology, biohacking, and application security, AppSec. GitGuardian is proud to have been part of AppSec Village this year, which drew almost 9,000 attendees over three days.
Hunt the Hacker CTF
Inside of AppSec Village, there was a stage for a speaking track, a workshop stage, an ongoing CTF, and multiple activity pods. The activity pods were independent exercises that took on many forms, from hands-on trainings to trivia challenges to CTFs. GitGuardian introduced dozens of participants to using honeytokens in our CTF, "Hunt the Hacker - Detect compromises in your repositories!"
In our exercise, competitors worked to solve the most challenges to defend a GitLab organization that had been compromised by a hacker. The participants deployed honeytokens to determine what repos had been compromised and triangulate who in the org was the adversary. The later stages focused on remediating the incident by removing the threat and rotating exposed keys.
"I love the blue team approach, with an automated attack, because it’s really different from most CTFs I have done."
- Hunt the Hacker participant
Over the three days we ran the pod, we answered many questions. Here are a few common ones that might also help you when hunting for adversaries in your environments.
What is a honeytoken?
Honeytokens are decoy credentials that don't allow any access to any real system or data but instead trigger an alarm when someone attempts to use them. The honeytokens in the CTF were decoy AWS credentials, just like the real ones you can make with GitGuardian Honeytoken. For the sandbox CTF we ran, we only displayed the repository the honeytoken was deployed to when compromised by the automated hacker, but in the real world, GitGuardian Honeytoken would also give you the IP address, user agent, attempted action, and timestamps. These data points will help you quickly mount a defense when a breach or leak happens.
Valid credentials can lurk in your git history
Git is an interesting technology that is almost always self-taught. Everyone has their own preferred way of working with repositories, and there are many routes to the same results. Players needed to search for secrets in the codebase at one point in the CTF. They could do this manually by clicking through all the commits in the repo, which most people immediately found impractical. They could also use tools like ggshield to automate the search process. This path required either copying the repository locally or making a CLI call to the remote repos.
For those using the git clone command or performing a secret scan remotely, they quickly found the valid credentials in past commits. However, they would be stuck if they downloaded the code as a .zip archive. This is because the created archive does not give you the .git folder or the git history, just a copy of the most recently saved state of the files.
If you only look for credentials in your production branch or just in the code in the latest commits, then there is a good chance you have secrets in your git history that need to be remediated. This is exactly why GitGuardian Secret Scan will perform a historical scan of any repository connected to the platform to give you a full report of any credentials hiding in the git logs.
You must balance operational needs with security
In the later stages of the Hunt the Hacker CTF, participants needed to identify and remove the active hacker who had infiltrated the organization without disrupting the "availability score." If a player removed all the users and tokens, they would, in fact, have removed the hacker, but they also would have made it impossible to complete the final challenges, making it impossible to win the CTF.
We wanted to mimic the real-world balancing act of needing to act fast when an attacker is discovered but still maintain production and the CI/CD pipelines that the DevOps teams are counting on. Revoking a credential without understanding the implications or removing access to teams unnecessarily might cause downtime or major disruptions. This is another reason it is important to know when a breach or leak occurs as soon as possible, giving you time to execute your remediation plan to remedy the situation safely in the shortest time possible.
AppSec Village talks
The speaker lineup was fantastic. It featured GitGuardian's Senior Developer Advocate Mackenzie Jackson giving an updated version of his talk, "The attackers guide to exploiting secrets in the universe." He was one of 22 speakers who gave talks on various application security topics. Here are highlights from just a few other sessions.
A keynote conversation with Chris Roberts
The legendary security researcher, and hacker of airplanes, Chris Roberts, kicked off the first day of talks. In his informal chat with the audience titled "From Camels to Collaboration, A Journey Through Technology AND Humans," he walked us through his history at DEF CON, some great advice for building a better community, and shared the story of re-geolocating cows and camels.
He said over the last 20+ years he has been in the security space he has watched the community repeatedly chase technology and forget the humans. We too often lament the user as someone who won't listen or who is "using the product wrong." We look for someone to blame for not grasping our tech choices instead of taking the feedback and finding new ways to empower the users and teams, ultimately delivering the products.
He shared an anecdote of a colleague complaining that the network team was fighting against his project. It turned out he had never actually met with the network team, and they had no idea what he was trying to accomplish. Chris suggested he buy the network team some donuts and apologize. When his colleague did, it got him out of his silo, fixed the relationship issue, and got his project back on track once everyone was on the same page about the project.
He also had some common sense advice for vendors. He first suggested we all go thank them for being involved, as sponsors are needed to make things like DEF CON a reality. But he also charged vendors with needing to be better at finding common ground with technologists and security leads. He gave three specific suggestions:
- Do your research. If you do not customize your outreach to show you know who he is as a CISO and what real-world challenges he is up against, he will likely not engage. This is true of email, phone, and LinkedIn-based outreach.
- Understand CISOs gatekeep their team's time and are very busy throughout the week. He personally sets his Friday afternoons for vendor meetings and said it is common to have such a block on the calendar. It never hurts to ask about this. If he sees merit in the solution, he will pass you along to his team, who are more directly involved in that area; he will pass us on if it is worth their time.
- Expect and accept feedback. If you do get a meeting with a CISO, be prepared to listen more than you talk. He has helped many vendors tune their pitches to get to the point and drop absolutes and promises from their presentations, giving CxOs what they need to hear to quickly evaluate your solution. If you just show up and repeat the same pitch without listening, then you will waste everyone's time.
Generative AI for tuning your WAF
In one of the more technical sessions of AppSec Village, Ankur Chowdhary, Cybersecurity Researcher at Arizona State University presented "Generative Adversarial Network (GAN) based autonomous penetration testing for Web Applications" and walked us through how his team is leveraging machine learning to find and patch vulnerabilities in record time.
For many attendees, this was their first introduction to Generative Adversarial Networks, GANs, an approach to finding exploits by pitting machine learning algorithms against each other in a zero-sum game, where one is always attacking, and one is always improving the defensive rules of their web application firewall, WAF.
The main advantage of this approach is that a GAN can quickly test and apply rules you would think to set by hand and ones that would be too tedious to account for through manual configuration. WAF security is an endless game of cat and mouse. Being able to safely attack your own application with an ever-evolving adversary you control can surface vulnerabilities long before real-world attackers can formulate and attempt exploits.
His team plans to release their finding in a research paper released on the Multidisciplinary Digital Publishing Institute, MDPI website. This is a very worthwhile approach to investigate and one of the more impressive uses of Generative AI in the security space so far.
How not to succeed in DevSecOps
Tanya Janca, who is now the Head of Community and Education at Semgrep and lead of the WeHackPurple community, really needs no introduction. In her session "DevSecOps Worst Practices" she shared many lessons learned from a career working as a developer and security professional in the DevSecOps space, laid out as a list of specific behaviors to avoid.
These bad practices included technology issues, like not tuning your tools to avoid false positives or making test data hard to replicate or discover. They also included a good number of issues around timing and network usage. She said she once ran a DAST tool against the development environments in a company that prevented anyone from pushing new code for a full day. She learned to run tests when there was less need, such as at night and on the weekend.
Most of the issues came down to education and communication with our teams. If you have a tool that no one is properly trained to use, you can not expect anyone to use it properly. She said we must demand better training from our vendors. At the same time, we need to get our teams on board with fixing issues and not just adding them to the backlog. From the perspective of moving features out the door, security issues might get pushed further down the backlog. It is up to us as security professionals to communicate the urgency and importance of these fixes, which requires a human connection and a larger organizational view of the project.
We need to remember that we are working with people, and human beings need positive reinforcement. She said it is not just little kids who like to hear 'good job.' Motivating people to do the right things will always yield better results than blaming and shaming someone for getting it wrong.
We were very fortunate to have Tanya on an episode of The Security Repo Podcast, where she discussed this exact topic. Check it out to get more insights into running a successful DevSecOps team.
War Stories
Bringing together folks from all corners of the security and cryptography world, the War Stories track gave a platform for folks to explain their research to an audience outside their normal operational sphere.
A call to action by the EFF
The Electronic Frontier Foundation, EFF, has been on the front lines of cybersecurity policy-making at the national and international levels, with a mission to ensure that technology supports freedom, justice, and innovation for all people of the world. In their session "Unconventional Cybercrime: How a bad Anti-hacking treaty is becoming a law"
Global Privacy Policy Director Katitza Rodriquez and Sr. Staff Technologist Bill Budington gave us a call to action around a new international treaty that would redefine cyber crime and potentially make it much harder for researchers to surface vulnerabilities safely.
The UN Ad Hoc Committee is currently debating resolution 74/247 and is set to vote soon on creating a binding global treaty that defines cybercrime as illegal access, illegal interception, data interference, computer system interference, and misuse of a device.
The EFF wants them to eliminate a potential loophole concerning illegal access. If adopted, the resolution would potentially make anyone sharing a password for a video streaming service a criminal, not just someone violating the terms of service. It would also possibly allow non-democratic nation-states to prosecute researchers in democratic countries if they unknowingly or unintentionally gain access to a database with any data the state deems private or sensitive.
They also discussed a new Cyber Resilience Act that would compel companies to disclose vulnerabilities upon discovery. While well-intentioned to keep bad actors from hiding security flaws, the language as it stands would force public disclosure upon discovery, not as it is currently properly done, when a fix is available.
There is an upcoming UN session to vote on these measures from August 21 to September 1, 2023. It is not set in stone yet; as Bill said, "Nothing is agreed until everything is agreed." You can find out more by visiting the EFF website.
Rewarding hardware security research
How do you motivate security researchers to find potential remote execution vulnerabilities in printers and other internet-connected devices? Cash seems to work, at least according to researchers James Horseman and Zach Hanley. In their session "New isn't always novel: grep'ing your way to $20K at PWN2OWN, and how you can too," they revealed how they were able to overcome the defenses of a specific model of printer to win one of the prizes from TrendMicro's PWN2OWN. Along the way, they shared some fundamental goal-setting and execution strategies that you can apply no matter what you are working towards.
They said the most important step they took was to set goals along the way. Breaking the challenge into achievable steps lets them tackle a very complex mission. In their case, they had milestones that included getting their hands on an obscure printer far past its end of life, removing the components without breaking anything, reading the extracted flash drive, and finding a vulnerability to exploit.
At every stage, they employed a methodology called The OODA Loop, which stands for:
- Observe - what are the facts of the situation?
- Orient - consider what should be done next based on the facts.
- Decide - make an action plan and plan for all likely outcomes.
- Act - execute the plan and be ready to repeat the OODA loop when done.
They encouraged everyone interested to get involved in the PWN2OWN competition. The hardest part, they said, was obtaining the printer and working with the old and fragile motherboards. They pointed to off-the-shelf tools like CodeQL, Sonarcube, and Semgrep as great resources for finding vulnerabilities, but more importantly, as tools to prevent you from adding exploitable vulnerabilities into your code.
Hacker fun and games
While there are a lot of sessions and serious conversations at DEF CON, there are also a lot of fun and inclusive community activities. DEF CON strives to make the whole experience welcoming and safe for everyone involved.
For example, on the first day of DEF CON, there was the ToxicBBQ, the "unofficial DEF CON Kick-off Cookout." For many attendees, this was their first time at this event, and everyone was made to feel welcome by the organizers and repeat attendees. The donate-what-you-can model paid for the space and food that accommodated most all dietary restrictions, making everyone feel like they belonged no matter their circumstances.
In the evenings, DEF CON held movie nights on a big screen featuring some true sci-fi classics. There was also Hacker Jeopardy, a rowdy good time where three teams competed against one another for glory and a Black Badge. It was so popular on the first night they had to move it to the largest auditorium in Ceasars Forum on the second night to accommodate all the fans. And Hacker Karaoke, where the community got together to sing our hearts out, was a serious highlight for many.
An event like none other
There are a lot of legendary stories that come from DEF CON. Some should likely be taken with a grain of salt, and others are understated in their retelling. This event holds a very special place in the hearts of everyone involved. If you are involved or interested in security, this author would highly recommend checking out this one-of-a-kind event next year. We already plan to return to AppSec Village for another round of learning and fun at DEF CON 32 in 2024.
