Developer Week 2023 - The Enterprise Community Sharing Security Best Practices

For the first time since 2019, the "world's largest developer and engineering expo" was back in person, this time in Oakland in February: DeveloperWeek 2023! Approximately 2000 attendees, speakers, and exhibitors got together face to face to meet and talk about the state of the software industry. People came from all over the world to be part of this 15+ track event that covered everything from application and API design, to Kubernetes and Terraform deployment fundamentals, and basically everything in between. I got to give a talk as well on the state of cloud development environments, which I have written about here before.

There was a noticeable thread around cybersecurity throughout all the tracks. As security becomes more and more of a focus for the enterprise, it should be no surprise to see so many talks about securing your data and your applications. Here are just a few highlights from some of the sessions centered around security from DeveloperWeek 2023.

Securing Your Applications and APIs

Secure Your APIs

There is a lot you could cover about application security, but one particular topic popped up in several talks, API security. There were two talks on this matter from slightly different perspectives, and I think both approaches are worth looking into, so I am highlighting them here.

The first session was a holistic approach from Farshad Abasi, Founder and CEO of Forward Security, in his talk "Designing Secure API and Microservices-Based Applications." He explained the history of Service Oriented Architecture. While we think of 'web APIs' when we think of Application Programming Interfaces, but APIs have been around for a very long time.  He talked about the transformation at IBM, which went from 1000s of independent applications, all reinventing the wheel to recognizing there were multiple overlapping services underlying every application and starting to deploy addressable services.

Farshad shared that in those early days, heavy-weight solutions like SAML and SOAP worked well behind firewalls, but soon, the rise of web applications and open networks drove HTTP and JSON as the new standards. With this change, we lost the benefit of the single entry into the 'trust zone' where monolithic applications and service-oriented architecture, SOA deployments ran. Running safe Microservices means we can trust no one and need to authenticate all the way through for every request.

At the heart of his session, he explained how API gateways should act as a single point of entry into your application, giving you an isolated service to handle both authentication and authorization. Both of those jobs are related, but he spent some time explaining the difference.

Authorization vs Authentication

To make it easier to conceptualize, he suggested thinking about the real-world example of a bouncer at a fancy nightclub. When you first arrive, the bouncer asks for your ID, normally some government-issued document, such as a driver's license or passport. At this stage, the authenticity of the ID is scrutinized to make sure it is legit and not a forged document. The contents are normally confirmed as well, sometimes by the bouncer asking you questions and verifying the answers. This is the authentication process, often abbreviated as AuthN.

Once the bouncer confirms you are who you say you are, they proceed to make sure you are supposed to be there by consulting the guest list. If your name is on the list, then you are authorized to go in. This is the authorization process, abbreviated as AuthZ.

API gateways can help with both AuthN and AuthZ, by examining requests for authenticity and checking for the correct signatures or certificates, and then consulting access controls to ensure only the right requests get through.  Just because you have a valid ID does not mean you get full access to all the DBs and servers.

This approach assumes you are starting with Zero Trust in mind, granting the least privileges possible to allow work to be performed. Farshad wrapped up his talk by talking about the importance of logging, including monitoring and reviewing them consistently. The earlier you take into account how you will deal with AuthN, AuthZ, permissions, and logging, the safer your application will be as you move it toward production.

API Security And OWASP

Thinking about API security from early in the design process is an excellent idea, but it can be challenging to know where to start when dealing with an existing or legacy system where you want to update security. This is where the approach from Colin Domoney,

Chief Technology Evangelist at 42Crunch, really comes into its own. In his session "Everything You Need to Know About API Security," he walked us through the OWASP API Top 10. While there is some crossover from the OWASP Top 10 standard awareness project, the API Top 10 focuses on the unique vulnerabilities and security risks that the API brings.

Colin walked us through the list, and the vulnerabilities at play then shared some industry horror stories that highlighted what can go wrong. In one of the more amusing stories, a microbrewery forgot to lock down the testing credentials that let users get unlimited beer tokens. In another, more widespread example, a WordPress plugin gave full site admin rights to any user who knew how to access the plugin's control panel through URL manipulation; a good reminder to always only grant the least privileges by default.

Scaling Safely, Even When Your Apps Are Under Attack

In his talk "Your Apps Are Under Attack," Cory Gideon Manager Consultant at Sogeti USA, wasted no time in getting to the point that security is hard and takes work. While it might be tempting to think you can just buy a single product to make you unhackable, the reality is no single vendor can deliver on such a promise. This is partly due to the pace of evolving threats combined with the complexity and size of our applications. He also shared some sobering stats:

• 36% of devs surveyed attributed priority of meeting deadlines as to why code still contains vulnerabilities
• 33% admitted that they don't know what makes their code vulnerable
• 30% said their in-house security training could be improved with practical real-world examples.
• 30% said their biggest concern with implementing secure coding is dealing with vulnerabilities introduced by co-workers.  
  

When something goes wrong, many teams just fire up that blamethrower.

Fortunately, Cory shared a lot of straightforward best practices and open source tools that can help us all work more securely while still meeting our deadlines.  This starts with securing our password policies. He shared a project called changeme that can detect default and backdoor credentials throughout any system. He also urged teams to review their stacks for deprecated protocols and encryption strategies, as they become obsolete for a reason.  

When focusing on points of access, he said it is important to understand attackers have two critical jobs, 1) get in and 2) get out. If you can disrupt either of those, you can buy time to act. This is where honeytokens would be helpful as well.

There are basically too many good suggestions to list here, but fortunately, Cory shared that OWASP has already laid all of this out in their CheatSheet series, which you should check out.

The last points he made were around ChatGPT and some of the potential ways it is already changing security. Cory believes that attackers are already asking ChatGPT to help them and we should be too if we want to keep up in the fight. Toward the end, he shared one of my favorite quotes from the whole event:

ChatGPT will not take your job. A developer who knows how to use ChatGPT will.

Beyond just securing your application architecture, developers also need to worry about the issues when you deploy and then try to scale your application. That was the very subject Kerim Satirli, Senior developer advocate at HashiCorp, covered in his talk "Scaling Security when Deploying Globally."  At the core of his talk, Kerim told us that "Security is a team sport' that requires us all to automate as much as possible and educate everyone on best practices.

He shared that 10% of companies still say they use Root access for things daily.  While most of us know we shouldn't do that, it is up to all of us to make sure everyone knows that. Another thing we need to do better around is Zero Trust. Kerim encourages us to think about the default access policy as 'deny,' so "if the power goes out, all the doors remain locked".  

Another topic he brought up was locking out unneeded regions. For example, if you have your whole infrastructure in AWS zone us-east-2, there is no reason you should leave it open to requests from ap-east-1 or eu-central-2, to just pick a couple at random.

He also spent some time enumerating the advantages of short-lived dynamic secrets. The best secrets are the ones that never get shared; that way, they can never be hardcoded, which we here at GitGuardian wish happened less than it does.  

Securing your Data

While securing the code and your environments is certainly an important part of a mature security posture, if you are careless with your data, then you are just as vulnerable to catastrophe.  

In their session "Don’t DIY PII. Five Data Privacy Challenges and How to Solve Them," the team from Skyflow, Sean Falconer, Head of Developer Relations, and Manish Ahluwalia, Technology Executive, shared that one of the main reasons that large companies have seen so many breaches is because they hold so much customer personal identifying information, PII. This is extremely valuable to bad actors. Once an attacker has enough data points, they can launch spear phishing attacks or, worse, steal the customer's identity.

They summed up 5 data privacy challenges that you need to consider when dealing with PII:

1. Secure storage - Knowing where your PII rests has the right encryption.
2. Access controls - Knowing who has access to PII at any given point.
3. The right to be forgotten - Knowing where all the PII lives so you can remove all PII when needed.
4. Residency - Knowing all the laws about any geographical region or state where PII lives.
5. Secure data utilization - Knowing what systems are accessing your data and that all those services are secure.

They talked about possible solutions where real PII or other sensitive data is never actually stored in the DB or inside applications, instead storing those data points in secure data vaults. Data vaults can safely and securely store sensitive data and generate pointers to the real data, which can be safely stored in DBs and open applications with no fear of real data leakage. Endpoints that need to ingest the real data can get it from the vaults; otherwise, they get served the pointers which have no value without the right access rights. No matter what solution you find is right for your architecture, addressing those five main points will help you get on the path to better data security.

Even if we have our data locked down and we personally trust the security, how do we objectively know if we can trust it? That was at the heart of the session from Soumen Chatterjee,  Associate Vice President at HCLTech CloudSMART. In his talk "Building the Data Trust on Cloud - Confidential Computing, Sovereign Cloud and Secure Cloud," he brought up that we have spent a lot of time securing our data encryption at rest and in transit. But when it comes to data in use, we mostly just adopt a 'magical trust' philosophy that assumes the app itself must be secure. This is as true of Gmail as it is of a healthcare portal, though we know these are very different applications.

Soumen says we need a standard for data governance in the cloud, a way that will lead to trusted applications. He spelled out Five Elements of a Sovereign cloud:

• Data sovereignty
• Data localization
• Data ownership
• Data traceability
• Data access controls

While there is no single magic bullet to grant trust for your application, if your cloud-based solution can answer all of those points, then you are on a good path.  

While insecure data can cost your company a lot in the case of a breach, Shailvi Wakhlu, Former Head of Data & Analytics at Strava, says bad data is costing your company a lot right now. While not expressly a security talk, her session "Preventing, Diagnosing & Curing Bad Data" provided great insight into data issues I think can be applied by development and security teams.

By "bad data," she means any data that is inaccurate, incomplete, or misleading. Bad data leads to biased decision-making, which will always lead to worse outcomes. From a security perspective, it means you can be overoptimizing for the wrong vulnerabilities. In marketing, it means you might be undercharging for your services, leading to a failed business in the worst extremes. Ultimately though, bad data can lead to legal issues and terrible employee morale; no one loves dealing with data they need to clean up continually.

To identify and fix bad data, she laid out the flow of data creation, ingestion, and usage to help identify where issues arise. According to Shailvi, all data gets:

• Defined
• Logged and stored
• Transformed, such as when we apply business rules to it
• Analyzed
• Shared

At each step, the chance for data corruption arises, and the earlier we can catch issues, the easier it is to solve in the long term.

To illustrate her point about data aggregation and how human bias seeps in, she showed a picture of a set of ingredients, including butter, eggs, flour, salt, and some bowls. If you asked some people what they were making in the picture, some would guess pancakes, some would guess bread, and some would guess something else. The data simply showed ingredients, but if someone who labels the picture inserts their bias into the label, we might mislabel this as a pancake instead of what it really is.

While she would love a world of perfect data, she also said it is important to take into consideration the criticality of the data and the scope of attempting to clean it all up versus just focusing on where you can help. She summed it up very well with this paraphrased quote:

When you encounter bad data, you have to consider; should keep sweeping the floors or should we burn the building down?

Developer Week Is Helping Us All Dev Better

This article barely scratches the surface of all the amazing sessions at Developer Week. Beyond the talks and workshops, this event was meaningful in bringing a large and diverse community back together face-to-face.

The hallway conversations, the shared experiences at the toss-across and tic-tac-toe lounge, and the memories will last no matter what the AI revolution brings our way. At the end of the day, software is made by people. This was a great event for those people to share their passion and knowledge.  I am already looking forward to Developer Week 2024!

If you were not able to attend in person, Developer Week continues online in the virtual event portion. Check out the Developer Week website for more info on how to watch the recordings from the virtual event.