This is the third blog post in our "Bring Your Own Source" series. The first one covered n8n workflow integration and the second one covered Salesforce.

The Hidden Risk in Your CI/CD Pipeline

Your CI/CD pipeline is a treasure trove of sensitive information. While you've probably focused on scanning your source code for secrets, there's another critical attack vector that often goes unnoticed: CI/CD logs.

Your build logs may contain environment variables, connection strings, API responses, debugging output, and temporary credentials. These logs are often stored for weeks or months, accessible to anyone with pipeline visibility. A single exposed database password or API key in your logs could give attackers everything they need to compromise your infrastructure.

Recent breaches have shown that attackers increasingly target CI/CD environments, not just for supply chain attacks, but for the wealth of credentials and sensitive data flowing through build processes. GitLab, Jenkins, GitHub Actions – they all generate logs that can inadvertently expose secrets.

The Challenge: Comprehensive Secrets Detection Across Your Entire DevOps Lifecycle

Traditional secrets scanning focuses on source code repositories, but modern DevOps workflows create secrets exposure points everywhere:

  • Build logs containing environment variables and API responses
  • Deployment scripts with temporary credentials
  • Test outputs revealing database connections
  • Debug information exposing internal system details
  • Application logs

The problem is that most organizations have no systematic way to scan these ephemeral but critical data sources. That's where GitGuardian's Bring Your Own Source initiative comes in.

Solution: Automated CI Log Scanning with GitGuardian

In this tutorial, we'll show you how to automatically scan all GitLab CI pipeline logs using ggshield and GitGuardian's Bring Your Own Source initiative. Every time your pipeline runs, we'll capture and scan the logs from all jobs, automatically creating incidents in your GitGuardian dashboard when secrets are detected.

Here's what we'll build:

  1. Automated log collection from all pipeline jobs
  2. Real-time secrets scanning using ggshield
  3. Incident creation in the GitGuardian dashboard
  4. Zero-friction integration with your existing CI/CD workflows

What You'll Need

  • A GitGuardian Business account (required for Custom Sources)
  • A HashiCorp Vault server for secure token storage
  • A GitLab project with existing CI/CD pipelines
  • Vault-GitLab integration configured

Step 1: Configure GitGuardian Custom Source

First, let's set up GitGuardian to receive and process your CI log data.

Create Your Custom Source

  1. Navigate to Settings > Sources in your GitGuardian dashboard
  2. Find the "Custom sources" section and click "Add Custom Source"
  3. Name it something descriptive like "GitLab CI Logs - [Your Project Name]"
  4. Copy the unique Source ID – you'll need this for the CI configuration

Generate a Service Account Token

Your CI pipeline needs authentication to send scan results to GitGuardian:

  1. Go to Settings > API > Service Accounts
  2. Click "Create service account"
  3. Name it "GitLab CI Log Scanner"
  4. Select the full scanning scope with both "scan" and "scan-create-incidents" permissions
  5. Click "Create" and securely store the generated token

Pro tip: This token has powerful permissions – treat it like a production database password.

Step 2: Configure GitLab API Access

Your scanning job needs to fetch logs from other pipeline jobs via the GitLab API.

Generate a GitLab Personal Access Token:

  1. Go to GitLab > User Settings > Access Tokens
  2. Create a token with read_api scope 
  3. Set an appropriate expiration date
  4. Store this token securely alongside your GitGuardian token

Step 3: Secure Token Storage in Vault

Security first! Both tokens need secure storage and injection into your CI environment.

Create a new secret in your Vault (we'll use path ci/[Your Project Name]/scan_logs)

{
  "GITGUARDIAN_API_KEY": "your-gitguardian-service-account-token",
  "GITLAB_TOKEN": "your-gitlab-api-token"
}

Step 4: Build Your Custom ggshield Image

The official ggshield Docker image needs additional tools to interact with the GitLab API. Let's create an enhanced version:

FROM gitguardian/ggshield:latest

WORKDIR /app

# Add curl for API calls and jq for JSON parsing
RUN \
    apt-get update \
    && apt-get install -y --no-install-recommends curl jq \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /data
VOLUME [ "/data" ]
CMD ["ggshield"]

Build and publish your image:

# Build the enhanced image
docker build . --tag enhanced-ggshield:latest

# Tag for your registry
docker tag enhanced-ggshield:latest your-docker-account/enhanced-ggshield:latest

# Push to registry
docker push your-docker-account/enhanced-ggshield:latest

Step 5: Create the Log Scanning Script

Now for the core logic. Create .gitlab/ci_scripts/scan_logs.sh in your repository:

#!/bin/bash
set -e

# Validate required environment variables
for var in SOURCE_UUID GITLAB_TOKEN GITGUARDIAN_API_KEY; do
    if [ -z "${!var}" ]; then
        echo "❌ Error: $var environment variable is not set"
        exit 1
    fi
done

# GitLab API configuration from CI environment
GITLAB_API_URL="$CI_API_V4_URL"
PROJECT_ID="$CI_PROJECT_ID" 
PIPELINE_ID="$CI_PIPELINE_ID"
AUTH_HEADER="Authorization: Bearer $GITLAB_TOKEN"

echo "🔍 Scanning logs for pipeline $PIPELINE_ID in project $PROJECT_ID"

scan_failures=0

# Function to download and scan individual job logs
scan_job_log() {
    local job_id="$1"
    local job_name="$2" 
    local log_file="/tmp/job_${job_id}_log.txt"
    
    echo "📋 Processing job: $job_name (ID: $job_id)"
    
    # Download job log via GitLab API
    if curl -s -H "$AUTH_HEADER" \
        "$GITLAB_API_URL/projects/$PROJECT_ID/jobs/$job_id/trace" \
        -o "$log_file"; then
        
        # Only scan non-empty logs
        if [ -s "$log_file" ]; then
            echo "🔎 Scanning log file for job $job_name..."
            
            # Run ggshield with custom source UUID
            if ggshield secret scan path "$log_file" --source-uuid "$SOURCE_UUID"; then
                echo "✅ Scan completed successfully for job: $job_name"
            else
                echo "❌ Scan failed for job: $job_name"
                ((scan_failures++))
            fi
        else
            echo "⚠️  Log file for job $job_name is empty, skipping"
        fi
        
        # Cleanup temporary file
        rm -f "$log_file"
    else
        echo "❌ Failed to download log for job: $job_name"
        ((scan_failures++))
    fi
}

# Fetch all jobs in current pipeline
echo "📡 Fetching jobs for pipeline $PIPELINE_ID..."

jobs_data=$(curl -s -H "$AUTH_HEADER" \
    "$GITLAB_API_URL/projects/$PROJECT_ID/pipelines/$PIPELINE_ID/jobs")

# Validate API response
if ! echo "$jobs_data" | jq -e . >/dev/null 2>&1; then
    echo "❌ Failed to fetch jobs data or invalid JSON response"
    echo "Response: $jobs_data"
    exit 1
fi

job_count=$(echo "$jobs_data" | jq '. | length')
echo "📊 Found $job_count jobs in pipeline"

# Process each job
for i in $(seq 0 $((job_count - 1))); do
    job_id=$(echo "$jobs_data" | jq -r ".[$i].id")
    job_name=$(echo "$jobs_data" | jq -r ".[$i].name") 
    job_status=$(echo "$jobs_data" | jq -r ".[$i].status")
    
    # Skip jobs without completed logs
    case "$job_status" in
        created|pending|running)
            echo "⏳ Skipping job $job_name (status: $job_status) - no logs available"
            continue
            ;;
    esac
    
    scan_job_log "$job_id" "$job_name"
done

# Report final results
if [ $scan_failures -gt 0 ]; then
    echo "❌ $scan_failures job(s) failed during log scanning"
    exit 1
fi

echo "🎉 Log scanning completed successfully for all jobs in pipeline $PIPELINE_ID"

This script leverages GitLab's built-in CI environment variables:

  • CI_API_V4_URL: GitLab API base URL
  • CI_PROJECT_ID: Current project identifier
  • CI_PIPELINE_ID: Current pipeline identifier

Note: We use a personal access token instead of CI_JOB_TOKEN because the job token has insufficient scope for cross-job API access.

Step 6: Configure the CI Job

Add this job to your .gitlab-ci.yml or jobs configuration:

security:ggshield:logs:
  stage: .post  # Runs after all other stages
  image: your-docker-account/enhanced-ggshield:latest
  
  variables:
    SOURCE_UUID: "your-custom-source-uuid-from-gitguardian"
  
  # Vault integration for secure token injection
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://your-vault-server.com"
      
  secrets:
    GITGUARDIAN_API_KEY:
      vault: [Your Project Name]/scan_logs/GITGUARDIAN_API_KEY@ci
      file: false
    GITLAB_TOKEN:
      vault: [Your Project Name]/scan_logs/GITLAB_TOKEN@ci  
      file: false
      
  script:
    - cd .gitlab/ci_scripts && bash scan_logs.sh
    
  rules:
    - when: always  # Run even if previous jobs failed

The Result: Comprehensive CI/CD Secrets Detection

Once deployed, here's what happens automatically:

  1. Every pipeline triggers log scanning – regardless of success or failure
  2. All job logs are collected and analyzed using ggshield's advanced detection
  3. Secrets are immediately flagged and incidents created in GitGuardian
  4. Your security team gets real-time alerts about CI/CD secrets exposure
  5. Audit trails are maintained for compliance and incident response

Why This Matters: Real-World Impact

This implementation addresses critical security gaps that traditional scanning misses:

Catching Runtime Secrets

  • Environment variables leaked in error messages
  • API responses containing credentials
  • Debug output revealing internal tokens
  • Temporary credentials in deployment logs

Compliance & Governance

  • Complete audit trail of secrets across your entire CI/CD pipeline
  • Automated evidence collection for SOC 2, ISO 27001, and other frameworks
  • Real-time compliance monitoring and alerting

Developer Experience

  • Zero friction – no changes to existing development workflows
  • Automatic incident creation and notification
  • Educational feedback helps teams improve security practices

Next Steps: Expanding Your Secrets Security Program

This GitLab CI log scanning is just one piece of our comprehensive secrets security strategy. Consider expanding to:

  • Multi-platform coverage: Apply similar techniques to GitHub Actions, Jenkins, and Azure DevOps
  • Infrastructure scanning: Monitor Kubernetes logs, server logs, and application logs
  • Real-time monitoring: Set up webhooks for immediate incident response
  • Policy automation: Create custom remediation workflows based on secret types

Ready to Secure Your CI/CD Pipeline?

Secrets in CI/CD logs represent a critical but often overlooked attack vector. With GitGuardian's Bring Your Own Source initiative and this automated scanning approach, you can close these security gaps without impacting developer productivity.

Want to see this in action? Book a demo to explore how GitGuardian can secure secrets across your entire development lifecycle – not just in source code, but everywhere they hide.

Questions about implementation? Our customer relations team is here to help. Reach out for technical support and best practices.

Stay tuned for the next post in our Bring Your Own Source series, where we'll tackle secrets detection in Notion.

Bring Your Own Source: Plug GitGuardian into Any Workflow in Minutes
Discover how GitGuardian’s “Bring Your Own Source” initiative enables security teams to extend secrets detection beyond code repositories, leveraging custom integrations to eliminate a significant hidden attack surface.
GitGuardian Blog - Take Control of Your Secrets SecuritySoujanya Ain
When Google Says “Scan for Secrets”: A Complete Guide to Finding Hidden Credentials in Salesforce
The Salesloft Drift breach affected hundreds of organizations through Salesforce, including Cloudflare, Palo Alto Networks, and Zscaler. Google now explicitly recommends running secrets scanning tools across Salesforce data—here’s your complete guide.
GitGuardian Blog - Take Control of Your Secrets SecurityGuillaume Valadon