AWS Security Agent Verification — How Source Code Improves Detection Rate

Introduction

In the previous article, we ran AWS Security Agent's penetration test with URL-only configuration and detected 4 out of 5 known vulnerabilities (0 false positives, cost ~$136). The only miss was Path Traversal — the /api/files/download endpoint had no links on any HTML page, so the CRAWLER likely never discovered it.

This article examines how providing source code changes detection outcomes. Security Agent offers two methods for supplying source code:

sourceCode — Upload as a ZIP archive to S3. The CLI description explicitly states "for static analysis"
documents — Upload to S3 or via artifacts. Described as "providing context for testing"

We ran an A/B comparison across three conditions — URL-only (previous article), documents, and sourceCode — to determine whether these two methods produce meaningfully different results. The source code was stripped of vulnerability markers (# VULN: comments etc.) to evaluate whether the AI can discover vulnerabilities purely through code structure analysis.

Prerequisites:

Reused the same environment from the previous article (Agent Space, EC2, Flask app)
Test regions: ap-northeast-1 (documents), us-east-1 (sourceCode)

Source Code Preparation

We used a clean version of the previous article's app.py with vulnerability-indicating comments (# VULN:, # --- Vulnerability N:, # No sanitization, etc.) and WARNING docstrings removed. This ensures the AI discovers vulnerabilities through code structure analysis rather than comment hints.

Registering Source Code

documents (Tokyo Region)

Upload source code to an S3 bucket and reference it via update-pentest's assets.documents. No ZIP required — raw files can be specified directly.

Steps to register documents

Reuse the variables created in the previous article: $ACCOUNT_ID, $REGION, $AGENT_SPACE_ID, $PENTEST_ID, $SERVICE_ROLE_ARN, $PRIVATE_DNS.

Terminal

S3_BUCKET="security-agent-pentest-source-${ACCOUNT_ID}"
 
# Create S3 bucket and upload
aws s3 mb s3://${S3_BUCKET} --region $REGION
aws s3 cp app-clean.py s3://${S3_BUCKET}/app.py
 
# Grant S3 read access to the service role
aws iam put-role-policy \
  --role-name SecurityAgentPentestRole \
  --policy-name SecurityAgentS3ReadAccess \
  --policy-document "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [{
      \"Effect\": \"Allow\",
      \"Action\": [\"s3:GetObject\", \"s3:ListBucket\"],
      \"Resource\": [\"arn:aws:s3:::${S3_BUCKET}\", \"arn:aws:s3:::${S3_BUCKET}/*\"]
    }]
  }"
 
# Add S3 bucket to Agent Space (preserve existing vpcs and iamRoles, add s3Buckets)
aws securityagent update-agent-space \
  --agent-space-id $AGENT_SPACE_ID \
  --name "pentest-verification" \
  --aws-resources "{
    \"vpcs\": [{
      \"vpcArn\": \"$VPC_ID\",
      \"securityGroupArns\": [\"$SG_ID\"],
      \"subnetArns\": [\"$SUBNET_ID\"]
    }],
    \"iamRoles\": [\"$SERVICE_ROLE_ARN\"],
    \"s3Buckets\": [\"arn:aws:s3:::${S3_BUCKET}\"]
  }" \
  --target-domain-ids "$TARGET_DOMAIN_ID" \
  --region $REGION
 
# Register documents in the pentest
aws securityagent update-pentest \
  --pentest-id $PENTEST_ID \
  --agent-space-id $AGENT_SPACE_ID \
  --service-role $SERVICE_ROLE_ARN \
  --assets "{
    \"endpoints\": [{\"uri\": \"http://${PRIVATE_DNS}\"}],
    \"documents\": [{\"s3Location\": \"s3://${S3_BUCKET}/app.py\"}]
  }" --region $REGION

sourceCode (us-east-1)

Upload a ZIP archive to S3 and reference it via assets.sourceCode. Specifying a raw file triggers a "Source code S3 URL must point to a ZIP archive" error.

A notable caveat: two attempts to use sourceCode in the Tokyo region both failed with INTERNAL_ERROR during the Setup Testing Environment phase. S3 Source Access Validation passed successfully, confirming that source code retrieval itself was not the issue. Removing sourceCode from the same environment allowed tests to complete normally, pointing to sourceCode processing as the root cause. Since us-east-1 did not exhibit this error, sourceCode in the Tokyo region may be unstable as of April 2026.

Steps to register sourceCode (new environment in us-east-1)

Because sourceCode did not work in Tokyo, we built a new EC2 + Agent Space environment in us-east-1. The EC2 and Flask app deployment steps are the same as the previous article's "EC2 Deployment Steps" (just change the region to us-east-1). Below covers only the Security Agent-specific setup.

Terminal (us-east-1)

REGION=us-east-1
S3_BUCKET="security-agent-pentest-source-use1-${ACCOUNT_ID}"
 
# Create Agent Space
AGENT_SPACE_ID=$(aws securityagent create-agent-space \
  --name "pentest-verification-use1" \
  --region $REGION --query "agentSpaceId" --output text)
 
# Register and verify Target Domain (UNREACHABLE is OK — VPC Config handles connectivity)
TARGET_DOMAIN_ID=$(aws securityagent create-target-domain \
  --target-domain-name "$PRIVATE_DNS" \
  --verification-method HTTP_ROUTE \
  --region $REGION --query "targetDomainId" --output text)
aws securityagent verify-target-domain \
  --target-domain-id $TARGET_DOMAIN_ID --region $REGION
 
# Create S3 bucket and upload ZIP
aws s3 mb s3://${S3_BUCKET} --region $REGION
zip app-clean.zip app-clean.py
aws s3 cp app-clean.zip s3://${S3_BUCKET}/app.zip --region $REGION
 
# Grant S3 read and CloudWatch Logs access to the service role
# (IAM roles are global, but policies are added for us-east-1 resources)
aws iam put-role-policy --role-name SecurityAgentPentestRole \
  --policy-name SecurityAgentS3ReadAccessUSE1 \
  --policy-document "{...}"  # Specify S3 bucket ARN
aws iam put-role-policy --role-name SecurityAgentPentestRole \
  --policy-name SecurityAgentLogsAccessUSE1 \
  --policy-document "{...}"  # Specify /aws/securityagent/* in us-east-1
 
# Register VPC, IAM, S3, and Target Domain with Agent Space
aws securityagent update-agent-space \
  --agent-space-id $AGENT_SPACE_ID \
  --name "pentest-verification-use1" \
  --aws-resources "{
    \"vpcs\": [{\"vpcArn\": \"$VPC_ID\", \"securityGroupArns\": [\"$SG_ID\"], \"subnetArns\": [\"$SUBNET_ID\"]}],
    \"iamRoles\": [\"$SERVICE_ROLE_ARN\"],
    \"s3Buckets\": [\"arn:aws:s3:::${S3_BUCKET}\"]
  }" \
  --target-domain-ids "$TARGET_DOMAIN_ID" \
  --region $REGION
 
# Create pentest with sourceCode
PENTEST_ID=$(aws securityagent create-pentest \
  --title "vulnerable-flask-app-pentest-sourcecode" \
  --agent-space-id $AGENT_SPACE_ID \
  --service-role $SERVICE_ROLE_ARN \
  --assets "{
    \"endpoints\": [{\"uri\": \"http://${PRIVATE_DNS}\"}],
    \"sourceCode\": [{\"s3Location\": \"s3://${S3_BUCKET}/app.zip\"}]
  }" \
  --vpc-config "{
    \"vpcArn\": \"$VPC_ID\",
    \"securityGroupArns\": [\"$SG_ID\"],
    \"subnetArns\": [\"$SUBNET_ID\"]
  }" \
  --region $REGION --query "pentestId" --output text)
 
# Start the test
aws securityagent start-pentest-job \
  --pentest-id $PENTEST_ID \
  --agent-space-id $AGENT_SPACE_ID \
  --region $REGION

Results: documents (Tokyo Region)

Full test results:

#	Finding	confidence	riskLevel
1	SQL Injection in Login — Complete Auth Bypass	HIGH	CRITICAL
2	JWT Token Forgery — Missing Signature Verification	HIGH	CRITICAL
3	JWT + XSS Chain Attack — Stored XSS in Admin Context	HIGH	CRITICAL
4	Path Traversal in File Download — Arbitrary File Read	HIGH	HIGH
5	Complete Auth Bypass on File Download	HIGH	HIGH
6	Unauthorized File Download — No Access Control	HIGH	HIGH
7	Arbitrary File Write via Path Traversal in Upload	HIGH	HIGH
8	Unrestricted File Type Upload	HIGH	HIGH
9	IDOR — Unauthorized User Profile Access	HIGH	MEDIUM
10	Stored XSS in Comments	HIGH	MEDIUM
11	SQLite Error Messages Expose DB Structure	HIGH	MEDIUM
12	Server Crash on Upload to Non-Existent Subdirectories	HIGH	MEDIUM
13	Internal Server Error on File Download Without Param	HIGH	MEDIUM

13 findings, all with HIGH confidence. A 2.6x increase from the URL-only run (5 findings).

What changed compared to the URL-only run:

Path Traversal newly detected — Previously missed, but with source code context the agent discovered /api/files/download and reproduced arbitrary file reads via ../../etc/passwd. This brings detection to 5/5
Chain attack newly discovered — JWT token forgery combined with Stored XSS for admin-context exploitation (#3). This multi-step finding was absent from the URL-only run
File upload vulnerabilities newly discovered — Arbitrary file writes via Path Traversal (#7), missing file type restrictions (#8)
Error information leakage newly discovered — SQLite error messages exposing DB structure (#11), server errors on missing parameters (#13)
Previously detected vulnerabilities (SQLi, JWT, IDOR, XSS) remain detected. All confidence levels improved to HIGH (XSS was LOW in the URL-only run)

Results: sourceCode (us-east-1)

28 findings were reported (24 HIGH confidence, 4 FALSE_POSITIVE).

The presence of FALSE_POSITIVE findings is significant. The sourceCode CODE SCANNER flagged numerous candidates through static analysis, and the VALIDATOR TASK dynamically tested each one — ultimately classifying 4 as false positives. Neither documents nor the URL-only run produced any FALSE_POSITIVE results. This reflects the combination of broad static analysis coverage with precise dynamic validation.

Key findings (HIGH confidence, CRITICAL/HIGH only):

#	Finding	riskLevel	Category
1	SQL Injection — Complete Auth Bypass	CRITICAL	Vuln 1
2	JWT Authentication Bypass — No Signature Verification	CRITICAL	Vuln 4
3	JWT Accepts Invalid Signature + Identity Spoofing	CRITICAL	Vuln 4 related
4	JWT Temporal Claims Not Validated	CRITICAL	Vuln 4 related
5	Multi-Stage Privilege Escalation Chain	CRITICAL	Chain attack
6	MD5 Password Hashing Without Salt	CRITICAL	Unexpected
7	Missing Rate Limiting — Brute Force	CRITICAL	Unexpected
8	Lack of Input Validation Framework	CRITICAL	Unexpected
9	Path Traversal in File Download	HIGH	Vuln 5
10	File Overwrite via Filename Collision	HIGH	Unexpected
11	Auth Bypass on File Download	HIGH	Vuln 5 related
12	Unrestricted File Upload with Path Traversal	HIGH	Unexpected
13	JWT Token Validation Failures — Type Confusion	HIGH	Vuln 4 related
14	Complete JWT Authentication Bypass	CRITICAL	Vuln 4 related

Additional findings include 8 MEDIUM-severity items (XSS, IDOR, missing security headers, GDPR concerns, DB foreign key constraints, JWT null value handling) and 2 INFORMATIONAL items (Flask dev server settings, SQLite temp directory).

5/5 vulnerabilities detected. Beyond that, sourceCode reported issues that documents did not detect — MD5 hashing, missing rate limiting, absent input validation, JWT type confusion, and expired token acceptance. These were only reported by the CODE SCANNER's static analysis. Whether documents missed them due to the absence of static analysis or due to non-determinism in test execution cannot be determined from this verification alone.

Three-Way Comparison

Pipeline Differences Between documents and sourceCode

Internal task comparison observed during testing:

Phase	documents (Tokyo)	sourceCode (us-east-1)
PREFLIGHT	S3 Source Access Validation, Setup Testing Env	S3 Source Access Validation, Setup Testing Env
STATIC_ANALYSIS	DOCUMENTS, SCANNER, TLS SCANNER, CRAWLER	CODE SCANNER [BUSINESS LOGIC / IMPORTANT FLOWS / FRAMEWORKS], SCANNER, TLS SCANNER, CRAWLER
PENTEST	13 attack categories + VALIDATOR	13 attack categories + VALIDATOR

The critical difference is in the STATIC_ANALYSIS phase:

documents: A DOCUMENTS task processes the source code, followed by the standard CRAWLER. The source code serves as contextual information for the AI agent, and findings are only reported during the dynamic attacks in the PENTEST phase
sourceCode: Three CODE SCANNER tasks (BUSINESS LOGIC, IMPORTANT FLOWS, FRAMEWORKS) run in parallel for systematic static analysis, detecting vulnerabilities before PENTEST even begins. A VALIDATOR TASK dynamically reproduces findings, then proceeds to CRAWLER and PENTEST

Detection Comparison

Metric	URL-only (prev.)	documents (Tokyo)	sourceCode (us-east-1)
Findings	5	13	28
HIGH confidence	4	13	24
FALSE_POSITIVE	0	0	4
Planted vulns	4/5	5/5	5/5
Path Traversal	No	Yes	Yes
Chain attacks	No	Yes (JWT+XSS)	Yes (Multi-Stage Escalation)
Static analysis findings	None	None	Yes (pre-PENTEST detection)
JWT-related findings	1	2	6
Code quality findings	None	None	Yes (MD5, rate limiting, CSRF, security headers, etc.)

Summary

Providing source code dramatically improves detection rates. URL-only produced 5 findings with 4/5 detection; documents jumped to 13 findings with 5/5 detection; sourceCode reached 28 findings with 5/5 detection. The sourceCode method is particularly notable for its CODE SCANNER, which performs systematic static analysis and reported code-quality vulnerabilities (MD5 hashing, missing rate limiting, absent input validation) that were not detected by the other two methods.

Aspect	URL-only	documents	sourceCode
Setup effort	None	S3 upload + update-pentest (minutes)	S3 ZIP upload + update-pentest (minutes)
Finding count	5	13 (2.6x)	28 (5.6x)
Processing pipeline	CRAWLER, PENTEST	DOCUMENTS, CRAWLER, PENTEST	CODE SCANNER x3, CRAWLER, PENTEST
Static analysis	None	None	BUSINESS LOGIC / IMPORTANT FLOWS / FRAMEWORKS
Tokyo region	Yes	Yes	No (INTERNAL_ERROR)

In this verification, source code provision increased findings by 2.6-5.6x — Detection of endpoints with no HTML links (API endpoints, etc.) improved significantly. Source code registration takes only a few minutes, making it a high cost-effectiveness option
Choose sourceCode when available — The CODE SCANNER's static analysis offers unique value absent from documents. Code quality findings (MD5, rate limiting, security headers) make it practical as a security review tool
Fall back to documents in the Tokyo region — sourceCode hit INTERNAL_ERROR in Tokyo as of April 2026. documents still achieved 5/5 detection in this verification and is perfectly viable
FALSE_POSITIVE only occurs with sourceCode — Broader static analysis naturally produces candidates that cannot be dynamically reproduced. The VALIDATOR TASK automatically flags false positives, so no manual triage is needed

Cleanup

Resource deletion