Listen to this Post
Scrapy’s S3DownloadHandler converts an S3-scheme bucket and key request into a plaintext HTTP request to the corresponding S3 endpoint unless `request.meta[“is_secure”]` is explicitly enabled, then signs and sends the plaintext request with configured AWS credentials. The vulnerable code in `scrapy/core/downloader/handlers/s3.py` constructs the URL using scheme = "https" if request.meta.get("is_secure") else "http", defaulting to HTTP. This means AWS authorization material—including the `Authorization` header and temporary X-Amz-Security-Token—can be transmitted without TLS encryption. A network attacker able to observe traffic between Scrapy and S3, such as a public Wi-Fi attacker, compromised router, ISP/corporate network observer, or local network attacker using ARP spoofing, can read the bucket/key path, AWS Authorization header, `X-Amz-Security-Token` (if temporary credentials are used), S3 object contents, and S3 response headers. An active MITM attacker can also modify the plaintext S3 response body, status code, and headers before Scrapy processes it. This can cause scraped data poisoning, poisoned exports, HTTP cache poisoning when cache is enabled, and influence over later crawl targets through forged redirects or attacker-controlled links. Suggested classification: CWE-319: Cleartext Transmission of Sensitive Information. A minimal PoC creates an `s3://` request, enables fake AWS settings, captures the request produced by S3DownloadHandler, and prints the rewritten URL and auth headers. This issue is fixed in version 2.17.0.
DailyCVE Form:
Platform: Scrapy framework
Version: < 2.17.0
Vulnerability: Cleartext transmission
Severity: High (CVSS 7.4)
date: 2026-09-01
Prediction: Already patched (2.17.0)
What Undercode Say:
Analytics:
Check Scrapy version pip show scrapy | grep Version Verify S3 requests are using HTTPS (port 443) tcpdump -i any -n "host s3.amazonaws.com and port 443" Monitor for plaintext S3 traffic (port 80) tcpdump -i any -n "host s3.amazonaws.com and port 80" List all s3:// requests in your Scrapy project grep -r "s3://" --include=".py" .
Exploit: (Educational Purposes!)
Minimal PoC - demonstrates the vulnerability
from scrapy import Request
from scrapy.core.downloader.handlers.s3 import S3DownloadHandler
from scrapy.crawler import Crawler
from scrapy.settings import Settings
Setup
settings = Settings({
'AWS_ACCESS_KEY_ID': 'fake_key',
'AWS_SECRET_ACCESS_KEY': 'fake_secret',
})
crawler = Crawler(spidercls=None, settings=settings)
handler = S3DownloadHandler(crawler)
Vulnerable request (is_secure not set → defaults to HTTP)
request = Request('s3://my-bucket/sensitive/data.csv')
This will generate an HTTP request with AWS auth headers in plaintext
The request URL becomes: http://my-bucket.s3.amazonaws.com/sensitive/data.csv
response = handler.download_request(request)
Protection:
Upgrade to patched version
pip install --upgrade scrapy>=2.17.0
Or explicitly set is_secure=True in requests
request.meta['is_secure'] = True
AWS Bucket Policy requiring Secure Transport
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "",
"Action": "s3:",
"Resource": "arn:aws:s3:::your-bucket/",
"Condition": {
"Bool": {
"aws:SecureTransport": "false"
}
}
}
]
}
Verify no plaintext S3 traffic remains
grep -r "is_secure" --include=".py" . | grep -v "is_secure.True"
Impact:
- Confidentiality: Exposure of AWS credentials, temporary tokens, bucket names, object keys, and S3 object contents to network eavesdroppers.
- Integrity: Active MITM attackers can modify S3 response bodies, status codes, and headers, leading to scraped-data poisoning, poisoned exports, and HTTP cache poisoning.
- Availability: Forged redirects or attacker-controlled links can influence later crawl targets, potentially disrupting scraping operations.
🎯Let’s Practice Exploiting & Learn Patching For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
Sources:
Reported By: github.com
Extra Source Hub:
Undercode
