Summary
I am trying to implement a scraping process that bypasses Cloudflare in an OCI environment, but I am not successful. It works in my local environment using several methods, but it fails in the cloud environment. Should I assume that OCI's IP addresses are blacklisted by Cloudflare?
If anyone has knowledge about this, I would appreciate your guidance.
What I've Tried
Scraping with Python
- nodriver: Module
- drissionpage: Module
- flaresolverr: Docker proxy
Environment
Local
- Machine: Mac mini with M2 chip
- macOS: Sonoma 14.6.1
Cloud
- OKE (Oracle Linux, ARM-based)
- OCI Instance (Oracle Linux, ARM-based, Ubuntu)
Success/Failure
nodriver
- Local: Successfully bypasses Cloudflare on Mac
- Cloud: Fails to bypass Cloudflare on OCI instance (though the process finishes normally)
drissionpage
- Local: Successfully bypasses Cloudflare on Mac
- Cloud: Fails to bypass Cloudflare on OCI instance (though the process finishes normally)
flaresolverr
- Local: Successfully bypasses Cloudflare by running Docker Compose to set up a proxy container and using the proxy in Python
- Cloud (OKE): Fails to bypass Cloudflare with the proxy container running on a pod and Python using the proxy (process finishes normally)
- Cloud (OCI Instance): Fails to bypass Cloudflare with the proxy container running on the instance and Python using the proxy (process finishes normally)