I have a wordpress application hosted on a digital ocean droplet. I have noticed in my access logs that I am getting lots of requests from various IPs that appear to be crawling my website and trying to access suspicious URL's, plugin files that do not exist. Unfortuntely I believe these are causing my CPU to hit 100% and php fpm to crash due to max children reached.
I was wondering whether it would be okay for some help in updatin fail2ban to try and catch these requests and ban the IP?
I am also unsure whether there also maybe an issue as I noticed some requests that look okay are 444?
I have updated my failregex in /etc/fail2ban/filter.d/nginx-botsearch.conf
failregex = ^<HOST> - - \[.*\] "(GET|POST|HEAD|PUT|DELETE|OPTIONS|TRACE|CONNECT) /.*\.(php|bak|sql|txt|env|git|htaccess)" \d+ .+$ ^<HOST> - - \[.*\] "(GET|POST|HEAD|PUT|DELETE|OPTIONS|TRACE|CONNECT) /.*/\.env.* HTTP/\d+\.\d+" \d+ .+$ #Block /x00/ Encoded URLs ^<HOST> - - \[.*\] ".*\x00.*" \d+ .+"-" "-" ^<HOST> - - \[.*\] ".*HTTP/.*" \d+ \d+"-" "-" ^<HOST> - - \[.*\] ".*\x16\x03\x01.*" \d+ .+"-" "-" #Block Go-http-client ^<HOST> - - \[.*\] "GET .*HTTP/.*" \d+ \d+"-" "Go-http-client/.*"$ #Block XDeBug ^<HOST> - - \[.*\] "(GET|POST|HEAD|PUT|DELETE|OPTIONS|TRACE|CONNECT) /.*XDEBUG_SESSION_START=phpstorm.*HTTP/.*".*$ #Block Suspicious WP Requests ^<HOST> - - \[.*\] "(GET|POST|HEAD|PUT|DELETE|OPTIONS|TRACE|CONNECT) /.*/(wp-admin|wp-login|xmlrpc\.php|phpmyadmin|admin|login\.php)" \d+ .+$ ^<HOST>.*"(GET|POST).*\/.*(\.php\?|wp-(?!admin)|chosen|wso|json|index\.php|phpmailer|file\.php|css|themes|plugin|api|include|request)(?!\/uploads).*HTTP/.*".*$ #Block Multiple Errors Requests ^\[error\] \d+#\d+: \*\d+ (\S+ )?\"\S+\" (failed|is not found) \(2: No such file or directory\), client: <HOST>And here is an example of my access logs:
149.50.103.48 - - [13/Sep/2024:03:44:29 +0000] "GET / HTTP/1.1" 444 0 "-" "-"95.214.55.138 - - [13/Sep/2024:03:46:47 +0000] "GET / HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"185.191.126.213 - - [13/Sep/2024:03:49:47 +0000] "GET / HTTP/1.1" 444 0 "-" "-"185.224.128.59 - - [13/Sep/2024:04:06:24 +0000] "GET /cgi-bin/luci/;stok=/locale HTTP/1.1" 444 0 "-" "Go-http-client/1.1"34.77.176.45 - - [13/Sep/2024:04:11:43 +0000] "GET / HTTP/2.0" 301 162 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"143.110.222.166 - - [13/Sep/2024:04:16:53 +0000] "GET / HTTP/1.1" 444 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1"185.16.39.118 - - [13/Sep/2024:04:17:24 +0000] "GET / HTTP/1.1" 444 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 Edg/90.0.818.46"66.249.66.192 - - [13/Sep/2024:04:18:03 +0000] "GET /n405-npors-crane-lift-supervisor-2/ HTTP/2.0" 301 162 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.137 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"62.146.169.255 - - [13/Sep/2024:04:23:24 +0000] "GET /wp-includes/ID3/index.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:23:48 +0000] "GET /wp-content/plugins/index.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:24:14 +0000] "GET /wp-admin/network/index.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:25:01 +0000] "GET /wp-includes/IXR/index.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:25:25 +0000] "GET /wp-admin/images/index.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:25:48 +0000] "GET /wp-login.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:26:23 +0000] "GET /wp-content/plugins/admin.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:26:50 +0000] "GET /xmlrpc.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:27:13 +0000] "GET /wp-admin.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:27:41 +0000] "GET /wp-content/plugins/Core-Econ/upH.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:28:03 +0000] "GET /wp-content/plugins/fix/up.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:28:32 +0000] "GET /wp-content/plugins/pwnd/pwnd.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:28:59 +0000] "GET /wp-content/plugins/shell/noimg.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:29:23 +0000] "GET /WSOEnigma.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:29:46 +0000] "GET /wp-content/plugins/pwnd-1/pwnd.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"40.77.167.79 - - [13/Sep/2024:04:30:10 +0000] "GET /a64-overhead-travelling-crane-gantry-crane/ HTTP/2.0" 301 162 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:30:11 +0000] "GET /wp-content/plugins/dummyyummy/wp-signup.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"62.146.169.255 - - [13/Sep/2024:04:30:37 +0000] "GET /wp-content/plugins/clara/clara.php HTTP/2.0" 301 162 "-" "Mozlila/5.0 (Linux; Android 7.0; SM-G892A Bulid/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/60.0.3112.107 Moblie Safari/537.36"167.94.145.106 - - [13/Sep/2024:04:48:45 +0000] "GET / HTTP/1.1" 400 248 "-" "-"167.94.145.106 - - [13/Sep/2024:04:48:48 +0000] "GET / HTTP/1.1" 400 248 "-" "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)"167.94.145.106 - - [13/Sep/2024:04:48:48 +0000] "PRI * HTTP/2.0" 400 150 "-" "-"52.189.74.240 - - [13/Sep/2024:04:54:54 +0000] "MGLNDD_165.232.110.85_443" 400 150 "-" "-"143.110.222.166 - - [13/Sep/2024:05:13:05 +0000] "GET / HTTP/1.1" 444 0 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1"149.50.103.48 - - [13/Sep/2024:05:18:14 +0000] "GET / HTTP/1.1" 444 0 "-" "-"162.216.149.10 - - [13/Sep/2024:05:21:01 +0000] "GET / HTTP/1.1" 444 0 "-" "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com"54.36.115.221 - - [13/Sep/2024:05:25:19 +0000] "GET /.env HTTP/1.1" 444 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36"Any help in understanding and resolving my CPU issues by blocking requests would be much appreciated? Thanks