Denial of Service Day Three
On Monday November 5th I realized we were undergoing a Denial of Service attack aimed at draining bandwidth. I believe it started rather weak in October but I began to notice the Kilobytes of traffic had grown far higher then it should based on other traffic statistics. See: Day 1 Analyzing and adjusting on day two: Day Two
The flow of hits from this attack continues peaking during the day and slowing down quite a bit at night. Bandwidth is back to normal range with the 403 redirects but they are still getting at new images so someone or bot with a different signature is picking the images for the rest.
Today I just focused on monitoring the logs to see if there were any changes in their pattern, and hopefully finding out how they were getting new images into their que.
I put a large chunk of the Java1.* user agents suspected of being bots into a database for easier searching: Suspected DDOS hits Page containing 9,735 unique IPs with 91966 requests (most of them 403′d).
Because my mod_rewrite to block user agents that match ^Java/1.* the user agent must begin with J this one 70.81.196.195
Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_02 which matched the pattern of nothing but continual image loading made it through.
Now I want to be real careful not to block legitimate traffic so I added a redirect for exactly that user agent as I do not believe it is real.
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(Windows\ XP\ 5\.1\)\ Java/1 [NC,OR]
While still nothing near a serious threat it would be for a site with a lower monthly bandwidth allowance.
Lessons Learned So Far
(1) Keep more than 5 logs. I’ve always browsed over my logs and let them disappear. I didn’t see a need to have them. Now I wish I could look over last months logs to see when this actually began. I will keep for an indefinite time now as space is not an issue. Perhaps work on the quick script I wrote to import the suspected bot hits and routinely insert the newest available logs as the ability to do some quick sql to find patterns would be quite nice.
(2) Some IP’s and User Agents are no good, traffic will not be hurt by shutting them down. While looking for information on this attack noticed how many spam bots and content scrapers were activly abusing the site and added these three even though I’m sure some legitimate people may use them.
RewriteCond %{HTTP_USER_AGENT} ^Jakarta [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget
I might later change the 403 redirect to a custom page alerting the actual user that their user agent is not acceptable because of abuses by others, but I’m assuming very few actual visitors use that form to access the site.
(3) Better understanding of the Linux command line. I’ve always just spit out my log files with cat and occassionally greped that to narrow what comes back, but still not fully using all of the available tools to show me exactly what I wanted.
Show me all the 403 hits in the current log
>cat /var/log/httpd/access_log | grep \ 403\
69.108.125.30 - - [08/Nov/2007:21:54:48 -0800] “GET /stimg/november082007/simpson_oj_350.jpg HTTP/1.1″ 403 241 “-” “Java/1.5.0_12″
I had tried this before but woudl get every hit that had the number 403 in it. Escaping the space before and after the 403 requires that 403 be seperated like a real 403 line would.
Without escapes this one and lots of other lines match because 4034 (size in bytes) counts as a match.
66.249.66.199 - - [08/Nov/2007:22:14:33 -0800] “GET /index.php HTTP/1.1″ 200 4034 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
Show me every hit with no referer and no user agent. (first “-” is user agent second is referer)
>cat /var/log/httpd/access_log | grep \”\-\”\ \”\-\”
Everything must be \escaped
I have noticed a few requests for RSS feeds and favicons with no useragent or referer but as a percentage of total hits quite low and not worth the costs of allowing the attackers to succeed in using up bandwidth.
(4) The writers of Salem-News.com upload one or more pictures for each article they post. I have warned in the past that they should optimize the file size before uploading. Of course they do not and a look at the size of the images these bots were aiming for were all 3 to 5 times the needed file size. I manually optimized and re-uploaded some of these reducing most from 80 to 120k to 20 to 30k.
Instead of hoping they follow my suggestions I will recode the image uploader to put it through an optimization step right after upload. This will not only reduce the damage that can be done by this type of attack but will reduce download times for all.
I will continue monitoring this and updating the Suspected DDOS hits Page and writing again if the attack pattern changes or stops.
November 8th, 2007 at 10:44 pm
On a Plus note I went about 10 months without writing a blog entry now 3 days straight of posting. Thanks for bringing back.
November 8th, 2007 at 10:45 pm
[…] cyanide bunny : My Telegraph wrote an interesting post today onHere’s a quick excerpt On Monday November 5th I realized we were undergoing a Denial of Service attack aimed at draining bandwidth. I believe it started rather weak in October but I began to notice the Kilobytes of traffic had grown far higher then it should based on other traffic statistics. See: Day 1 Analyzing and adjusting on day two: […]
November 9th, 2007 at 7:57 am
[…] You can read the full story here […]