Currently many (More than one is to many) US Government websites are under the control of criminals. Finding Hacked Goverment and School Web Sites Try a Google Search for site:*.gov porn or swap the words for other terms and you will most likely turn up sites that redirect to obviously not Government sites for pornography or Malware downloads.

Now it is understandable that sometimes hackers win a battle and gain control over something they shouldn’t. But it has been days since I first read of this and began finding sites owned by hackers. This should not be and the agencies involved should be far quicker at fixing the problem or lose their right to be responsible for maintaining a public website.
My email to one of the websites hacked.

I understand that sometimes sites become hacked but you should be far
quicker at shutting down or correcting the problem. Or you should not
be allowed on the Internet.

Now I receive from multiple sources spam emails and blog postings. I could just quickly delete them and move on but that does little to fight the problem. By contacting the sources and eventual domain name destinations of these attacks far more can be accomplished.

Most of my spam both email and blog spam comes from commen peoples computer or advertises sites that are quick and easy to put up or hide behind a URL redirection services. At least one of them 110mb.com a free hosting site quickly responds and removes the offending site.:

Site has been removed.

Thank you for reporting it.

Best Regards,
110mb.com Abuse

Or on the opposite end a supposedly well intentioned URL redirection service offers no contact to report abusers. (Approximatly 25% of Salem-News.com comment spam uses RubyURL services to mask themselves)

The team behind RubyURL hates spam just as much as you do.

Unfortunately, we cannot prevent people from using the RubyURL service and pasting links into spam emails, wiki pages, and blog comments. If you found your way to RubyURL due to spamming, please to report the offending RubyURL and we’ll flag it for review.

Thanks!
-The RubyURL team

In order to ever keep the Internet somewhat secure people must do more than just ignore or delete problems they must join the fight against it or criminals will always win.

The least you can do

• Install up to date anti virus software (A large amount of spam and malware distribution comes from zombied computers)
• Install and configure a firewall

The better you can do

• Learn to track spam sources(Spam Tracking)
• If you can not keep yourself secure do not connect to the Internet until you can

The flow of hits from this attack continues peaking during the day and slowing down quite a bit at night. Bandwidth is back to normal range with the 403 redirects but they are still getting at new images so someone or bot with a different signature is picking the images for the rest.

Today I just focused on monitoring the logs to see if there were any changes in their pattern, and hopefully finding out how they were getting new images into their que.

I put a large chunk of the Java1.* user agents suspected of being bots into a database for easier searching: Suspected DDOS hits Page containing 9,735 unique IPs with 91966 requests (most of them 403′d).

Because my mod_rewrite to block user agents that match ^Java/1.* the user agent must begin with J this one 70.81.196.195
Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_02 which matched the pattern of nothing but continual image loading made it through.

Now I want to be real careful not to block legitimate traffic so I added a redirect for exactly that user agent as I do not believe it is real.
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(Windows\ XP\ 5\.1\)\ Java/1 [NC,OR]

While still nothing near a serious threat it would be for a site with a lower monthly bandwidth allowance.

Lessons Learned So Far

(1) Keep more than 5 logs. I’ve always browsed over my logs and let them disappear. I didn’t see a need to have them. Now I wish I could look over last months logs to see when this actually began. I will keep for an indefinite time now as space is not an issue. Perhaps work on the quick script I wrote to import the suspected bot hits and routinely insert the newest available logs as the ability to do some quick sql to find patterns would be quite nice.

(2) Some IP’s and User Agents are no good, traffic will not be hurt by shutting them down. While looking for information on this attack noticed how many spam bots and content scrapers were activly abusing the site and added these three even though I’m sure some legitimate people may use them.

RewriteCond %{HTTP_USER_AGENT} ^Jakarta [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget


I might later change the 403 redirect to a custom page alerting the actual user that their user agent is not acceptable because of abuses by others, but I’m assuming very few actual visitors use that form to access the site.

(3) Better understanding of the Linux command line. I’ve always just spit out my log files with cat and occassionally greped that to narrow what comes back, but still not fully using all of the available tools to show me exactly what I wanted.

Show me all the 403 hits in the current log
>cat /var/log/httpd/access_log | grep \ 403\
69.108.125.30 – - [08/Nov/2007:21:54:48 -0800] “GET /stimg/november082007/simpson_oj_350.jpg HTTP/1.1″ 403 241 “-” “Java/1.5.0_12″
I had tried this before but woudl get every hit that had the number 403 in it. Escaping the space before and after the 403 requires that 403 be seperated like a real 403 line would.
Without escapes this one and lots of other lines match because 4034 (size in bytes) counts as a match.

66.249.66.199 – - [08/Nov/2007:22:14:33 -0800] “GET /index.php HTTP/1.1″ 200 4034 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Show me every hit with no referer and no user agent. (first “-” is user agent second is referer)
>cat /var/log/httpd/access_log | grep \”\-\”\ \”\-\”
Everything must be \escaped
I have noticed a few requests for RSS feeds and favicons with no useragent or referer but as a percentage of total hits quite low and not worth the costs of allowing the attackers to succeed in using up bandwidth.

(4) The writers of Salem-News.com upload one or more pictures for each article they post. I have warned in the past that they should optimize the file size before uploading. Of course they do not and a look at the size of the images these bots were aiming for were all 3 to 5 times the needed file size. I manually optimized and re-uploaded some of these reducing most from 80 to 120k to 20 to 30k.

Instead of hoping they follow my suggestions I will recode the image uploader to put it through an optimization step right after upload. This will not only reduce the damage that can be done by this type of attack but will reduce download times for all.

I will continue monitoring this and updating the Suspected DDOS hits Page and writing again if the attack pattern changes or stops.

After stopping the bandwidth waste with mod_rewrite it was time to poor through the log files to try and get some more information on how this was happening.

This morning Wednesday November 7^th the hits from the bots where about the same even though they had been receiving 403 Forbidden pages for more than 12 hours.

Rarely did one single bot hit more than 4 or 5 times an hour which I’m assuming must be on purpose to keep the owners of infected PCs from noticing and actually doing anything to keep them free from viruses and trojans.

I noticed however that they had added new images to their list of files being downloaded. This meant that something other than the bots doing the downloading was selecting the images on their behalf.

I had uploaded a set of images for the comic Nota Bene and noticed within 20 minutes one of them was in the rotation of images being downloaded. In order for these bots to have the address of this image someone or other bot without a user agent of Java/1.(various) and must have visited within 20 minutes of the image being uploaded.

(image uploaded at about 8:05 IP’s not shown for possible privacy reasons)

cat /var/log/httpd/access_log | grep nota_bene-67.jpg
***.**.**.*** - - [07/Nov/2007:08:18:05 -0800] "GET /stimg/november072007/nota_bene-67.jpg HTTP/1.1" 200 23919 "-" "-"
***.**.**.*** - - [07/Nov/2007:08:22:56 -0800] "GET /stimg/november072007/nota_bene-67.jpg HTTP/1.1" 403 239 "-" "Java/1.6.0_03"


All of the hits for the image seemed normal but two, one of the Java bots that got a 403 error as expected and one 4 minutes prior to the first bot hit on that image. It contained no referrer and no user agent.

I parsed all of the log files for instances of that IP and it was scattered about and was only found to have accessed image directories and images not one page or non image hit. A whois of that IP showed it to be an AOL IP address.

I firewall blocked the IP wich I knew wouldn’t last too long as AOL IPs are quite dynamic and within 15 minutes it was back with a slightly different IP. So I added to my mod_rewrite block of the Java/1. to no longer allow an empty user agent as well.


RewriteCond %{HTTP_USER_AGENT} ^-?$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Java/1.*
RewriteRule ^.* - [F,L]


Now this may block some legitimate traffic but it should be minimum and I may write a custom page for those without a proper user agent set to be redirected to explaining why they can not view the site.

Commands I run from time to time to see how many are being blocked. First line shows someones RSS reader getting a 403

cat /var/log/httpd/access_log | grep \ 403\

***.**.**.*** - - [07/Nov/2007:18:58:59 -0800] "GET /index.rss HTTP/1.0" 403 211 "-" "-"
***.**.**.*** - - [07/Nov/2007:18:59:01 -0800] "GET /stimg/november072007/dow_jones.jpg HTTP/1.1" 403 236 "-" "Java/1.6.0-oem"
***.***.**.*** - - [07/Nov/2007:18:59:2 -0800] "GET /stimg/november072007/cheney310.jpg HTTP/1.1" 403 236 "-" "Java/1.5.0_10"


I have not yet had time to fully analyze the log files to see how many unique bots are involved but I roughly estimate at more than 1500 hitting approximatly 400,000 times over the last 3 days. Todays Bandwidth usage was 1/3^rd that of yesterday and 1/8^th of Mondays even though the real visitor and page count is higher.

Now I’m going to have to assume if they want to continue and actually do some damage they will have to alter their user agent string. I’m hoping this won’t be easily done as it would be much harder to stop.

(The last spike was from Digg on a news story)

An investigation began into what could be causing this to happen. Some possible reasons that came to mind:

• Large image files used in heavy hitting stories
• Heavy spider / bot traffic
• Corrupted log file / analyzer

It ended up being what I believe to be a large scale bandwidth sapping DDOS^?

The log files looked normal with visits up so I wrote it off a paranoia. But by Monday I knew something was up. The bandwidth for the day was five times that of Sunday with only a 10% increase in other visitors and page views.

A heavy review of log files showed lots of traffic with variations of the following User-Agents: “Java/1.6.0_03″,”Java/1.6.0_02″,”Java/1.5.0_06″,”Java/1.5.0_02″,”Java/1.5.0_07″, And about 30 other variations of these.

Searching through forums and blogs turned up nothing but information on spam harvesters but all of the hits I was seeing were for the same small set of images and coming from an unbelievable amount of IP addresses all of them hitting on a slow but regular basis.

I tried the robots.txt but within a few hours noticed if anything an increase in the traffic.

I tried to find a pattern to these IPs but they were global and non that I tested were another webserver, leading me to believe a large group of zombie PC’s are loaded with DDOS software and a controller somewhere had sent them a list of images to download. None of the IPs showed up as ever visiting anywhere else on the site, only the small set of images.

So then I began to look for a way to stop them. Because the IP’s were to varied to firewall that was not an option. While the user agent was varied it was quite possible to block them with 2 additional lines to my Rewrite rules

RewriteCond %{HTTP_USER_AGENT} ^Java/1.*
RewriteRule ^.* - [F,L]

Another option would have been using the SetEnvIfNoCase User-Agent ^$ bad_bot way but the Rewrite way worked.
Fight Blog Spam with Apache

Now about 8 hours later the hit count continues at about the same rate but they are just receiving a 403 Forbidden instead of a 100k image on each request.

I pulled all of the hits from apaches log files for later analysis.

cat access_log | grep Java >> DDOS.1107.log


I wonder how many of these hits where from some unsuspecting person sitting on their PC unaware their computer is at the control of someone somewhere far away.

Day two of defending against this has been posted: The Denial of Service Attack Continues