Currently many (More than one is to many) US Government websites are under the control of criminals. Finding Hacked Goverment and School Web Sites Try a Google Search for site:*.gov porn or swap the words for other terms and you will most likely turn up sites that redirect to obviously not Government sites for pornography or Malware downloads.

Now it is understandable that sometimes hackers win a battle and gain control over something they shouldn’t. But it has been days since I first read of this and began finding sites owned by hackers. This should not be and the agencies involved should be far quicker at fixing the problem or lose their right to be responsible for maintaining a public website.
My email to one of the websites hacked.

I understand that sometimes sites become hacked but you should be far
quicker at shutting down or correcting the problem. Or you should not
be allowed on the Internet.

Now I receive from multiple sources spam emails and blog postings. I could just quickly delete them and move on but that does little to fight the problem. By contacting the sources and eventual domain name destinations of these attacks far more can be accomplished.

Most of my spam both email and blog spam comes from commen peoples computer or advertises sites that are quick and easy to put up or hide behind a URL redirection services. At least one of them 110mb.com a free hosting site quickly responds and removes the offending site.:

Site has been removed.

Thank you for reporting it.

Best Regards,
110mb.com Abuse

Or on the opposite end a supposedly well intentioned URL redirection service offers no contact to report abusers. (Approximatly 25% of Salem-News.com comment spam uses RubyURL services to mask themselves)

The team behind RubyURL hates spam just as much as you do.

Unfortunately, we cannot prevent people from using the RubyURL service and pasting links into spam emails, wiki pages, and blog comments. If you found your way to RubyURL due to spamming, please to report the offending RubyURL and we’ll flag it for review.

Thanks!
-The RubyURL team

In order to ever keep the Internet somewhat secure people must do more than just ignore or delete problems they must join the fight against it or criminals will always win.

The least you can do

• Install up to date anti virus software (A large amount of spam and malware distribution comes from zombied computers)
• Install and configure a firewall

The better you can do

• Learn to track spam sources(Spam Tracking)
• If you can not keep yourself secure do not connect to the Internet until you can

The flow of hits from this attack continues peaking during the day and slowing down quite a bit at night. Bandwidth is back to normal range with the 403 redirects but they are still getting at new images so someone or bot with a different signature is picking the images for the rest.

Today I just focused on monitoring the logs to see if there were any changes in their pattern, and hopefully finding out how they were getting new images into their que.

I put a large chunk of the Java1.* user agents suspected of being bots into a database for easier searching: Suspected DDOS hits Page containing 9,735 unique IPs with 91966 requests (most of them 403′d).

Because my mod_rewrite to block user agents that match ^Java/1.* the user agent must begin with J this one 70.81.196.195
Mozilla/4.0 (Windows XP 5.1) Java/1.6.0_02 which matched the pattern of nothing but continual image loading made it through.

Now I want to be real careful not to block legitimate traffic so I added a redirect for exactly that user agent as I do not believe it is real.
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/4\.0\ \(Windows\ XP\ 5\.1\)\ Java/1 [NC,OR]

While still nothing near a serious threat it would be for a site with a lower monthly bandwidth allowance.

Lessons Learned So Far

(1) Keep more than 5 logs. I’ve always browsed over my logs and let them disappear. I didn’t see a need to have them. Now I wish I could look over last months logs to see when this actually began. I will keep for an indefinite time now as space is not an issue. Perhaps work on the quick script I wrote to import the suspected bot hits and routinely insert the newest available logs as the ability to do some quick sql to find patterns would be quite nice.

(2) Some IP’s and User Agents are no good, traffic will not be hurt by shutting them down. While looking for information on this attack noticed how many spam bots and content scrapers were activly abusing the site and added these three even though I’m sure some legitimate people may use them.

RewriteCond %{HTTP_USER_AGENT} ^Jakarta [OR]
RewriteCond %{HTTP_USER_AGENT} ^libwww-perl [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget


I might later change the 403 redirect to a custom page alerting the actual user that their user agent is not acceptable because of abuses by others, but I’m assuming very few actual visitors use that form to access the site.

(3) Better understanding of the Linux command line. I’ve always just spit out my log files with cat and occassionally greped that to narrow what comes back, but still not fully using all of the available tools to show me exactly what I wanted.

Show me all the 403 hits in the current log
>cat /var/log/httpd/access_log | grep \ 403\
69.108.125.30 - - [08/Nov/2007:21:54:48 -0800] “GET /stimg/november082007/simpson_oj_350.jpg HTTP/1.1″ 403 241 “-” “Java/1.5.0_12″
I had tried this before but woudl get every hit that had the number 403 in it. Escaping the space before and after the 403 requires that 403 be seperated like a real 403 line would.
Without escapes this one and lots of other lines match because 4034 (size in bytes) counts as a match.

66.249.66.199 - - [08/Nov/2007:22:14:33 -0800] “GET /index.php HTTP/1.1″ 200 4034 “-” “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”

Show me every hit with no referer and no user agent. (first “-” is user agent second is referer)
>cat /var/log/httpd/access_log | grep \”\-\”\ \”\-\”
Everything must be \escaped
I have noticed a few requests for RSS feeds and favicons with no useragent or referer but as a percentage of total hits quite low and not worth the costs of allowing the attackers to succeed in using up bandwidth.

(4) The writers of Salem-News.com upload one or more pictures for each article they post. I have warned in the past that they should optimize the file size before uploading. Of course they do not and a look at the size of the images these bots were aiming for were all 3 to 5 times the needed file size. I manually optimized and re-uploaded some of these reducing most from 80 to 120k to 20 to 30k.

Instead of hoping they follow my suggestions I will recode the image uploader to put it through an optimization step right after upload. This will not only reduce the damage that can be done by this type of attack but will reduce download times for all.

I will continue monitoring this and updating the Suspected DDOS hits Page and writing again if the attack pattern changes or stops.

After stopping the bandwidth waste with mod_rewrite it was time to poor through the log files to try and get some more information on how this was happening.

This morning Wednesday November 7^th the hits from the bots where about the same even though they had been receiving 403 Forbidden pages for more than 12 hours.

Rarely did one single bot hit more than 4 or 5 times an hour which I’m assuming must be on purpose to keep the owners of infected PCs from noticing and actually doing anything to keep them free from viruses and trojans.

I noticed however that they had added new images to their list of files being downloaded. This meant that something other than the bots doing the downloading was selecting the images on their behalf.

I had uploaded a set of images for the comic Nota Bene and noticed within 20 minutes one of them was in the rotation of images being downloaded. In order for these bots to have the address of this image someone or other bot without a user agent of Java/1.(various) and must have visited within 20 minutes of the image being uploaded.

(image uploaded at about 8:05 IP’s not shown for possible privacy reasons)

cat /var/log/httpd/access_log | grep nota_bene-67.jpg
***.**.**.*** - - [07/Nov/2007:08:18:05 -0800] "GET /stimg/november072007/nota_bene-67.jpg HTTP/1.1" 200 23919 "-" "-"
***.**.**.*** - - [07/Nov/2007:08:22:56 -0800] "GET /stimg/november072007/nota_bene-67.jpg HTTP/1.1" 403 239 "-" "Java/1.6.0_03"


All of the hits for the image seemed normal but two, one of the Java bots that got a 403 error as expected and one 4 minutes prior to the first bot hit on that image. It contained no referrer and no user agent.

I parsed all of the log files for instances of that IP and it was scattered about and was only found to have accessed image directories and images not one page or non image hit. A whois of that IP showed it to be an AOL IP address.

I firewall blocked the IP wich I knew wouldn’t last too long as AOL IPs are quite dynamic and within 15 minutes it was back with a slightly different IP. So I added to my mod_rewrite block of the Java/1. to no longer allow an empty user agent as well.


RewriteCond %{HTTP_USER_AGENT} ^-?$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^Java/1.*
RewriteRule ^.* - [F,L]


Now this may block some legitimate traffic but it should be minimum and I may write a custom page for those without a proper user agent set to be redirected to explaining why they can not view the site.

Commands I run from time to time to see how many are being blocked. First line shows someones RSS reader getting a 403

cat /var/log/httpd/access_log | grep \ 403\

***.**.**.*** - - [07/Nov/2007:18:58:59 -0800] "GET /index.rss HTTP/1.0" 403 211 "-" "-"
***.**.**.*** - - [07/Nov/2007:18:59:01 -0800] "GET /stimg/november072007/dow_jones.jpg HTTP/1.1" 403 236 "-" "Java/1.6.0-oem"
***.***.**.*** - - [07/Nov/2007:18:59:2 -0800] "GET /stimg/november072007/cheney310.jpg HTTP/1.1" 403 236 "-" "Java/1.5.0_10"


I have not yet had time to fully analyze the log files to see how many unique bots are involved but I roughly estimate at more than 1500 hitting approximatly 400,000 times over the last 3 days. Todays Bandwidth usage was 1/3^rd that of yesterday and 1/8^th of Mondays even though the real visitor and page count is higher.

Now I’m going to have to assume if they want to continue and actually do some damage they will have to alter their user agent string. I’m hoping this won’t be easily done as it would be much harder to stop.

(The last spike was from Digg on a news story)

An investigation began into what could be causing this to happen. Some possible reasons that came to mind:

• Large image files used in heavy hitting stories
• Heavy spider / bot traffic
• Corrupted log file / analyzer

It ended up being what I believe to be a large scale bandwidth sapping DDOS^?

The log files looked normal with visits up so I wrote it off a paranoia. But by Monday I knew something was up. The bandwidth for the day was five times that of Sunday with only a 10% increase in other visitors and page views.

A heavy review of log files showed lots of traffic with variations of the following User-Agents: “Java/1.6.0_03″,”Java/1.6.0_02″,”Java/1.5.0_06″,”Java/1.5.0_02″,”Java/1.5.0_07″, And about 30 other variations of these.

Searching through forums and blogs turned up nothing but information on spam harvesters but all of the hits I was seeing were for the same small set of images and coming from an unbelievable amount of IP addresses all of them hitting on a slow but regular basis.

I tried the robots.txt but within a few hours noticed if anything an increase in the traffic.

I tried to find a pattern to these IPs but they were global and non that I tested were another webserver, leading me to believe a large group of zombie PC’s are loaded with DDOS software and a controller somewhere had sent them a list of images to download. None of the IPs showed up as ever visiting anywhere else on the site, only the small set of images.

So then I began to look for a way to stop them. Because the IP’s were to varied to firewall that was not an option. While the user agent was varied it was quite possible to block them with 2 additional lines to my Rewrite rules

RewriteCond %{HTTP_USER_AGENT} ^Java/1.*
RewriteRule ^.* - [F,L]

Another option would have been using the SetEnvIfNoCase User-Agent ^$ bad_bot way but the Rewrite way worked.
Fight Blog Spam with Apache

Now about 8 hours later the hit count continues at about the same rate but they are just receiving a 403 Forbidden instead of a 100k image on each request.

I pulled all of the hits from apaches log files for later analysis.

cat access_log | grep Java >> DDOS.1107.log


I wonder how many of these hits where from some unsuspecting person sitting on their PC unaware their computer is at the control of someone somewhere far away.

Day two of defending against this has been posted: The Denial of Service Attack Continues

The first step I take is to use The W3C Markup Validation Service. An invaluable free service allowing you to easily see a broad range of mistakes such as non valid attributes, failure to close tags or improperly nested tags.

A large amount of errors came on my first try using the validation services. Errors I never saw from my browsers, but I hate bringing up my site from a friends computer only to see it displayed horribly. But it only took a few changes with the help of the validation service to bring it into at least 4.01 translational compliance.

Next I wanted to check on load time, which I knew would be bad because of the large number of images, and many nested tables. There were quite a few results googling for a speed analyzer and I ended up preferring Web Page Analyzer - free website optimization tool website speed test check website performance report from web site optimization.

My initial tests showed up at over 800k for all html, images, javascript and css combined. Most of this was from the large number of images, and I knew of know way to get rid of that and still be able to keep the lay out preferred. But by optimizing the larger images (opening with Gimp and saving at 75 to 80% usually greatly reduces file size without reducing quality enough to be noticed), getting rid of a lot of the tables (Still more to come), reducing some of the redundant CSS, and removing unneeded white space from the html and css, stopped thumbnails from showing on articles toward the bottom of the front page. I did get it reduced to 465k so about 40%. Still a long way from what the service recommends but far better than what it was.

While removing unneeded whitespace from files seemed to do little, if you remove 1k or 2k per page view it can aded up dramatically on a heavily traveled site.

Next goal is to remove all unneeded tables and go xhtml

The Luxury Cruise to Mars advertisement caught my attention, I had to click on it but if you’ll notice the

www.example.com

Is not a valid domain name. Clicking on it will lead to this:

You have reached this web page by typing "example.com",
"example.net",
or "example.org" into your web browser.

These domain names are reserved for use in documentation and are not available
for registration. See RFC
2606, Section 3.

Is this Google running a test, a customer running a test.

Step 1-1 Create plain white image

Create a plain white image - the size should depend upon how big of an effect you wish. I ended up using 420 wide by 256 (Drop Shadow will add 30px to both height and width) just because it was the size of a previous image I was working on.

I didn’t want it square and may go back and shorten the height just a bit more later on.

Step 1-2 Apply Drop Shadow

Now I applied a Script-Fu > Shadow > Drop Shadow: which increased the image size by 30 pixels in each direction by adding a transparent bar to the right and bottom of the image with a black (My current foreground color) shadow.

I then flattened the image - getting rid of the transparency to just have a plain white square with the shadow effect in the lower right corner.

Step 1-3 - fading the shadow

I liked what I had but because of the dynamic size of the div I was going to use this for I wanted the shadow to fade when towards the top and left. For this I used the gradient tool and used the FG to transparent.

Clicking on the top and pulling down until meeting with the lower shadow causing the right shadow to fade the higher it went. I then did the same for the bottom shadow clicking on the left of the image and pulling right.

You can mess with this for a while undoing and redoing until the fade looks like what you want.

I then uploaded the final image up and started looking for the proper CSS file and class to edit.

Step 2-1 Editing the CSS

The post class was the class I needed to edit and the css file was homepage.css. If I was still using the default them it would have just been style.css. most themes use different names and multiple CSS files.

My theme had no real styling for the post div but had it joined up with a few others for text-align: center
So I removed it from that list and created an entry just for it.
My first attempt it didn’t look to good and the text of the post over ran the shadow on the right and crowded the bottom so I added some padding to the right and bottom.


.post {
background-image: url('/wp-content/themes/tuned-100/images/drop_shadow_lr.png');
background-position: 100% 100%;
background-repeat: no-repeat;
padding-right: 50px; /*text over-ran the shadow*/
padding-bottom: 50px;/*so added some padding*/
}


I have thought about trying different colors for the drop shadow as it might look better with a color matching up with other parts of the page.

Fairly quick and easy - it took much longer creating the screen shots and writing this entry.

It went well after relearning how to set up virtual hosts for the sub-domains as I havn’t done that in a few months.

httpd.conf partial
# Use name-based virtual hosting.
#Remember to uncheck this
NameVirtualHost *:80
#

Then repeat this for each virtual host

<VirtualHost *:80>
DocumentRoot “/path/to/matts/blog”
ServerName designs.salem-news.com
<Directory “/path/to/matts/blog”>
allow from all
Options +Indexes
#mod_rewrite for wordpress
  <IfModule mod_rewrite.c>
   RewriteEngine On
   RewriteBase /
   RewriteCond %{REQUEST_FILENAME} !-f
   RewriteCond %{REQUEST_FILENAME} !-d
   RewriteRule . /index.php [L]
  </IfModule>
</Directory>
</VirtualHost>

The basic changes done from within the administration section was fairly easy. If I was willing to set my permissions correctly I could edit more deeply from within the administration as all template files can be edited but I prefer to do that myself.

After getting comfortable with altering the layout I decided to dump the default theme and try another for a better starting place. I’m not a big fan of narrow sites with large amount of white space down either side so I installed Tuned 1.00 as as starting point.

It seems as easy to edit as the default theme but is fluid and comes with an adsense tool built in (Although I’ll probably try and find a plugin just for further testing purposes)

While I did find several with more features than WordPress I ended up going with it as the winner. The one I really wanted was still too buggy (and is not quite blogging) and would take far more time than I have to get running properly (Will discuss in another post.)

WordPress was not only easier to install and get at least a default blog up and running it is also (IMHO) far easier to modify and customize, and has a large number of plug-ins and templates.

Initial installation consisted of creating a database and entering that info (database name user name pass word) into one config file. Opening the install file. No CHMODing needed, I was sure I was missing a few steps here as all of the others I tried required lots of this.

Next came customizing…