cornerblog

mercury offline this morning

2009-09-11T08:48:00.003-04:00

The data center has taken mercury offline due to some sort of attack this morning. They said:

Your server is experiencing a large inbound attack. We placed your server under guard protection, however the attack exceeded the guard threshold and we were forced to null route your IP. The null route will remain in effect until traffic has reached levels manageable by guard.

I'll post more when I know more.

mercury back online

2009-05-19T18:04:00.003-04:00

Mercury is back online and the misbehaving firewall has been disabled.

Sorry, everyone. Needless to say, that security group won't be messing with my servers again.

Still waiting...

2009-05-19T16:51:00.002-04:00

The firewall does seem to be out of control, blocking everything. This issue has been passed to a level 2 tech at the data center...

Mercury down... (maybe?)

2009-05-19T15:54:00.000-04:00

I signed up for a security audit and ongoing monitoring on mercury... It appears they've installed some kind of overly paranoid monitoring system that is blocking legitimate customers at the firewall level.

I seemed to have triggered this myself and can't even tell if the server is up anymore.

I have a ticket open with the planet and we're getting this sorted out ASAP.

mercury restored

2009-03-25T01:28:00.002-04:00

Mercury is back online. I've updated the mod_security rules and enabled suhosin, which adds an extra layer of security.

One thing suhosin does is completely remove the possibility of remote includes for php. So if you have an include() call that starts with "http://" it will no longer work. What you need to do is put the include file on your account and replace the "http" path with the corresponding local file system path.

mercury taken offline

2009-03-25T00:34:00.002-04:00

Apparently there was a security breach on Mercury, and the data center was unable to contact me. They disabled the server several hours ago.

I am working to resolve this issue right now.

mercury down

2009-03-13T16:38:00.002-04:00

Mercury just stopped responding and I'm unable to bring up a remote terminal.

I've opened a ticket for the data center to investigate.

update: it's back online now. Was down for about 8 minutes.

what to do if your old PHP script broke

2009-03-10T12:29:00.001-04:00

Hi guys,

The number 1 problem cropping up for people whose PHP scripts have stopped working correctly since the upgrade is that they rely on the old, insecure way that PHP handled variables passed in from the query string in the URL.

This old behavior is one of the major causes of security holes in PHP. Basically, if the program is not well, written, it may be possible for an attacker to change variables that were not meant to be passed in through the URL, and possibly take control of your site.

Although this only affects a very small percentage poorly written scripts, the PHP community has come up with a much smarter alternative, which is to move these variables into special associated array variables like $_GET, $_POST, and $_REQUEST.

So, if your script depends on a variable named $foo being passed in through the URL, like so...

http://whatever/script.php?foo=bar

... Then you should put this line at the top of your script:


  $foo = $_GET['foo'];      // read from query string

Similarly, you can read from a form post with:


  $foo = $_POST['foo'];     // read from form post

If you prefer to accept both GET and POST requests, you can use this line, which handles either method:


  $foo = $_REQUEST['foo'];  // read from either

Add one of these lines to the top of the file for each variable in your form. (Make sure you put it INSIDE the <?php ... ?> area!)

Having said that, I realize this is short notice, so as an alternative, you can revert to the old behavior for your entire site by adding this line to your .htaccess file:


   php_flag register_globals on

However, at some point in the future, I will probably disable this option completely... So if you don't feel comfortable editing your php scripts yourself, please get in touch with me and ask for help.

For more information on this PHP issue, see here:

http://us2.php.net/register_globals

Thanks!

mail to comcast to be fixed shortly

2009-03-07T14:56:00.001-05:00

It seems comcast is rejecting mail from the new machines until we get reverse dns records (PTR records) set up. I've put a request in to the data center so hopefully this should be fixed this weekend.

SMTP and control panel login fixed

2009-03-06T10:02:00.004-05:00

The SMTP server is working correctly now on all boxes.

(For the curious, the problem was that even though SASL was installed and configured correctly, saslauthd wasn't actually running by default anymore.)

The control panel is also working correctly now. (Though the logfile data is still missing).

post upgrade issues, revised

2009-03-06T09:01:00.003-05:00

fixed:

logfiles have been restored
SSL for mail is now working correctly

in progress

control panel is rejecting logins
SMTP seems to be rejecting logins for some people

(These are two unrelated issues as far as I can tell. The control panel will be fixed shortly. SMTP is still a mystery.)

new ip addresses

2009-03-05T13:24:00.003-05:00

If anyone needs the new IP addresses, they are:

mercury.sabren.com.     IN      A       75.125.245.26
scandium.sabren.com.    IN      A       75.125.250.26
titanium.sabren.com.    IN      A       174.133.41.42
vanadium.sabren.com.    IN      A       174.133.41.90
manganese.sabren.com.   IN      A       174.133.33.154

control panel working

2009-03-05T13:19:00.000-05:00

The control panel should be working again (except for logfile reports).

all servers online

2009-03-05T10:44:00.002-05:00

All servers are now back online.

Still working on:

SSL for mail
old logfiles
getting the control panel hooked up

manganese fully functional

2009-03-05T09:36:00.002-05:00

Mail for manganese (and all servers except titanium) is back online.

Titanium is still chugging along...

manganese and titanium partially restored

2009-03-05T08:50:00.001-05:00

I've turned on the web server for manganese and titanium.

I'm still waiting on mail to finish syncing before people can log back in. Looks like it's getting near the end on both machines.

known issues

2009-03-05T07:07:00.004-05:00

Known issues for anyone looking at the new servers:

SSL isn't quite working yet for mail. (but you can use the webmail interface
Your web logfiles may or may not be there yet.
Control panel changes won't take effect immediately (if you can log in at all).

All of this should be fixed shortly.

scandium and vanadium online

2009-03-05T07:06:00.002-05:00

Scandium and Vanadium are back online.

Titanium and Manganese look like they're pretty close.

mercury online

2009-03-05T06:31:00.003-05:00

Mercury is back online.

The DNS change appears to have gone through as well. Please let me know if you're having any issues.

The other servers are still waiting on rsync. I'll bring them online as they finish. Hopefully it shouldn't be more than another hour or two.

final sync in progress

2009-03-05T04:10:00.002-05:00

The final rsync is in progress. All accounts are locked to prevent file changes during this time (including web,mail,and ftp access).

This will take a couple hours. Speed varies depending on the server.

I am also initiating the DNS changes for all domains.

down for the upgrade

2009-03-05T03:39:00.000-05:00

It's about 3:30AM eastern time, and I'm taking the servers down for maintenance now.

reminder: upgrades tonight

2009-03-04T11:20:00.001-05:00

Just a reminder that the servers will be down for several hours late tonight while I do the final rsync and switch us over to the new boxes.

Hydrogen is already switched over so I'll be posting updates on cornerhost.net as things develop.

innercircle downtime

2009-03-03T08:54:00.002-05:00

~~The innercircle list is down while I figure out how to get mailman to recognize my old data...~~

Never mind. It's back. :)

don't use www for testing

2009-03-02T08:54:00.002-05:00

Just a quick note: the www prefix isn't set up for the test domains. Just use http://yourdomain.com.new-xx.sabren.com/whatever

Test your site on the new servers!

2009-02-28T22:06:00.002-05:00

The new boxes are almost ready.

There are a few things left to install, but most sites should now be operational.

My plan is to make the cutover later this week (probably Wednesday night). In the meantime, please test out your site and help me identify any remaining issues.

To see your site on the new server, append new-xx.sabren.com to your domain, where xx is the short name of the server:

if you're on...	... then add this:
mercury	.new-hg.sabren.com
scandium	.new-sc.sabren.com
titanium	.new-ti.sabren.com
vanadium	.new-v.sabren.com
manganese	.new-mn.sabren.com

So, for example, if your site is:

http://tempytantrum.com/ (on mercury)

You should go to:

http://tempytantrum.com.new-hg.sabren.com/

I've tested a variety of the major software packages (wordpress, drupal, MT, etc) and they all appear to be working, but of course I haven't tested every possible configuration or plugin, so please take this opportunity to check your site out before the switch.

Known issues:

The files on the new boxes are a day or two old. I'll refresh them tonight or tomorrow, and again right before the switch.
The new servers will probably not send not send mail correctly to your cornerhost accounts (they'll deliver locally).
You cannot log into the server with ftp/ssh yet (to keep people from making changes that will get overwritten with the next rsync). If you need to log in, please contact me and I'll give you a temporary password.
python2.2 is no longer available. If your site uses python2.2, please try switching to /usr/bin/python2.5 or /usr/bin/python2.4 (test this on the *current* box)
FastCGI sites are not working yet.

Feel free to contact me directly with any issues you turn up, or post on innercircle to avoid duplicate bug reports.