Cementhorizon Status

2009-02-22T09:32:00.000-08:00

This morning at 8:37am cementhorizon became unavailable due to a server problem. I'll have it back up Monday morning. eloise.cementhorizon.com is unaffected.

2009-01-03T16:08:00.000-08:00

At 9:06 AM PST Saturday Jan. 3rd gloria became unavailable. I will request a power cycle of the server which will not happen until Monday morning. All services served from gloria will be unavailable until Monday morning.

2008-12-21T14:41:00.001-08:00

Saturday December 20th at 10:09am eloise.cementhorizon.com and kalxstaff.org went down for scheduled maintenance while we moved the server from it's existing facility which is closing down at the end of the year to its new facility in Sacramento. The server came back up as planned but there were settings I failed to setup correctly. I've only just now gotten back to a computer to fix them. Those sites came back up at 2:26pm today Sunday December 21st.

2008-12-05T09:00:00.001-08:00

Starting around 1:30am November 29th sites hosted on the odessa server (e.g. photo galleries, confluence, jira) started experiencing 5% packet loss. This has caused the sites to be intermittently unavailable. Nevin is going to go down to the datacenter tomorrow and swap out what he suspects is a bad cable.

2008-11-05T14:20:00.000-08:00

On 10/29 at 10:14am odessa (the new server) restarted for some reason. I'm looking into the cause of this. Additionally I moved DNS services over from gloria to odessa a month ago. I forgot to configure DNS to automatically startup when the server restarted and as a result DNS services were down at that point.

Our backup DNS provider has been providing DNS resolution since 10/29 at 10:14am but finally gave up realizing that we weren't fixing our DNS server this morning 11/5 at 8:45am.

Between 11/5 at 8:45am and 2:02pm all cementhorizon DNS services were down. The result of this is that though our web services were up, they couldn't be reached by name (for example, typing in http://www.cementhorizon.com/ into your browser would not pull up the web page). The second result is that email sent to addresses @cementhorizon.com during this 5 hour window would be delayed potentially up to 10 or 12 hours from when it was sent.

I've set DNS to autostart in case the server reboots again and I've also setup monitoring explicitly on DNS so that I'm alerted if this problem occurs again.

2008-10-07T13:47:00.000-07:00

From Sunday October 5th at 9:30am until Tuesday October 7th at 1:19pm cementhorizon and associated sites were down. This was caused by blog spam triggering oom-killer. There was nobody at the site on Monday to powercycle the server.

2008-09-23T16:44:00.000-07:00

eloise.cementhorizon.com went down yesterday. I've moved kalxstaff.org back to gloria and it should be available shortly. I'll try to send eloise.cementhorizon.com to a maintenance page.

2008-08-24T15:33:00.001-07:00

Yesterday morning, Saturday Aug 23 at 3:05am the server became unavailable. I'll have it powercycled tomorrow morning.

2008-08-15T00:09:00.001-07:00

At 12:10pm today Thursday August 14 the site became unavailable. I've requested our provider restart the server but I expect it won't happen until Friday morning.

2008-06-22T21:04:00.000-07:00

Cementhorizon became unavailable at around 9pm tonight, Sunday night. I'll keep trying to contact the server but if I'm unable to, I'll have to wait to have it powercycled tomorrow morning.
-Gene

2007-12-10T08:41:00.001-08:00

At 8:15am this morning our provider powercycled our server and it came back up. Sorry for the downtime.

2007-12-08T14:03:00.000-08:00

at 1:50pm the Cementhorizon web server became unavailable (due to my own screwing around on it. Doh!). Email (since it's been moved to google) is still working fine, it's just that the various websites hosted are unavailable. I'll contact our provider and get the server powercycled, though I don't expect it will happen until Monday morning.
-Gene

2007-11-16T00:33:00.001-08:00

I've completed all email migration. My original method of transferring the email turned out to be way too slow and imperfect. I have to thank Ryan Grove for writing a simple and elegant piece of code that allowed me to use an alternative method which is much faster to move all the email ( http://wonko.com/page/about )

I've moved over everyone's address book entries as well.

At this time you can login to http://mail.cementhorizon.com/ or http://mail.chuckbeat.com/ and see all of your old email as well as all new email. All email going forward will be delivered here, in your google apps account.

Please login and change your password to something only you know. I will check back in 2 weeks and for anyone who hasn't changed their password, I'll set it as required for your next login.

I'll leave the old email where it is for the time being. When I get back to the states in 2 weeks I'll retire it.

If anyone has questions about how to use the new interface, how it works, if you're having trouble with anything, or if any mail seems missing let me know. I went through and verified email counts for every user and every folder so we should be good.

Thanks for everyone's cooperation and patience. I'm really glad to have this completed.

-Gene

2007-11-15T00:04:00.000-08:00

Between 12:02am and 12:18am I cutover all new email to google. I'll now begin moving all of the old email.

2007-11-12T23:23:00.000-08:00

Coming up this Wednesday evening (11/14/2007) I'll be cutting over all email to google apps. I've emailed everyone details. I'll post here again on Wednesday with status updates on the move as it happens.

2007-09-05T10:38:00.002-07:00

I went to the colo this morning and cleaned the remaining file systems. I found that they'd never been cleaned since I built the system because I hadn't configured them to clean themselves regularly. I've now set all volumes to to a filesystem check every 3 months. I replaced one of the old disks as well. I also paid for hosting up to the end of 2007.

Last night I enabled a set of backup DNS servers and a set of backup mail servers in Reno. Now if the server fails, we still have DNS resolution and we control the queuing of the mail instead of depending on the sender.

At this point I feel confident that the recent problems of the last week (other than the power failure) have been solved and we're back at the level of stability we had before (good, but not perfect).

I'm planning to migrate email to google this week but with time running short and me being out of town this coming weekend it may not happen.

2007-09-05T10:38:00.001-07:00

I went down and tried to figure out the problem. I had little luck determining what was wrong but I did get the problem to manifest again. I then succeeded in cleaning the filesystem on the main volume on the server which had some corruption on it. Since then I haven't seen the problem so hopefully that fixed it. I'm going to go back to the factory tomorrow morning and attempt to clean the remaining filesystems and replace some disks which are really old.

2007-09-05T10:37:00.001-07:00

The server was power cycled this morning at 7:05am and came back up. All email that senders were attempting to send to our users during that time period will be queued up on the sending side and senders will continue to attempt to send the mail. Typically senders attempt this for up to 72 hours before giving up.

2007-09-05T10:36:00.001-07:00

Mail services started acting funny on Cementhorizon. I went on and couldn't figure it out so I attempted to reboot the server. The machine never came back up. There was nobody with physical access to the server over the weekend which was the reason it was down the entire time.

2007-09-05T10:35:00.001-07:00

PG&E and AT&T finished getting everything fixed and our server is back online

2007-09-05T10:33:00.001-07:00

A semi truck drove into the power substation that serves the building our server is hosted in. This caused a loss of power to the building. It also caused electrical damage to the telecom equipment which serves the building. All sites are down.

2007-09-04T10:28:00.000-07:00

The same mail problem occured again. It appears that the cause of both problems is some hardware or disk issue. I'm going to be riding down to San Leandro to the server at around noon to try to figure out the problem.