|
Wednesday, November 12. 2008
XMission Outage
---------------
XMission experienced a serious outage while we were performing some
standard UPS maintenance today. The outage affected all services and
started at approximately 2:00 p.m. on Tuesday, November 11th. Network
services for many were partially restored by about 2:30 p.m. but some
other services required a lot of attention and took much longer.
Details
-------
About 40% of our data center, including our server room, suffered a power
outage when a technician flipped a mislabled breaker during some standard
maintenance on one of our 3 UPS units. Although the power outage was
momentary, servers and routers often respond very poorly to losing power
and sometimes take extensive work to come back up. Unfortunately, such was
the case today with many systems.
Seriously Affected Systems
--------------------------
** An important router, which some connections and servers rely on,
required extensive attention from our network administrators.
** DNS (Domain Name Service) was sporadic for some customers for over an
hour.
** Email services were down for over 5 hours.
** Web hosting suffered the longest outage because our NetApp storage
appliance which houses all customer files and web sites lost multiple
hard drives. As a result, we are currently restoring files to our new
NetApp 2020 from our November 9th backup, which will take many hours
yet to complete. We recently purchased this new NetApp and were merely
days away from getting it online.
Conclusion
----------
Today's outage was exascerbated by multiple systems responding poorly to
losing power. In spite of the holiday, our systems administrators were on
site within minutes and continue to work tirelessly to restore all
services. In the end, we should have performed this maintenance on a day
when our systems administrators were on site because problems can arise no
matter how carefully you proceed.
Sunday, September 28. 2008
Our blog has finally outgrown my Pentium 3 with 256MB of RAM - I'm shuffling things around to make it work a little
better. In the meantime, you may see some outages or other strange messages. Please bear with me.
Tuesday, May 27. 2008
s9y upgraded to 1.3.1
PHP upgraded to 5.2.6
Apache upgraded to 2.2.8
Let me know if anything's broken.
Thursday, April 3. 2008
He Said It
MG:
so you're coming up for Rock Band and Racquetball, right?
AoD:
OH, YES
MG:
great
AoD:
That's like asking someone if they're up for a great meal, a nice hotel, and awesome sex. Clearly Ming's is the great
meal, I'll leave you to deduce the other two relationships
My lords, my ladies ( and everybody else here not
sitting on a cushion! ) let me present a link to a comment AoD made this week.
Continue reading "From the New Game Show"
Friday, February 1. 2008
When Radar was here tonight, after sharing Rodgasms of
laughter, we were looking at the blog and noticed that the per-category view was not working - a bug that crept in
during the 1.2.1 upgrade due to a missed SQL upgrade script on my part. This has now been rectum, you can again enjoy
per-category viewing.
Wednesday, January 30. 2008
[Ed: moved image below the fold per Daboo's request]
Continue reading "Just Marvey"
Tuesday, January 29. 2008
We've moved to s9y - that's the software used to run this blog - version 1.2.1.
As always, please let me know ASAP if you see any issues.
Saturday, January 19. 2008
Daboo pointed out that people were putting spam and offensive comments in the gallery, so I've disabled comments there
for the time being.
Wednesday, January 16. 2008
The server may be up-and-down over the next few days
Friday, January 4. 2008
Our server was finally shipped, and arrived (at the wrong house) yesterday. Luckily, the person who got the package
noticed my phone number on the shipping label and called me to tell me they had my package. I drove over there last
night and picked it up.
Amazingly enough, the machine booted and all data seems to remain in place - I was CERTAIN that they would destroy my
box in the process of removing it from the racks, and blame it on the shipping.
I'm currently working on getting bids from data centers, and hope to have the machine back online sometime in February.
Tuesday, January 1. 2008
Our production server is offline, and awaiting shipment in San Jose, as is our secondary server. Our tertiary server,
the first off-site backup for the other two machines, had a hard drive failure this morning (thanks for the call,
Sideshow). Our secondary off-site backup has more limited space and bandwidth, so I have (to this point) only been
performing secondary off-site backups once a month...which means that when the hardware failed this morning, we had to
go back to a 1 month old version. If I don't have more info on when the other servers will show up by the end of this
week, I will locate a better secondary off-site provider, and move the current one to a tertiary role.
Honestly, I just can't provide more redundancy than that  Sorry for the lost work, everyone.
Friday, August 31. 2007
Now that most of you have had a chance to look at http://www.internetisseriousbusiness.com/ I just wanted to let you know how bad it got the Ancient of Days:
Joey: CURTIS
FREAKING
MORTENSEN
You are my arch-nemesis
It's a darn good thing for your sake that you're fleeing to St. George to outwait the days of my wrath
me: What's the matter?
internet is serious business?
Joey: yes
jerkface
me: enjoyed every click, didn't you?
Joey: I lost 154 tabs
me: roflmao
I'm sorry about your tabs, but that just made my day.
I acquired a serious headache at lunch, but knowing that you found that site as frustrating as I did made this day so
much better. 
I can't wait until Anne and Katie click on that link.
Saturday, August 25. 2007
Now that we're a paying customer of our ISP, they've moved our server into a better location. In the move, it seems the
database server didn't come back up on its own. As soon as I noticed, I manually restarted the database and everything
seems fine now.
Sunday, August 19. 2007
Well, I finally broke down and upgraded php on jubilee (the machine that hosts this blog) to 5.2.0. The thing that was
holding me back was testing s9y under this version of PHP, and I finally made time for it this weekend. Everything
appears to be going fine, but please let me know ASAP if you see any problems.
Hopefully, this will solve the problem of the blog going down and not coming back up every Sunday morning.
Friday, May 4. 2007
We have moved to the new 1.1 branch of s9y, our blogging software. Among other things, this should allow for better page
compression resulting in a smaller bandwidth bill for yours truly.
As always, please let me know immediately if you see any issues.
|
Comments
Fri, 21.11.2008 02:44
Um - I already did, like 2 mon ths ago (~30 minutes into epis ode 2)Comments ()
Thu, 20.11.2008 19:48
Then stop watching Heroes.Comments ()
Thu, 20.11.2008 19:15
Amen brother. Amen.Comments ()
Thu, 20.11.2008 13:55
Very cool. I like the smoky sh ading you've got going. It loo ks very sophisticated.Comments ()
Thu, 20.11.2008 12:59
Vampires like to dance next to candy canes too, you know!Comments ()
Thu, 20.11.2008 11:16
Oh, they're all vampiresComments ()
Thu, 20.11.2008 08:53
I really like zuce; it's cool and simultaneously disorientin g how I keep expecting her sho ulders and the dress bel [...]Comments ()
Wed, 19.11.2008 17:56
I miss the PBF so bad. Good t imes!!Comments ()
Tue, 18.11.2008 14:12
It all comes down to both side s actually doing their own res earch to find out what the arg uments are with prop 8 p [...]Comments ()
Tue, 18.11.2008 13:45
You're one up on me - I read t he list and thought "There's n ot a single item on there that is worth what they're a [...]Comments ()
Fri, 14.11.2008 16:12
Is it sad I read all of those and the thing I wanted the mos t was the gas grill? Am I get ting old?Comments ()
Tue, 11.11.2008 23:29
Does it say something about ou r blog that the first three co mments are all by the women wh o frequent this blog? [...]Comments ()
Tue, 11.11.2008 12:57
That's so exciting for you guy s! Keep us posted!Comments ()
Tue, 11.11.2008 11:53
AWESOME! So excited for you!Comments ()
Tue, 11.11.2008 11:08
CONGRATULATIONS!!!!!!Comments ()