Skip to content

Archive for July, 2008

30
Jul

“Tubes” Senator Indicited for Lying to Feds

Today, Alaska Senator Ted “Series of Tubes” Stevens was indicited for lying to federal investagors. Apparently he falsified some information on finacial records.

Stories like this reaffirm my belief in karma.

Senator Stevens became famous for his explanation of how the internet works by comparing it to a series of tubes, but also explaining that the internet isn’t a big truck. Yes, it sounds just as retarded listening to him explaining it as I did… but I posted a youtube video of him trying to explain.

Now remember, he’s trying to convince people on why Net Neutrality shouldn’t be enforced. Because we all know it’s okay for ISPs to filter data off the internet we’re using… like Charter nuking Vonage connections, Comcast dropping Peer to Peer connections, and the like. Good riddance Ted.

Listen in here:

15
Jul

Shocking… well, not really.

The last few months at my new job I’ve been squishing small and medium bugs to get systems up to par. Service packs, patching, firmware updates, software upgrades, or just organized to make life easier for everyone involved.

Our server room has been one of those infested areas… I’ve been squashing the easy bugs but the room is frankly a disaster waiting to happen. It’s not large by data center measurements, frankly it’s just a small class room with three ceiling mounted cooling units and seven racks of equipment. Three two post network racks and four cabinets for servers. Problem is almost nothing is labled. Power cables, random colors of ethernet, and dayglo orange fiber cables are intertwined in a quilt of choas behind the cabinets. The cable ladder above the racks is about 12 inches too far away and has a large power bus bar below it. But that’s not difficult to fix. Yes, it’s a time consuming job – but working for a college has advantages.

The biggest problem is: I don’t know how much power I have to work with. 30 circuits of power and I haven’t a clue what goes where or how much I’m using.

Today was the big day I was waiting for. An electrician arrived and performed a detailed analysis and audit of our power usage. He started from the UPS inputs and worked through the distribution panel and finally labled and measured the outlets in the server room. This is where my worst fears were realized…

This Close, man!

we were this close to a massive cascading power failure. Three circuits have been identified as being over 75% utilized, one is at 96%…

Bad news: Nine servers are connected to this circuit.

Worse news: Three servers totally reliant on it, both power supplies are connected to this.

Even worse news: Two of those servers are part of a three node ESX cluster with twenty two virtual machines hosted on them.

Worse bad news: If that circuit trips, it’ll force the other six servers to pull power from another circuit almost as loaded, which will most likely put it over the top and trip that second breaker.

UPS Truck Fire

And, to top it all off: Our UPS load is really unbalanced, but not in a way we can fix with medication. You see, this room is fed with three feeds of electricity called “phases” or “legs”. Equipment like large appliances or electric motors run more efficiently using more than one phase. In this case, the UPS (our battery backup device for the servers) pulls electricity equally from all three phases, conditions it, charges its batteries, and then feeds it to a breaker box. In this breaker box are thirty 20A circuits. Each is connected to one of those phases. Our core switches are large units, so they get two circuits (and two phases) for each of their power connections. It’s a bit complicated, but the simple rule is – load the boat evenly and it won’t capsize.

Right now, phase one is running 3% over, phase two is 33% under, and phase three is 24% over average. So the devation between L2 and L3 is 58%! It’s no wonder the UPSs have only been living for two or three years. When a UPS has to supply power to a system, it performs better when the load across all of its connections are close to the same. Deviations up or down simply chew up UPS components and spit them out. Oh, and there is no UPS maintenance by-pass switch so if the UPS dies – the room dies. If we want to replace the UPS we have to kill the room until the hardwire connection is bypassed by an electrician.

But all is not lost.

Now that I have a detailed map of our power usage and outlets that are labeled, I’m throwing together an emergency change plan to migrate servers onto other circuits to reduce the load on the heavily loaded circuits AND to balance the load across phases.

In August we plan on installing new three phase power distribution units from APC with onboard monitoring and access to all three phases on the PDU. This will make balancing and loading a lot easier. Until then, I’m juggling power cables to anonymous power strips… but at least NOW they’re labeled.

Knowing is half the battle.

Half the battle.

8
Jul

TrendMicro Appliance Randomly Blocking

I’ve been fighting an odd issue and finally found a resolution with the assistance of TrendMicro’s support.

A few users (six out of 22k) reported that they weren’t getting email from anyone outside of the network. A few test messages from my web mail accounts (Gmail, Hotmail, and my own domain) revealed an interesting issue.

These few accounts were getting this error:

Hotmail

Reporting-MTA: dns;blu0-omc1-s38.blu0.hotmail.com
Received-From-MTA: dns;BLU119-W30
Arrival-Date: Thu, 3 Jul 2008 06:02:56 -0700

Final-Recipient: rfc822;[deleted@for.security]
Action: failed
Status: 5.7.1
Diagnostic-Code: smtp;554 5.7.1 : Recipient address rejected: Access denied

GMail

This is an automatically generated Delivery Status Notification

Delivery to the following recipient failed permanently:

[deleted@for.security]

Technical details of permanent failure:
PERM_FAILURE: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 554 554 5.7.1 < [deleted@for.security]>: Recipient address rejected: Access denied (state 14).

I tested SMTP connectivity to the Exchange server by telnetting to the device from outside and inside the network to attempt to narrow down the block. Our Exchange server is protected by TrendMicro ScanMail, and we utilize a TrendMicro Interscan Messaging Security Appliance on our DMZ to provide more spam and virus protection.

I narrowed it down to the IMSA appliance but couldn’t locate the problem in the logs. The MTA logs simply stated Access Denied… not very helpful. So after a short wait on hold, TrendMicro support asked me to deactivate the Network Reputation Services, a learning adaptive IP filtering system that blocks spam senders before they finish connecting.

I later found that the NRS is configured on the appliance AND on TrendMicro’s Email Reputation Service website. lets you create an account using your IMSA’s activation code. Then you can log in and configure the “aggressiveness” of the NRS filters.

If you’ve already laid out the cash for the IMSA, get your email servers registered on this site to make sure they don’t get blocked or at least you’ll have a higher rating with other Trend users on the internet.

It was a frustrating problem that I hope nobody else has, but if they do I hope you find this helpful. If this doesn’t fix it, give Trend a call. Enterprise wait time was less than a minute and had me up and running in less than 10 minutes.

5
Jul

Life update.

It’s been a while since I’ve updated this blog, the real life has been keeping me busy but I try to get to the forums every day or two.

My new career has kept me hopping. The college is busy and turn over has lead to an open position that I’m going to try to fill. Unfortunately it was the position that performed similar duties, so the team’s current work load has increased.

I have a few large projects under my wing. I’m working hard to meet our budget restraints but keep service levels high.

We’re looking at new UPS installations in two campus buildings, one a small 6-10KVA and the other is for our core server room weighing in at 60-80KVA. We lack a generator and no budget to buy one so we’ll need to run on battery long enough to shutdown gracefully in the even of a power outage.

Ramp up our ESX cluster and virtualize as much as possible. Many of our HP servers are out of warranty and still run critical apps. I’ll be starting with some of the easier redundant servers and working toward more critical servers as we balance the load and ensure our core switch fabric can handle the increased iSCSI load the ESX and SAN environments will be placing on them.

In my spare time, I’m troubleshooting our TrendMicro AV solution along with Exchange email issues that creep up. Personally, I think Trend is bucket of chum with a pretty bow on it – but now isn’t the time to forklift in a new solution, it’s time to wring as much support out of them as our contact gives us… so far I’m less that impressed at their customer service. It’s frustrating when you want to give your clients the best service possible, but a vendor prevents you from meeting those goals.

On the home front, life has been equally busy. Sam crawled for the first time. He’s getting more mobile every day. Anna scored three ribbons in her last swim competition… I could see her smile from a block away when she brought them home.

Nap time is over for Sam… time to save and log off. Peace!

%d bloggers like this: