Skip to content

Posts from the ‘Uncategorized’ Category


External Log Collection for UCS Fabric Interconnects

I’ve been troubleshooting some pretty annoying bugs in our Cisco UCS environments. Most are easily solved by collecting some techsupport files, opening a TAC case and working through the glitch or config issue. However one has me really stumped and frustrated. 

When we collect these techsupport files, more specifically techsupport files directed at a specific chassis – an IO Module will randomly disconnect, reboot, or reconfigure – dropping half or both connections to the fabric (we have one uplink per IOM, two per chassis).

As we continue to troubleshoot with Cisco TAC, we mostly find out later that the tech support file we generate after the issue doesn’t contain the information they need, the logs have rolled over… or over written due to activity in the domain. We gather syslogs religiously but the information necessary isn’t sent out via syslog when it happens. Feature request?

After pressing one of the TAC engineers on my fifth case this year on this issue, he clued me in on a feature for exporting logs to an external server. Click away if you know this, I certainly didn’t.

Here’s a quick and dirty on how to do it with a generic Ubuntu server.

I’m going to write this soup to nuts for someone who’s a novice and never setup a Linux server. By no means will this be hardened and secured for public visibility – just a place for your FIs to dump their logs. Chime in with a comment if you have improvements or suggestions.

  1. Deploy Ubuntu Server on a VM with a few gigs of space
  2. Update your server once its online with these two commands:
    1. sudo apt-get update
    2. sudo apt-get upgrade
  3. Reboot your server once upgrades are complete
  4. Create a dedicated user
    • sudo adduser ucsloguser
    • provide password info but nothing else matters
  5. Let’s assume you’ll be dumping logs to the user’s home directory, so the path will be /home/ucsloguser
    • For a permanent home, you could add a second disk and mount it under a dedicated path or get your neckbeard on and use LVM to create a logical volume you can add disks and grow later. For now we’ll keep it simple, stupid.
  6. Using your favorite SSH client (Terminal, PuTTY, XTerm, etc) connect to your new server using the ucsloguser account to verify you can SSH to your Ubuntu server.
  7. Ok, Linux server is ready to go.

Configuring the Fabric Interconnects to dump logs onto your Ubuntu server

  1. SSH to your FI – doesn’t matter which, both will respect the monitoring change. 
  2. Run the follow commands:
    1. scope monitoring
    2. scope sysdebug
    3. scope log-export-policy
  3. Now we set the log export policy
    1. set hostname [Linux server IP or FQDN if your DNS is updated]
    2. set user ucsloguser
    3. set passwd [press enter, then enter the password of the ucsloguser]
    4. set admin-state yes
    5. set proto scp
    6. set path /home/ucsloguser/
    7. commit-buffer
  4. That’s it. Now log into your Linux server and see if log file .tgz bundles are showing up in your home directory.
Configuring log export policy in UCS
Logs arriving in external server!


Use the command sudo ls -thl to sort by newest to oldest files with a human readable size.

Use the command sudo df -h to show the space consumed. 

Use an SCP utility like WinSCP to retrieve files from your log server so you can send them to Cisco TAC now. 


VMware Tools Downgrade

I ran into a situation where I needed to test upgrading VMware tools using an alternative method other than directly through vCenter or auto update. To do this testing I had to roll back the most recent version of VMware Tools on a VM to an older one. Uninstall latest, reboot, install older, reboot. No sweat, right?

Well this wouldn’t be much of a blog entry if that’s all there was…every time I rebooted after installing the older version – vCenter continued to report the tools version was up to date and everything was A-Ok. What was strange is even running the command line to verify the versions returned the latest version – and I know I used the older tools installer.

C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe -v

After uninstalling the newest version, I used the VMware KBase article for manually uninstalling VMware Tools to verify everything was gone – and it was – until I hit the last step: Delete the %ProgramFiles%\VMware\VMware Tools folder.

I found a single text file called Manifest.txt which contained the current versions of everything that was installed with the latest VMware Tools. I deleted this file and ran the old VMware Tools installer and successfully reported back an “Out Of Date” Tools installation.

Tip: You can grab any version of VMware Tools from VMware’s Packages site here:


Repairing AirPort coverage

I hope this helps someone but I’ve been chasing a wifi problem in my house for a few days and finally got to fixing it.

Equipment: 4th Gen Airport Extreme (802.11 a/b/g/n; 2.4GHz & 5GHz)

Symptoms: poor range, slow speeds outside of the room the airport was located.

Configuration: I have it configured auto everything across the board, no unique SSID for the 5Ghz network, and nothing customized other than DNS – thanks to DDOS on Charter’s DNS servers I swapped in Google and Level 3’s DNS servers.

I loaded a free app on my MacBook called WiFi Explorer that displays wifi signals, noise, and occupied channels but the one built into OS X works equally well. I noticed that the 2.4Ghz network was dropping off for 30-45 seconds every minute even though there are devices on my network that require 2.4GHz.

5GHz was solid but as expected it’s range was poor and signal strength at the distances I needed it to work in my house were very low.

Started with power cycling the router. No difference. Soft then Hard reset, no change. 2.4GHz just wouldn’t stay on.

Launched Airport Utility and reset the AirPort to the factory setting, no difference. So now I’m thinking hardware issue. Of course, this is not under warranty anymore.

The last trick I had available was to roll back the firmware. In the airport utility click on the AirPort Extreme to display the serial number and firmware version. Option click on the version and pick 7.6.3 from the list. The utility downloads and installs the firmware. Really couldn’t be more simple.

Bam all is good after the reboot. Both radios functioning at expected levels. So is it a firmware issue or a glitch? So I decided to upgrade the firmware to the latest. The latest does have some good fixes.

After the reboot everything has been solid again. WiFI Explorer shows 5GHz and 2.4GHz on solid and never dropping off. I’m going to chalk it up to a glitch in the firmware that was cleared by reloading the firmware. The only way to do that for these is to rollback then upgrade again. Luckily though in true apple fashion – the utility does all the hard work and maintains your configurations.

Also the iOS utility offers the same functionality, so it’s easy to repair these.


VMware Visio Stencils

After dredging around looking for Visio Stencils I found the official post, then a broken link, and then finally a working link from Technodrone website. The files themselves are hosted on mediasite which is a pretty obnoxious with pop-ups and seizure inducing ads.

Well, I’m reposting them here for an easy download:


VMWorld and Work

VMware Inc.
Image via Wikipedia

VMWorld 2009  and many other VMWarevirtualisation topics are still rolling around in my head. So many ideas, so many paths… so many vendors calling me after the conference.

Some interesting tidbits of discussions included baremetal VMWare View clients. In response to portable thinclients like the WyseMobile Thinclient, VMWare wants to install its hypervisoron your hardware then install the VM workstation on top of that – checking in the deltas to a central server for management, backup, and business continuity.

VMWare View workstations with support for VoIP, presumably leveraging the PCoIP protocol learned from Teradici.

And this week after a 64GB memory upgrade on our virtual desktop environment, the talk of thinclients has risen to a roar. It looks like we’re going to be spending upwards of a half a million on workstation refresh this year – do we plunk down a core duo micro tower twenty five times a room or invest in a virtual desktop environment that’s infinitely easier to manage and much more mobile.

We’re also implementing a new NetAppSAN. The gear just arrived this afternoon – five pallets worth. It’s a thing of beauty, and it’s not even out of the box yet. Dual FAS3140 controllers, 28 x 450GB 15,000 RPM FiberChannel drives and 14 x 1TB 7200 RPM SATA drives.

We’ve already moved the LeftHand/MPC equipment to a temp rack  until we can migrate the data off of them.

It’s going to be a busy week.

Reblog this post [with Zemanta]
%d bloggers like this: