External Log Collection for UCS Fabric Interconnects

I’ve been troubleshooting some pretty annoying bugs in our Cisco UCS environments. Most are easily solved by collecting some techsupport files, opening a TAC case and working through the glitch or config issue. However one has me really stumped and frustrated. 

When we collect these techsupport files, more specifically techsupport files directed at a specific chassis – an IO Module will randomly disconnect, reboot, or reconfigure – dropping half or both connections to the fabric (we have one uplink per IOM, two per chassis).

As we continue to troubleshoot with Cisco TAC, we mostly find out later that the tech support file we generate after the issue doesn’t contain the information they need, the logs have rolled over… or over written due to activity in the domain. We gather syslogs religiously but the information necessary isn’t sent out via syslog when it happens. Feature request?

After pressing one of the TAC engineers on my fifth case this year on this issue, he clued me in on a feature for exporting logs to an external server. Click away if you know this, I certainly didn’t.

Here’s a quick and dirty on how to do it with a generic Ubuntu server.

I’m going to write this soup to nuts for someone who’s a novice and never setup a Linux server. By no means will this be hardened and secured for public visibility – just a place for your FIs to dump their logs. Chime in with a comment if you have improvements or suggestions.

  1. Deploy Ubuntu Server on a VM with a few gigs of space
  2. Update your server once its online with these two commands:
    1. sudo apt-get update
    2. sudo apt-get upgrade
  3. Reboot your server once upgrades are complete
  4. Create a dedicated user
    • sudo adduser ucsloguser
    • provide password info but nothing else matters
  5. Let’s assume you’ll be dumping logs to the user’s home directory, so the path will be /home/ucsloguser
    • For a permanent home, you could add a second disk and mount it under a dedicated path or get your neckbeard on and use LVM to create a logical volume you can add disks and grow later. For now we’ll keep it simple, stupid.
  6. Using your favorite SSH client (Terminal, PuTTY, XTerm, etc) connect to your new server using the ucsloguser account to verify you can SSH to your Ubuntu server.
  7. Ok, Linux server is ready to go.

Configuring the Fabric Interconnects to dump logs onto your Ubuntu server

  1. SSH to your FI – doesn’t matter which, both will respect the monitoring change. 
  2. Run the follow commands:
    1. scope monitoring
    2. scope sysdebug
    3. scope log-export-policy
  3. Now we set the log export policy
    1. set hostname [Linux server IP or FQDN if your DNS is updated]
    2. set user ucsloguser
    3. set passwd [press enter, then enter the password of the ucsloguser]
    4. set admin-state yes
    5. set proto scp
    6. set path /home/ucsloguser/
    7. commit-buffer
  4. That’s it. Now log into your Linux server and see if log file .tgz bundles are showing up in your home directory.
Configuring log export policy in UCS
Logs arriving in external server!

Tips:

Use the command sudo ls -thl to sort by newest to oldest files with a human readable size.

Use the command sudo df -h to show the space consumed. 

Use an SCP utility like WinSCP to retrieve files from your log server so you can send them to Cisco TAC now. 


You might be interested in …

vCloud Director Failing to Customize Windows 2008 Server

Technology, Uncategorized, VMware

This evening I got an urgent email from a colleague that just got the rug pulled out from under him. His customer decided that the two web servers on the front end of a SharePoint farm he was building couldn’t be 2008 R2 because they require 32bit servers. I stayed far away from the “why” – […]

Read More

Repairing AirPort coverage

Apple, Technology, Uncategorized

I hope this helps someone but I’ve been chasing a wifi problem in my house for a few days and finally got to fixing it. Equipment: 4th Gen Airport Extreme (802.11 a/b/g/n; 2.4GHz & 5GHz) Symptoms: poor range, slow speeds outside of the room the airport was located. Configuration: I have it configured auto everything […]

Read More

STARTTLS and OS X Mail

Apple, Technology, Uncategorized

A change recently in OS X’s Mail application has caused an unusual problem of not sending mail on the regular SMTP or IMAP ports. I’m not sure if its El Capitan that initiated this change because I do not send mail often from my desktop mail client. After a message stuck in my outbox, I […]

Read More
%d bloggers like this: