Skip to content

Recent Articles

7
Sep

External Log Collection for UCS Fabric Interconnects

I’ve been troubleshooting some pretty annoying bugs in our Cisco UCS environments. Most are easily solved by collecting some techsupport files, opening a TAC case and working through the glitch or config issue. However one has me really stumped and frustrated. 

When we collect these techsupport files, more specifically techsupport files directed at a specific chassis – an IO Module will randomly disconnect, reboot, or reconfigure – dropping half or both connections to the fabric (we have one uplink per IOM, two per chassis).

As we continue to troubleshoot with Cisco TAC, we mostly find out later that the tech support file we generate after the issue doesn’t contain the information they need, the logs have rolled over… or over written due to activity in the domain. We gather syslogs religiously but the information necessary isn’t sent out via syslog when it happens. Feature request?

After pressing one of the TAC engineers on my fifth case this year on this issue, he clued me in on a feature for exporting logs to an external server. Click away if you know this, I certainly didn’t.

Here’s a quick and dirty on how to do it with a generic Ubuntu server.

I’m going to write this soup to nuts for someone who’s a novice and never setup a Linux server. By no means will this be hardened and secured for public visibility – just a place for your FIs to dump their logs. Chime in with a comment if you have improvements or suggestions.

  1. Deploy Ubuntu Server on a VM with a few gigs of space
  2. Update your server once its online with these two commands:
    1. sudo apt-get update
    2. sudo apt-get upgrade
  3. Reboot your server once upgrades are complete
  4. Create a dedicated user
    • sudo adduser ucsloguser
    • provide password info but nothing else matters
  5. Let’s assume you’ll be dumping logs to the user’s home directory, so the path will be /home/ucsloguser
    • For a permanent home, you could add a second disk and mount it under a dedicated path or get your neckbeard on and use LVM to create a logical volume you can add disks and grow later. For now we’ll keep it simple, stupid.
  6. Using your favorite SSH client (Terminal, PuTTY, XTerm, etc) connect to your new server using the ucsloguser account to verify you can SSH to your Ubuntu server.
  7. Ok, Linux server is ready to go.

Configuring the Fabric Interconnects to dump logs onto your Ubuntu server

  1. SSH to your FI – doesn’t matter which, both will respect the monitoring change. 
  2. Run the follow commands:
    1. scope monitoring
    2. scope sysdebug
    3. scope log-export-policy
  3. Now we set the log export policy
    1. set hostname [Linux server IP or FQDN if your DNS is updated]
    2. set user ucsloguser
    3. set passwd [press enter, then enter the password of the ucsloguser]
    4. set admin-state yes
    5. set proto scp
    6. set path /home/ucsloguser/
    7. commit-buffer
  4. That’s it. Now log into your Linux server and see if log file .tgz bundles are showing up in your home directory.
Configuring log export policy in UCS
Logs arriving in external server!

Tips:

Use the command sudo ls -thl to sort by newest to oldest files with a human readable size.

Use the command sudo df -h to show the space consumed. 

Use an SCP utility like WinSCP to retrieve files from your log server so you can send them to Cisco TAC now. 


9
Dec

Reduction in the legal spam.

The majority of the legal spam that I’ve been recieving has been from major marketing mailing list managers. Unsubscribing from each company’s email campaign is a futile game of whack-a-mole. So I directed my efforts on the source of the email. Here are the steps I’ve used to get a perma-block on my email address for now and future email campaigns. I’ll be glad to add more if you know of any, hit me up in the comments.

 

Constant Contact:

Instructions:

Send an email to: Support@constantcontact.com

Subject: Permanent block

Body of the message: Please prevent any of your current and future customers from sending me emails through your service.

Submit and wait for a confirmation. I had a response in about six hours with a positive acknowledgement.

 

Mailchimp:                        

Visit: https://mailchimp.com/contact/abuse/

Instructions: fill in your full name and email address. For the required section labeled headers, simply enter “no headers”

Reason for report: Please prevent any of your current and future customers from sending me emails through your service.

Submit and wait for a confirmation. Mailchimp support responded in about an hour with a positive acknowledgment.

24
Aug

Hewlett Packard HyperConverged HC380 Fun

I’m 1600 miles away implementing a multi node cluster of these HC380s and have run into a few bugs. One is an absolute deal breaker and needed to share this in hopes it helps others in the same process.

During implementation, you are asked to choose a username and password for the VirtualStore appliances that make up the storage back end of this solution. There are a few characters that are not accepted like colons, semicolons, etc.

One of the characters that they did not call out is the ampersand (&) – DO NOT use this character when deploying the HC380 environment. Your deployment will begin and then fail during the VSA deployment and configuration. The end result is a smoldering pile of HP software that requires a manual “reset” process on each and every node that will take about 45 minutes to run on each.

Pre-Implementation Tips:

  1. Verify you’re running the latest version – don’t assume the fresh hardware arriving has the latest version.
  2. Download all of the images available before arriving onsite… just in case you need them. The management VM is 20GB large and installed on each node, better to have that before you need it.
  3. Bring some longer Ethernet cable if you don’t want to be standing behind the HPC node balancing your laptop in one hand and deploying or resetting the environment with the other.

Don’t disregard implementation support costs. If you’re not ready to lean hard on your HP reps – purchase implementation support. HP production support may ask you to pay up for help implementing their solution even if the problem is 100% their software.

7
Jan

VMware Tools Downgrade

I ran into a situation where I needed to test upgrading VMware tools using an alternative method other than directly through vCenter or auto update. To do this testing I had to roll back the most recent version of VMware Tools on a VM to an older one. Uninstall latest, reboot, install older, reboot. No sweat, right?

Well this wouldn’t be much of a blog entry if that’s all there was…every time I rebooted after installing the older version – vCenter continued to report the tools version was up to date and everything was A-Ok. What was strange is even running the command line to verify the versions returned the latest version – and I know I used the older tools installer.

C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe -v

After uninstalling the newest version, I used the VMware KBase article for manually uninstalling VMware Tools to verify everything was gone – and it was – until I hit the last step: Delete the %ProgramFiles%\VMware\VMware Tools folder.

I found a single text file called Manifest.txt which contained the current versions of everything that was installed with the latest VMware Tools. I deleted this file and ran the old VMware Tools installer and successfully reported back an “Out Of Date” Tools installation.

Tip: You can grab any version of VMware Tools from VMware’s Packages site here: https://packages.vmware.com/tools/esx/index.html

21
Oct

STARTTLS and OS X Mail

Screen Shot 2015-10-21 at 11.29.50 AMA change recently in OS X’s Mail application has caused an unusual problem of not sending mail on the regular SMTP or IMAP ports. I’m not sure if its El Capitan that initiated this change because I do not send mail often from my desktop mail client.

After a message stuck in my outbox, I fired up my firewall live log display and could see my computer hitting TCP port 587 a few times. This port is blocked on my firewall because I’ve never needed it open.

Normal humans don’t run a firewall at home that is this locked down – normally any outbound traffic is open – but this works for me.

I did some research and apparently TCP 587 is a known email port – for SMTP using STARTTLS, which I didn’t think Mail used (or it didn’t until recently) to send mail. Everything in Mail’s preferences mentions TCP 993 (IMAPS or IMAP using SSL encryption).

FastMail has a very good article on what this port is used for and why its being used. Essentially it’s a TLS encrypted SMTP connection that offers a better way of validating the destination – which could possibly coincide with Apple’s use of a “token” to authenticate iCloud users. So after opening up TCP 587, Mail was able to send my email message on its merry way.

 

 

%d bloggers like this: