Skip to content

Recent Articles


Hewlett Packard HyperConverged HC380 Fun

I’m 1600 miles away implementing a multi node cluster of these HC380s and have run into a few bugs. One is an absolute deal breaker and needed to share this in hopes it helps others in the same process.

During implementation, you are asked to choose a username and password for the VirtualStore appliances that make up the storage back end of this solution. There are a few characters that are not accepted like colons, semicolons, etc.

One of the characters that they did not call out is the ampersand (&) – DO NOT use this character when deploying the HC380 environment. Your deployment will begin and then fail during the VSA deployment and configuration. The end result is a smoldering pile of HP software that requires a manual “reset” process on each and every node that will take about 45 minutes to run on each.

Pre-Implementation Tips:

  1. Verify you’re running the latest version – don’t assume the fresh hardware arriving has the latest version.
  2. Download all of the images available before arriving onsite… just in case you need them. The management VM is 20GB large and installed on each node, better to have that before you need it.
  3. Bring some longer Ethernet cable if you don’t want to be standing behind the HPC node balancing your laptop in one hand and deploying or resetting the environment with the other.

Don’t disregard implementation support costs. If you’re not ready to lean hard on your HP reps – purchase implementation support. HP production support may ask you to pay up for help implementing their solution even if the problem is 100% their software.


VMware Tools Downgrade

I ran into a situation where I needed to test upgrading VMware tools using an alternative method other than directly through vCenter or auto update. To do this testing I had to roll back the most recent version of VMware Tools on a VM to an older one. Uninstall latest, reboot, install older, reboot. No sweat, right?

Well this wouldn’t be much of a blog entry if that’s all there was…every time I rebooted after installing the older version – vCenter continued to report the tools version was up to date and everything was A-Ok. What was strange is even running the command line to verify the versions returned the latest version – and I know I used the older tools installer.

C:\Program Files\VMware\VMware Tools\VMwareToolboxCmd.exe -v

After uninstalling the newest version, I used the VMware KBase article for manually uninstalling VMware Tools to verify everything was gone – and it was – until I hit the last step: Delete the %ProgramFiles%\VMware\VMware Tools folder.

I found a single text file called Manifest.txt which contained the current versions of everything that was installed with the latest VMware Tools. I deleted this file and ran the old VMware Tools installer and successfully reported back an “Out Of Date” Tools installation.

Tip: You can grab any version of VMware Tools from VMware’s Packages site here:



Screen Shot 2015-10-21 at 11.29.50 AMA change recently in OS X’s Mail application has caused an unusual problem of not sending mail on the regular SMTP or IMAP ports. I’m not sure if its El Capitan that initiated this change because I do not send mail often from my desktop mail client.

After a message stuck in my outbox, I fired up my firewall live log display and could see my computer hitting TCP port 587 a few times. This port is blocked on my firewall because I’ve never needed it open.

Normal humans don’t run a firewall at home that is this locked down – normally any outbound traffic is open – but this works for me.

I did some research and apparently TCP 587 is a known email port – for SMTP using STARTTLS, which I didn’t think Mail used (or it didn’t until recently) to send mail. Everything in Mail’s preferences mentions TCP 993 (IMAPS or IMAP using SSL encryption).

FastMail has a very good article on what this port is used for and why its being used. Essentially it’s a TLS encrypted SMTP connection that offers a better way of validating the destination – which could possibly coincide with Apple’s use of a “token” to authenticate iCloud users. So after opening up TCP 587, Mail was able to send my email message on its merry way.




Cloning and deploying ESXi

I’ve got a pile of MicroSD cards, a few racks worth of blades in different locations, and a hankering to deploy VMware in a stateful manner by simply plugging these SD cards in and booting up the blades or deploy blades with SD cards and perform remote installs with as little effort as possible.


First challenge is to reduce the amount of effort in prepping or installing the base hypervisor on the SD cards.

I can clone them quickly using an SD card reader from a base image – however ESXi has to be prepared for cloning (Windows engineers will recognize this as a Sysprep).

  1. Install ESXi on a blade as you normally would.
  2. Boot ESXi and make sure it boots up without issues. You could join it to vCenter and push any applicable patches or VIBs your blade will need (Nexus 1k VEM, EMC PowerPath, etc).
  3. Log into the console and select the last option “Reset System Configuration” , press F-11 to confirm, press Enter to restart the host.
  4. Once ESXi reboots, shutdown the blade and grab the SD Card, that’ll be your master image.
  5. You can now clone this SD card or better yet, create a master template image file so you can push it multiple times. Each copy will generate its own UUID and Service Console MAC address so there will be no conflicts.

One thing to note – your password is now BLANK, so when you deploy – make sure you get a strong password on it. I’ll be using host profiles so that’ll be taken care of during the prep for production.


My second option is to create a custom install ISO. I’m looking to have this ISO boot to a menu or prompt that will allow the installing engineer enter in the host name, IP address, and whatnot – then install and customize the ESXi install automatically and reboot.

Still investigating that.


Trust is Everything.

We spend so much time working toward a goal, delivering on an expectation, that it becomes business as usual. People trust me to do my job, I trust others to do theirs. Together we make great teams, great divisions, and a great company for our customers.

In the last two weeks, two major companies have lost my trust. One, Anthem, an insurance company, that had 80M customer medical records stolen and the other Lenovo, a computer manufacture, that installed software on their consumer laptops and desktops that intercepted TLS/SSL encrypted traffic using a self signed encryption certificate and embedded it into the operating system.

I trusted these companies like I know my company’s customers trust us. How can we prevent becoming the next untrustworthy company?

For Anthem, the problem was not following the basics of data security. Encrypt your data at rest, in flight, and protect your keys at all cost. Restrict data access to only those who need it – and ONLY the data they need. What a logistics nightmare to coordinate that among the entire company’s applications – but not as bad as the nightmare they’re living with now.

Regardless if you store your data in a colocation with insane physical and electronic boarder security or your own onsite datacenter with James Bond proof security – thieves don’t have to get out of their underwear to make off with a billion dollars with of data.Screenshot 2015-02-21 20.25.54

Lenovo had a simple task of maintaining their tradition of building computers that people want. Someone made the decision to install some extra software (presumably for profit) on their consumer computers that gathers “We thought [Superfish] would enhance the shopping experience…“. I call bullshit – the software was harvesting their customers data by decrypting encrypted browser traffic using a “Man in the Middle” attack. Lenovo decided to make an extra dollar and deceive their customers into thinking the lock on their browser actually meant they were secure. It took five months for the word to get out on the Lenovo malware, but in the week it hit the media – I’ve read about it everywhere. I’m watching LNVGY to see what happens when the stock holders finally figure out what this means.

It means companies large and small that buy one or thousands of their products in bulk may begin to question the integrity of Lenovo. If they installed privacy busting malware into their consumer goods – what kind of nefarious tricks were in the latest ThinkPad or X laptop in my business? Is my CIO’s bank account going to be hacked, company secrets leaked to a Lenovo partner in China, or my laptop used as a portal for Chinese hackers to run rampant in my network? A secret decryption chip and hook into the network hardware to leak secrets back to home base? Sure, it’s far fetched… or is it? Stranger things have been dreamed up.

Anyone can prevent these – stand up and say something. Call out bad ideas for what they are. Identify security risks when you see them. These are what makes great people, great. They take on the challenge  instead of letting the big sleeping dragons sleep. Eventually they’ll wake up, on their own or with the help of someone, and lay waste to your kingdom.

%d bloggers like this: