Skip to content

Recent Articles

6
May

Cloning and deploying ESXi

I’ve got a pile of MicroSD cards, a few racks worth of blades in different locations, and a hankering to deploy VMware in a stateful manner by simply plugging these SD cards in and booting up the blades or deploy blades with SD cards and perform remote installs with as little effort as possible.

 

First challenge is to reduce the amount of effort in prepping or installing the base hypervisor on the SD cards.

I can clone them quickly using an SD card reader from a base image – however ESXi has to be prepared for cloning (Windows engineers will recognize this as a Sysprep).

  1. Install ESXi on a blade as you normally would.
  2. Boot ESXi and make sure it boots up without issues. You could join it to vCenter and push any applicable patches or VIBs your blade will need (Nexus 1k VEM, EMC PowerPath, etc).
  3. Log into the console and select the last option “Reset System Configuration” , press F-11 to confirm, press Enter to restart the host.
  4. Once ESXi reboots, shutdown the blade and grab the SD Card, that’ll be your master image.
  5. You can now clone this SD card or better yet, create a master template image file so you can push it multiple times. Each copy will generate its own UUID and Service Console MAC address so there will be no conflicts.

One thing to note – your password is now BLANK, so when you deploy – make sure you get a strong password on it. I’ll be using host profiles so that’ll be taken care of during the prep for production.

 

My second option is to create a custom install ISO. I’m looking to have this ISO boot to a menu or prompt that will allow the installing engineer enter in the host name, IP address, and whatnot – then install and customize the ESXi install automatically and reboot.

Still investigating that.

21
Feb

Trust is Everything.

We spend so much time working toward a goal, delivering on an expectation, that it becomes business as usual. People trust me to do my job, I trust others to do theirs. Together we make great teams, great divisions, and a great company for our customers.

In the last two weeks, two major companies have lost my trust. One, Anthem, an insurance company, that had 80M customer medical records stolen and the other Lenovo, a computer manufacture, that installed software on their consumer laptops and desktops that intercepted TLS/SSL encrypted traffic using a self signed encryption certificate and embedded it into the operating system.

I trusted these companies like I know my company’s customers trust us. How can we prevent becoming the next untrustworthy company?

For Anthem, the problem was not following the basics of data security. Encrypt your data at rest, in flight, and protect your keys at all cost. Restrict data access to only those who need it – and ONLY the data they need. What a logistics nightmare to coordinate that among the entire company’s applications – but not as bad as the nightmare they’re living with now.

Regardless if you store your data in a colocation with insane physical and electronic boarder security or your own onsite datacenter with James Bond proof security – thieves don’t have to get out of their underwear to make off with a billion dollars with of data.Screenshot 2015-02-21 20.25.54

Lenovo had a simple task of maintaining their tradition of building computers that people want. Someone made the decision to install some extra software (presumably for profit) on their consumer computers that gathers “We thought [Superfish] would enhance the shopping experience…“. I call bullshit – the software was harvesting their customers data by decrypting encrypted browser traffic using a “Man in the Middle” attack. Lenovo decided to make an extra dollar and deceive their customers into thinking the lock on their browser actually meant they were secure. It took five months for the word to get out on the Lenovo malware, but in the week it hit the media – I’ve read about it everywhere. I’m watching LNVGY to see what happens when the stock holders finally figure out what this means.

It means companies large and small that buy one or thousands of their products in bulk may begin to question the integrity of Lenovo. If they installed privacy busting malware into their consumer goods – what kind of nefarious tricks were in the latest ThinkPad or X laptop in my business? Is my CIO’s bank account going to be hacked, company secrets leaked to a Lenovo partner in China, or my laptop used as a portal for Chinese hackers to run rampant in my network? A secret decryption chip and hook into the network hardware to leak secrets back to home base? Sure, it’s far fetched… or is it? Stranger things have been dreamed up.

Anyone can prevent these – stand up and say something. Call out bad ideas for what they are. Identify security risks when you see them. These are what makes great people, great. They take on the challenge  instead of letting the big sleeping dragons sleep. Eventually they’ll wake up, on their own or with the help of someone, and lay waste to your kingdom.

5
Oct

Sophos UTM and Xbox Live NAT issues

In a previous post about the Sophos UTM, I’ve received some comments about Xbox Live.

I’ve recently rebuilt my Sophos UTM and found that my backup configuration files were encrypted with a password I couldn’t locate… my bad. So I’m rebuilding all of my firewall rules again.

For the Xbox 360 Live and Kinect, there are some quirks that I hope these steps help you overcome.

  1. Setup a DHCP reservation for your Xbox so that your UTM will create a network definition and the IP will remain the same.
  2. Create new service definitions for each of the ports listed in the base article here
    1. Port 88 (UDP)
    2. Port 3074 (UDP and TCP)
    3. Port 53 (UDP and TCP)
    4. Port 80 (TCP)
    5. Port 1863 (TCP and UDP)
  3. In my UTM, I’ve named them Xbox Live UDP 88, Xbox Live TCP & UDP 3074, etc. This helps keep your definitions clean and searchable.
  4. Create a definition group and call it Xbox Live and Kinect (or whatever helps you keep organized).
  5. Create a new Firewall rule
    1. Source Xbox 360 -> Services Xbox Live and Kinect -> Destination Any
  6. Create a new NAT rule
    1. Rule Type: DNAT (Destination)
    2. Matching Condition
      1. For Traffic from: Any
      2. Using service: Xbox Live TCP and UDP 3074
      3. Going to: External (Address)
    3. Action
      1. Change the destination to: Xbox 360
    4. Automatic Firewall rule: Checked!
    5. Save

Save and don’t forget to turn on your Xbox 360 firewall rule before testing.

Here are some screen captures that may help show you what these all mean.

Sophos UTM Xbox 360 NAT Rule XBox 360 Firewall Rule Xbox Service Definitions

6
Sep

Repairing AirPort coverage

I hope this helps someone but I’ve been chasing a wifi problem in my house for a few days and finally got to fixing it.

Equipment: 4th Gen Airport Extreme (802.11 a/b/g/n; 2.4GHz & 5GHz)

Symptoms: poor range, slow speeds outside of the room the airport was located.

Configuration: I have it configured auto everything across the board, no unique SSID for the 5Ghz network, and nothing customized other than DNS – thanks to DDOS on Charter’s DNS servers I swapped in Google and Level 3’s DNS servers.

I loaded a free app on my MacBook called WiFi Explorer that displays wifi signals, noise, and occupied channels but the one built into OS X works equally well. I noticed that the 2.4Ghz network was dropping off for 30-45 seconds every minute even though there are devices on my network that require 2.4GHz.

5GHz was solid but as expected it’s range was poor and signal strength at the distances I needed it to work in my house were very low.

Started with power cycling the router. No difference. Soft then Hard reset, no change. 2.4GHz just wouldn’t stay on.

Launched Airport Utility and reset the AirPort to the factory setting, no difference. So now I’m thinking hardware issue. Of course, this is not under warranty anymore.

The last trick I had available was to roll back the firmware. In the airport utility click on the AirPort Extreme to display the serial number and firmware version. Option click on the version and pick 7.6.3 from the list. The utility downloads and installs the firmware. Really couldn’t be more simple.

Bam all is good after the reboot. Both radios functioning at expected levels. So is it a firmware issue or a glitch? So I decided to upgrade the firmware to the latest. The latest does have some good fixes.

After the reboot everything has been solid again. WiFI Explorer shows 5GHz and 2.4GHz on solid and never dropping off. I’m going to chalk it up to a glitch in the firmware that was cleared by reloading the firmware. The only way to do that for these is to rollback then upgrade again. Luckily though in true apple fashion – the utility does all the hard work and maintains your configurations.

Also the iOS utility offers the same functionality, so it’s easy to repair these.

21
Apr

Master Images – hidden CPU time bombs

Wrapping up master images has become something virtualization engineers of all product disciplines have to become familiar with. A bad master image can be deployed dozens or hundreds of times – only to find out a simple tweak could have saved you thousands in necessary hardware costs.

Here’s a new hidden gem I found and I hope to add to this list as more arrive.

 

Installing or updating Dot Net

Almost all Microsoft patching includes some form of a dot net update. When this product is updated, it likes to recompile a lot of code to help speed up launching dot net applications – pre-compiling actually does help user perception of application launch speeds.

Typically you run windows update on a server or workstation and dot net installs its updates and queues items in a work list that dot net executes later. This typically happens later in the day or evening and almost always pegs your CPU for a minute to 1/4 of an hour while is pre-compiles code.

Microsoft is pretty clear about this process in this MSDN Blog post.

The problem is, when you’re patching master images – you don’t want to leave the queued items for each deployed VM to have to execute. Deploy a dozen servers, and now you have a dozen servers with queued dot net jobs waiting to flog your CPUs.

For Windows 2008 R2 and Windows 2012 servers, you can easily kick off these queued items before you wrap up your images for templates by following these simple steps:

  1. Run a comand prompt or powershell prompt with administrative privlegdges.
  2. Run this command:
    c:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe executeQueuedItems
  3. Wait for the compiling to finish
  4. Exit

The blog post above contains other paths for other versions of Windows, but hopefully that helps others.

%d bloggers like this: