Skip to content

Posts from the ‘VMware’ Category

24
Aug

Hewlett Packard HyperConverged HC380 Fun

I’m 1600 miles away implementing a multi node cluster of these HC380s and have run into a few bugs. One is an absolute deal breaker and needed to share this in hopes it helps others in the same process.

During implementation, you are asked to choose a username and password for the VirtualStore appliances that make up the storage back end of this solution. There are a few characters that are not accepted like colons, semicolons, etc.

One of the characters that they did not call out is the ampersand (&) – DO NOT use this character when deploying the HC380 environment. Your deployment will begin and then fail during the VSA deployment and configuration. The end result is a smoldering pile of HP software that requires a manual “reset” process on each and every node that will take about 45 minutes to run on each.

Pre-Implementation Tips:

  1. Verify you’re running the latest version – don’t assume the fresh hardware arriving has the latest version.
  2. Download all of the images available before arriving onsite… just in case you need them. The management VM is 20GB large and installed on each node, better to have that before you need it.
  3. Bring some longer Ethernet cable if you don’t want to be standing behind the HPC node balancing your laptop in one hand and deploying or resetting the environment with the other.

Don’t disregard implementation support costs. If you’re not ready to lean hard on your HP reps – purchase implementation support. HP production support may ask you to pay up for help implementing their solution even if the problem is 100% their software.

6
May

Cloning and deploying ESXi

I’ve got a pile of MicroSD cards, a few racks worth of blades in different locations, and a hankering to deploy VMware in a stateful manner by simply plugging these SD cards in and booting up the blades or deploy blades with SD cards and perform remote installs with as little effort as possible.

 

First challenge is to reduce the amount of effort in prepping or installing the base hypervisor on the SD cards.

I can clone them quickly using an SD card reader from a base image – however ESXi has to be prepared for cloning (Windows engineers will recognize this as a Sysprep).

  1. Install ESXi on a blade as you normally would.
  2. Boot ESXi and make sure it boots up without issues. You could join it to vCenter and push any applicable patches or VIBs your blade will need (Nexus 1k VEM, EMC PowerPath, etc).
  3. Log into the console and select the last option “Reset System Configuration” , press F-11 to confirm, press Enter to restart the host.
  4. Once ESXi reboots, shutdown the blade and grab the SD Card, that’ll be your master image.
  5. You can now clone this SD card or better yet, create a master template image file so you can push it multiple times. Each copy will generate its own UUID and Service Console MAC address so there will be no conflicts.

One thing to note – your password is now BLANK, so when you deploy – make sure you get a strong password on it. I’ll be using host profiles so that’ll be taken care of during the prep for production.

 

My second option is to create a custom install ISO. I’m looking to have this ISO boot to a menu or prompt that will allow the installing engineer enter in the host name, IP address, and whatnot – then install and customize the ESXi install automatically and reboot.

Still investigating that.

21
Apr

Master Images – hidden CPU time bombs

Wrapping up master images has become something virtualization engineers of all product disciplines have to become familiar with. A bad master image can be deployed dozens or hundreds of times – only to find out a simple tweak could have saved you thousands in necessary hardware costs.

Here’s a new hidden gem I found and I hope to add to this list as more arrive.

 

Installing or updating Dot Net

Almost all Microsoft patching includes some form of a dot net update. When this product is updated, it likes to recompile a lot of code to help speed up launching dot net applications – pre-compiling actually does help user perception of application launch speeds.

Typically you run windows update on a server or workstation and dot net installs its updates and queues items in a work list that dot net executes later. This typically happens later in the day or evening and almost always pegs your CPU for a minute to 1/4 of an hour while is pre-compiles code.

Microsoft is pretty clear about this process in this MSDN Blog post.

The problem is, when you’re patching master images – you don’t want to leave the queued items for each deployed VM to have to execute. Deploy a dozen servers, and now you have a dozen servers with queued dot net jobs waiting to flog your CPUs.

For Windows 2008 R2 and Windows 2012 servers, you can easily kick off these queued items before you wrap up your images for templates by following these simple steps:

  1. Run a comand prompt or powershell prompt with administrative privlegdges.
  2. Run this command:
    c:\Windows\Microsoft.NET\Framework64\v4.0.30319\ngen.exe executeQueuedItems
  3. Wait for the compiling to finish
  4. Exit

The blog post above contains other paths for other versions of Windows, but hopefully that helps others.

25
Jan

Sophos Home Firewall with Captive Portal Guest Network using Airport Extreme APs

KISS, keep it simple stupid. An old saying I try to keep in mind when working on a solution. My home network hasn’t maintained this zen philosophy.

My most recent setup has been driven by a recent discovery that I can keep my Apple Airport’s built in Guest network, as long as I can connect the Airport’s to something that can pickup VLAN 1003, the VLAN Apple decided to isolate the users on the guest network so they wouldn’t be able to see file or print shares on the main network.

So I decided to build a firewall but the only hardware I had was a bit overkill and didn’t want to dedicate the whole thing to this one task. I started with a used Dell OptiPlex 745 micro tower. This little workhorse has a dual core Intel CPU chugging along at 1.8Ghz, it’s got the virtualization instructions and 64bit capable. The Dell came with a tiny hard drive and a few gig of ram, but I grabbed a spare SATA drive and purchased four sticks of 2GB RAM from an ebay seller. I also picked up an Intel gigabit NIC to compliment the onboard gigabit NIC.sophos

The Dell runs VMware ESXi 5.5 with a free license. I’ve configured the VMware host with four networks.

  1. External network, connected to the vSwitch that uplinks to the Intel Gigabit NIC
  2. Internal network, connected to the vSwitch that uplinks to the onboard Gigabit NIC
  3. Guest network, connected to the same vSwitch that uplinks to the onboard Gigabit NIC, but is on VLAN 1003
  4. DMZ network, connected to a vSwitch that doesn’t connect to anything physical, I’ll use it for test/dev VMs I create on the host.

 

I signed up for a free account with Sophos, here so I could download their UTM software. They also provide a generous license for home users that will protect up to 50 IP addresses. It’s almost nearly fully functional and includes managed end point protection for up to ten Windows computers. If I really wanted to go nuts, I could buy some Astaro (now Sophos) wireless access points and have them fully managed from this server but I think the Airport Extremes will work just fine.

Sophos offers a prebuilt appliance you can download and just run out of the box, but I think they let their junior assistant’s intern build it. It has poor performing configuration choices… and really you’re just better off building it from scratch. The .ISO is pretty universal – it’ll handle installing on bare metal, virtual, or being installed on Astaro/Sophos hardware appliances.

So going on information from the forums and documentation:

  • OS is based on SLES 11 64bit
  • 1 CPU, 2 Core
  • 4GB of RAM
  • 60GB of Disk, using a SAS controller
  • 4 x VMXNET3 vNICs – leave all but the internal vNIC disconnected (Intel E1000 vNICs work, too but will consume more resources than the VMXNET3 paravirtual vNICs)
  • USB controller (for USB backups)
  • Delete the floppy disk drive, serial, and parallel ports

Boot the VM from the .ISO, and go through a few basic questions. It’ll handle the disk partitioning, volume formatting, and even installs VMware Tools for you because it identifies itself as a VMware VM and goes that extra step. Once you do the install, you’ll have to browse to the server’s IP on port 4444. During this final setup you’ll be prompted to upload the license file Sophos sends you and answer a few questions about the preliminary firewall rules.

I would recommend checking all of the basic firewall rules – Email, DNS, Web, etc. This will make your start a little easier. Leave the rest of the monitoring, filtering, and all that disabled, it’s only going to get in the way.

Now comes the tricky part – matching the physical nics with the virtual networks and vNICs on the Sophos UTM virtual machine. Under Interfaces & Routing, click on Interfaces. You’ll see your Internal interface [UP]. Add a second interface, let’s start with the Guest network.

  • Name: Guest
  • Type: Ethernet Static
  • Hardware: Pick one
  • IPv4 Address: choose an IP for this interface and a net mask to set the size. This IP will be the gateway for everything on this network.
  • No other settings need to be changed or enabled

Once you save the interface, you’ll see that it’s [down]. Edit your VM, and enable the vNIC connected to the Guest network. If the interface doesn’t go up, edit it and pick a different eth in the hardware menu. Once you hit it, follow through on the other networks until you have all four up and running.

Now, Sophos UTM comes out of the box ready to support an internal LAN and external WAN. Additional network will require some more configuration, read on.

Network Services

DHCP and DNS needs to be configured for at least the Guest network. You can set it up for DMZ too, if you want. Navigate to Network Services, DNS. In the Global tab, click the folder icon and choose your Guest (Network) and DMZ (Network).

Then switch over to DHCP and click New DHCP Server, and add a new one for the Guest network. The gateway and DNS IP addresses will be the interface IPs you setup earlier, they end in .1.

While you’re here – switch to the NTP network service and allow Guest and DMZ to access the NTP server you’ll configure later.

Masquerading Rules

The one setting that got me stuck was the NAT settings. Switch to Network Protection – NAT. You’ll notice a rule already set for Internal (Network) -> External.

Clone this rule and set it up for DMZ and Guest so they can also connect to the outside world.

Airport Changes

Now that I had a functional firewall, I needed to get the Airport Extreme devices configured. I launched the Airport utility from my Mac (you can do the same from any iOS device) and configured the Airports for bridge mode and turned off the password on my Guest network – the UTM will handle authentication now. Everything else works fine – one of my AEs is a print server with two printers, and a Time Machine backup target with a 2TB USB disk attached… all work fine.

Pro tip: once the AE is in “bridged mode” , you lose the ability to use the WAN port for anything. Just use the LAN ports for connectivity back to the Sophos UTM or other Airport Extreme APs the WAN port becomes a LAN port in bridged mode (I had a different experience in previous firmwares, so something must have changed).  If you use a switch – make sure it can forward VLAN tagged traffic, most SOHO switches will not. If you have multiple Airport Extreme APs, daisy chain the AE APs to ensure VLAN 1003 packets get delivered back to the Sophos UTM and hang the switch off of the other ports to provide more access ports to your internal devices.

Captive Portal

Okay, this was the icing on the cake for this deployment. Now that I have my guest network isolated on a dedicated interface and VLAN, I can really do some neat stuff with Sophos UTM.

  1. In the Wireless Protection section visit Global Settings.
  2. Enable the Wireless Protection, then add the Guest interface to the allowed interfaces.
  3. Click Apply.
  4. Visit the Wireless Networks, Access Points, and clean out any auto-configured networks or access points.
  5. Then click on Hostspots. Enable the Hotspot feature.
  6. Switch to the Hotspots tab.
  7. Add a new one, call it Guest Portal or something easy to identify.
  8. Add the Guest interface to this hotspot and configure the rest of the options for your unique needs.

Tips: The Password of the Day will not be an easy password. It’ll be something like ogaleseh35 (that was yesterday’s password at my house). This password is good for the day.

You can opt for a Voucher system… which is really powerful. You can limit by time and or consumed bandwidth. I can see myself forcing my kids to use a guest network and only handing out vouchers when stuff is done around the house. You can delegate access to other people so they can log into a user portal on the Sophos UTM and print or PDF additional vouchers.

Additional Features

A few other features I turned on:

  1. Global IPS, I turned off the IPS for now on my internal network until I can isolate my dumb media devices (smart TVs, etc) and exclude them
  2. Endpoint Protection (free AV software) – sophos managed end point for Windows (OS X is free, but still not managed yet!)
  3. Uplink Monitoring – I won’t get emailed when it goes down (duh!) but I’ll get the down and up alert when the UTM can send email again.
  4. User Portal – for people to get VPN setup and vouchers for guest wifi
  5. NTP – accurate time isn’t an option any more.

Additional Firewall Rules

Sophos UTM is a true firewall. Nothing gets in or out with an explicit rule. This can be very challenging at home when you have a myriad of different devices. Most devices like Apple TV, Vizio TV apps, and the like catch a ride on port 80 or 443, so if you have web enabled, you’re good to go.

Additional rules are needed to allow stuff like MineCraft, Apple Push Notifications, AT&T Microcell, Mumble, Xbox Live, etc… be prepared to spend some time digging around knowledge bases or Googling to find the appropriate ports. Luckily, it’s real easy to build rules.

Tip: Use groups for rules that you may need to add different ports or services to a single rule. I made one called Games. Now when I run into another game (or service like Steam) that needs another port allowed out – I can create the Network Service, and just add it to the existing Games group.

I have a rule for Apple. Holy crap they have a lot going on. Most you don’t need to allow out or are handled by existing rules.

  • iCloud DAV services, iChat, FaceTime, Game Center, yuck… Lots of UDP port ranges, but again these are only allowing these apps OUT – not random internet person reach into your network on these same ports. On the topic of Apple, when watching the firewall logs I forgot they have a Class A IP range… so it may be easiest just to create a Network rule and call it Apple’s Network… then you can use that in each rule for apple.
  • Xbox Live has a list of UDP and TCP ports that need out, and a NAT rule to allow UDP/TCP 3074 back in
  • AT&T’s Microcell needs to have https, ntp, and some IPSec traffic allowed out to femtocell.wireless.att.com and after watching it fail – another IP, 12.230.209.70,  in AT&T’s IP range that wasn’t documented. Once that last IP was added to the rule, bam – solid 5 bars.

 

Final Tips

One last thing that can help your UTM perform better is to disable logging and reporting if you don’t need it. I turned off all the reports but after reviewing the rules – the logs can be retained for no less than 1 day. So I chose to disable logging. This caused a problem with troubleshooting the firewall – I couldn’t view the live log, it wasn’t being generated!

So I turn it on when I want to troubleshoot something, but I’ll leave it disabled. I’m not sure how the 5th amendment fits in to a firewall keeping logs of your internet traffic for a year… but if you don’t have to save all the data about your network activity – why flog your UTM’s storage when you don’t have to.

On the flip side, if somethings not working – almost every feature has a Live Log view so you can watch the blocked packets fly by. This is exceptionally helpful.

13
Apr

vCloud Director Failing to Customize Windows 2008 Server

This evening I got an urgent email from a colleague that just got the rug pulled out from under him. His customer decided that the two web servers on the front end of a SharePoint farm he was building couldn’t be 2008 R2 because they require 32bit servers. I stayed far away from the “why” – and just wanted to help deliver the “how” as easy and fast as possible.

The problem he ran into was when he deployed two Windows 2008 32bit Enterprise Edition servers… they never customized from the template. Still had the default password, network settings, and host names, just bit for bit clones of the template with a customized answer file waiting to be applied.

Normally, vCloud Director will clone the template, push an answer file to it that contains all the things that have to change to make it a unique server, then kick off a sysprep (in windows) to make it all happen. Within a few minutes your clone is now a new server with your settings already set – ready to run.

Not this time. And to be honest, I don’t ever think we deployed a 2008 32bit Enterprise server – so the template may have been broken from day -1.

After digging around in the sysprep logs, I find a break…

SYSPRP LaunchDll:Failure occurred while executing 'C:\Program Files\Internet Explorer\iessetup.dll,SysPrep_Cleanup', returned error code 2
SYSPRP RunExternalDlls:An error occurred while running registry sysprep DLLs, halting sysprep execution. dwRet = 2

Okay, so I verify iessetup.dll exists. So why aren’t you executing it.

After more digging and some Googling, I think I find the root of the issue… we’re missing a registry key.

HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce

It appears that when .NET is installed and removed before sys prep – it can remove the RunOnce key, which is used by Sysprep to store some commands using iessetup.dll.

I simply launched our template, added the key, and wrapped it back up. Redeployed the two servers – and bam…

Two 2k8 32bit servers deployed without a hitch. It’s now 1:10am and I haven’t even touched my other work tonight.

 

Source of Solution:
http://social.technet.microsoft.com/Forums/en-US/itprovistadeployment/thread/a85fa983-344c-49a4-ba64-904c241129bc/

%d bloggers like this: