vSphere 5 – how to dismiss heartbeat and network redundancy errors
Posted on 31 July 2012 | No responses
In some of the lab testing I do I frequently don’t have all the required network or storage redundancies that vSphere wants to satisfy HA. Because of this I end up with the yellow warning triangle on the hosts. No big deal, however if I’m doing testing and want to be alerted to any issues that I may encounter, not having a persistent triangle makes that easier.
Fortunately VMware has a nice KB that outlines a way to ignore some of these common warnings. Check out the KB here. It includes the parameters for a bunch of cool stuff.
Installing ESXi 5.0u1 on HP BL460c Gen8 Blade
Posted on 30 May 2012 | 3 responses
The new HP Gen 8 blades are out and I’ve been testing the install of vSphere ESXi 5.0u1 on them. The first thing that stood out is that the vanilla version of ESXi 5 isn’t compatible with the Gen 8 blade. After looking at the BL460c Gen8 quick spec sheet and trying a couple test installs, it became clear that the HP Array controller was the issue. The BL460c Gen8 blade comes with the HP Smart Array P220i controller whereas the G6 and G7 used the P410i.
Apparently the version of the scsi-hpsa driver that ships with ESXi 5.0u1 doesn’t include support for the P220i. Attempting an install with this version yields the following (notice the lack of a hard drive. The LUN listed is actually a 4 GB SD card installed in the blade).
The solution is to use the Image Builder function in vSphere PowerCLI to remove the stock driver and insert the newer OEM driver with support for the P220i controller. You can find lots of information on Image Builder on the web so the intent of this post is not to do a deep dive of image builder. I used this KB post from VMWare as a basic framework for inserting the driver. Here are the steps I used to add the proper P220i driver into the ESXi 5.0u1 install using Image Builder.
- Download the HPSA driver from VMWare. I got it from here but you should probably verify yourself.
- Download the ESXi 5.0u1 Offline bundle.
- Create a location to store the bundles (ex. D:\ImageBuilder\Depot)
- Add the ESXi offline bundle to the software depot { Add-EsxSoftwareDepot – DepotURL D:\ImageBuilder\Depot\update-from-esxi5.0-5.0_update01.zip }
- View the profiles in the bundle { Get-EsxImageProfile }
- Create a custom image profile by cloning one of the original profiles discovered in step 5. { New-EsxImageProfile -CloneProfile “ESXi-5.0.0-20120302001-standard” -name “custom-esxi50u1-g8″ }
- Check what software packages are part of the image profile you just created. { Get-EsxSoftwarePackage } Note the outlined scsi-hpsa is the version that comes as part of the ESXi 5.0u1 and 5.0 installs and doesn’t work on the Gen 8 blade.
- I removed the included version of the scsi-hpsa driver from my custom version so I can add the newer version that supports the P220i. { Remove-EsxSoftwarePackage -ImageProfile “custom-esxi50u1-g8″ -SoftwarePackage “scsi-hpsa” }
- I then added the HPSA OEM driver I downloaded in step 1 to the software depot. { Add-EsxSoftwareDepot -DepotURL hpsa-500.5.0.0-offline_bundle-537239.zip }
- Use Get-EsxSoftwarePackage to list out all packages in the depot. Note: you will see both the included and OEM scsi-hpsa packages listed. Look for the one that lists Hewlett Packard as the vendor. This is the package you want to add to your custom profile.
- Add the OEM scsi-hpsa package to your custom image. You will need to specify both the name and version (show in the screen shot from step 10) in the command or it will just add the version included with ESXi back in and you will be back to square one. { Add-EsxSoftwarePackage -ImageProfile “custom-esxi50u1-g8″ -SoftwarePackage “scsi-hpsa 5.0.0-24OEM.500.0.0.406165″ }
- Create an ISO and/or Bundle from your custom image. I created both so I could verify the correct driver was inserted by starting a new depot and adding that bundle in and using the Get-EsxSoftwarePackage command. To create the ISO or bundle use the { Export-EsxImageProfile custom-esxi50u1-g8 [-ExportToISO | -ExportToBundle] -filepath D:|blah.[iso | zip] }
- Now boot to the ISO and verify you can see a hard disk to install to. If so you are done.
Lost SA Password for SQL 2008 R2
Posted on 24 May 2012 | No responses
Couldn’t get logged into a lab SQL database using SQL management studio. This box had been cloned and I didn’t know the SA password. The local admin no longer could log in and I later found out this was because the SQL login was in the form of OLDSERVER\administrator, but choosing Windows Authentication on the MGMT studio tried to use NEWSERVER\administrator.
Found a great article on how to reset the local SA account on Gert Van Gorp’s blog. Worked perfectly. Thanks to Gert!
Sysprep causing BSOD on Windows 2008 R2 Clone
Posted on 24 May 2012 | No responses
I recently ran across an issue with cloning a powered on Windows 2008 R2 SQL server. After cloning the VM and successfully booting and rebooting the box several times, I ran sysprep. Shortly after running the sysprep command the VM rebooted and came up with the error “autochk program not found” and then a short view of a BSOD and then a reboot. Tried the clone again several times with no joy.
This thread on the vmware communities led to a couple solutions. Either power off the VM prior to the clone or clear the nodefaultdriveletter bit on the C drive using diskpart. Since I couldn’t easily take an outage and power down SQL I tried the second solution. Worked like a charm. Seems this error may also affect template creation from powered on VMs as well according to the thread.
- Run diskpart on cloned server
- list volume
- select volume 1 (or whichever volume is the c: drive)
- attributes volume
- attributes volume clear nodefaultdriveletter
- reboot
- run sysprep normally
It has been reported that this issue may also affect Windows 7 clones and also template creation from Windows 2008 R2 and Windows 7.
vMotion fails at 9% on ESXi 5.0u1
Posted on 8 May 2012 | No responses
A collegue of mine sent along a solution to a problem he recently encountered (thanks Jason D). A 6 node cluster of ESXi 5.0 u1 hosts experienced a power loss and went down hard. After the hosts were restarted and came back from the unexpected shut down he noticed that he was unable to vMotion virtual machines in that cluster. DRS was also unable to move any VMs around to balance load. The only way to move VMs between hosts was powering them off and using cold migration.
Any time a vMotion was atempted it would fail at 9% with the following error “A general system error occured: Failed to initialize migration at source. Error 0xbad0060. Necessary module isn’t loaded”. Attemps to disabled and re-enable vMotion in the host advanaced options or on the port group did not solve the issue. Here is the solution he came across:
NOTE: VMware has a KB listed here on this.
- SSH to the host
- Check if migration module is running ‘vmkload_mod -l | grep migrate’ . You shouldn’t see anything loaded.
- Disable vMotion on kernel port group.
- Load the module ‘vmkload_mod migrate’
- Verify module is now loaded ‘vmkload_mod -l | grep migrate’. You should seee the migrate module loaded.
- Restart managment agents on host ‘services.sh restart’ and wait for host to reconnect in vCenter.
- Enable vMotion on kernel port group.
- Test vMotion and if working it is recommended you evacuate the host and reboot and check step 2 and vMotion functionality again.
As always all cautions apply and your milage may vary. Don’t do this if you don’t know what you’re doing. If you bork your host I don’t know you. Thanks again Jason D. for the information.
Windows 2008 R2 – Unidentifed Network
Posted on 21 April 2012 | No responses
I came across this issue two times in the past week on two different Windows 2008 R2 virtual machines running on vSphere. Ultimately I was able to fix these two boxes and decided to post what worked for me in hopes it might save someone else some time and frustration.
In one case this error occured out of the blue on a previously working server after a reboot. The second case was a POC server VM that I hadn’t connected to in a couple weeks. When I went back to finish my testing I couldn’t access the server from the network.
In both cases the network connections where now listed and “Unidentifed Network” and all traffic to and from the server was being blocked even though the Firewall was disabled. I could ping the server IP from the server itself, but I could not access anything else on the network. I tried removing and adding the NICs from the OS and VM shell. I also tried the NETSH INT IP RESET to reset the TCP stack. I tried different subnets, IPs, Port Groups, etc, all with no joy. Below is how I finally fixed the issue.
NOTE: These errors occured for me on two different lab servers in somewhat isolated test networks. As such the fix I employed is appropriate for those environments. In a production environment I might be reaching for my credit card or support agreement with MS and give them a call.
- Run Group Policy Editior > gpedit.msc
- Navigate to Computer Configuration > Windows Settings > Security Settings > Network List Manager and select the Unidentified Network entry.
- Under User Permissions set it to ’User can change location’ . The setting under Location Type is proably optional and by default is set to ‘Not configured’
- Open a command prompt and type ‘gpupdate /force’ to update the group policy.
- I ended up having to restart the computer before I was able to change the location but I don’t know that this is a requirement.
QNAP replication with Amazon S3
Posted on 27 December 2011 | No responses
I’ve been backing up my important data on my QNAP using a couple different methods including synchronizing with another storage unit, Vmware VDR and the good old fashioned external drive. I figured it was time to get a better method figured out. After doing some research I decided to use Amazon Simple Storage Services (S3) to do the job. The service is dirt cheap and I feel I can pretty much trust the reliability. As of this writing the Free Usage Tier is 5GB, 20,000 Get Requests; 2000 Put Requests; and 15GB of data transfer a month for 1 year. Storage pricing is 0.140 per GB per month for the first 1 TB of data. Using their simple monthly calulator for 20GB of storage with 5 GB in / 5 GB out and 5000 Put / 5000 Get requests that is about $2.01 a month.
I signed up for an S3 account and was digging around for the best method to automate data synchronization. There are a variety of tools out there like s3sync, s3cmd, syncany, S3Fox for Firefox, JunleDisk, Backup Manager, Sync2S3, CrossFTP, GoodSync and probably a ton of others. Some require package installations or cron jobs to schedule and look like they would work well. However I stumbled on the fact that my QNAP 219P+ support Amazon S3 natively. Nice.
Setting up S3 with your QNAP
- Sign up for an S3 account.
- Create a bucket to use for the replication.
- Retrieve your Access Key ID and Secret Access Key. From within you S3 account pull down Your Account and the select Security Credentials.
- Login to your QNAP and go to the Backup > Remote Replication section.
- Follow the directions over at QNAP to configure the replication and schedule.
Automating SQL Express Backup
Posted on 13 December 2011 | No responses
If you are using MS SQL Express for your vcenter or web development project, you may want to automate the backup of your databases. Unlike the full SQL Server, SQL Express doesn’t have the ability to schedule database backups or maintenance directly. However, this can be accomplished easily if you have SQL Server Management Studio installed. I have a 2 step process running in my lab. The first step uses a scheduled task that runs a saved sql job to back up the database. The second step is to copy the back-up to another location and add the date stamp to the file name. That way I can have multiple day’s of backup stored.
Step 1: Create SQL Backup Job
- Open Microsoft SQL Server Management Studio and expand the databases section.
- Right click on the database you want to backup and select tasks > backup.
- Set the General backup settings like (Full, Differential), backup set name and backup destination and file name. I set my disk path as C:\Sql_Backup to keep it simple for my script in step 2.
- Set the Options for the backup like apend or overwrite backups and whether to verify after backup.
- When you have the settings the way you want them, from the scripts pull down menu choose Script Action to File (see screenshot).
- Save the .sql file to your hard drive.
- Open Task Scheduler and create a new task.
- In the RUN box on the Task enter the following: “C:\Program Files\Microsoft SQL Server\100\Tools\Binn\SQLCMD.EXE” -S .\SQLINSTANCENAME -i “C:\Program Files\Microsoft SQL Server\vcenter-bu-script.sql” .
- Schedule the task to repeat daily or however often you want.
- When you are finished with the task, Run it and verify it works correctly. The file will be saved to the location set in step 3.
Step 2: Create a batch file to copy daily backups to a different location
- Create a Windows batch file with the following code listed below.
- Set up a scheduled task to run the batch file. Make sure the start time is sufficently delayed to allow the sql backup script to complete.
@Echo Off
NET USE T: \\backupserver\backupshare\
@For /F “tokens=2,3,4 delims=/ ” %%A in (‘Date /t’) do @(
Set Day=%%A
Set Month=%%B
Set Year=%%C
Set All=%%C_%%B_%%A
)
@For %%a in (“C:\Sql_backup\*.bak”) do copy %%a “T:\Sql_backups\%%~na_%All%.bak”
NET USE T: /delete
NOTE: When I run the date /t command from the server this runs on I get ‘Tue 12/13/2011′ so the tokens 2,3,4 represent the month, day, year and I’m leaving out the Tue. If running date /t returns a different result you may need to modify the tokens in the script.
vCenter Appliance Memory Tweaks
Posted on 11 December 2011 | 3 responses
I’ve been playing around with the Virtual Center Server Appliance and a friend of mine pointed me to a post that has some memory tweaks that you can use to reduce the memory foot print of the appliance.
See the post on vCSA memory tweaks by Merijntje Tak here (and thanks to him for the information). In his post he describes a method to reduce the amount of RAM DB2 takes and also point to two other files that can be edited.
The following 3 files can be edited to allow you to reduce the RAM allocated to the vCSA.
- /etc/vmware-vpx/tomcat-java-opts.cfg
- /usr/lib/
vmware-vsphere-client/server/bin/dmk.sh - Change the RAM allocated to DB2 as posted by Merijntje
/etc/vmware-vpx/tomcat-java-opts.cfg: Sets the heap size for the JVM (scratch memory)
Default: Xmx1024m MaxPermSize=256m. The Xmx=max heap size; MaxPermSize=permanent heap isn’t subject to garbage collection.
Tweaked: I had changed the Xmx to 512m but experienced stability issues so I’ve set it back to 1024m.
/usr/lib/vmware-vsphere-client/server/bin/dmk.sh: Startup script for vCenter Service.
Default: JAVA_OPTS=”$JAVA_OPTS -Xmx1024m -Xms512m -XX:PermSize-128m”. Xmx=max heap; Xms=starting heap.
Tweaked: JAVA_OPTS=”$JAVA_OPTS -Xmx512m -Xms512m -XX:PermSize-128m”. This has been stable.DB2 Memory Tweak
- Login to the vCenter linux box
- Change to user db2inst1 (SU db2inst1)
- Start the db2 utility (/opt/db2/current/bin/db2)
- Issue the following statements (without quotes):
“connect to vcdb”
“select name, value from sysibmadm.dbmcfg where name = ‘instance_memory’”
“update dbm cfg using instance_memory 65536 immediate”. NOTE: I doubled my instance_memory to 131072
VMware VCenter Server Appliance (vCSA)
Posted on 8 December 2011 | 5 responses
VMware Virtual Center Server Appliance was recently released as GA. A colleague and I decided to take a look at the vCSA in one of our POC labs at work and I decided to try it in my home lab as well. I work at a large enterprise virtualization shop and the vCSA isn’t geared towards that space yet. However it seems that VMware’s long term road map is pointing in that directions. Keep in mind that this is basically a 1.0 release at this point so put it in production at your own risk.
As has been widely published, there are a few features that vCSA doesn’t yet support.
- Linked Mode
- vCenter Heartbeat
- IPv6
- Microsoft SQL Database
- Suggested limit of 5 Hosts and 50 VMs.
The vCSA is delivered as a OVF file that is deployed with 2 vCPU, 8 GB RAM and 82 GB of Disk. You have the option of using the embedded DB2 database or an external Oracle database (in case you have one lying around the lab). It should be noted that vCSA is a licensed product and will require a vCenter license. The same vCenter license will work on either the Windows based or appliance based virtual center.
Installation
Follow instructions from VMware here under the Configuring VMware vCenter Server Appliance section.
Installation Tips
- Connect to the appliance console to configure some basic settings like IP address, time zone and proxy server information.
- Then connect to the web management interface at http://ipaddress:5480 to configure the database settings, AD information and the like.
.
- Set the time zone before you set up the database.
- Set up NTP to keep time sync on the appliance before you set up the database if you want to do so.
- If you select the embedded DB2 database, be sure to click the TEST button before saving the selection on the Database configuration screen. Be sure the test is successful.
- Always restart the appliance from the System Tab. I ran into several instances where the embedded DB2 database became corrupted after rebooting using tools or from the command line. This is just a guess at this point but I’m sticking with it.
Stability Issues with vCSA
Now I need to preface this part of the post with the statement that the stability issues I’ve experienced so far could be due to my environment, set-up procedures, stupidity, sun spots and a variety of other issues that are not related to the actual product.
I have a couple different types of issues with vCSA. First was around setup. Several different times I’ve had the database appear to become corrupt. This has either happened during or shorty after setup or after making changes to the time zone. That is where the suggestions above come from. This has manifested itself with the following systems.
- vSphere Client will not connect. Looking at the web management interface will show the vcenter service stopped. Attempting to manually start the service will fail with various errors.
- Examining the boot process you will see the DB2 service start normally ‘waiting for embedded db2 database to start: success’ , then you will see the message ‘cleaning session lock table: failed’. After that the vmware-vpxd service will start but will fail to initialize with ‘waiting for vpxd to initialize: failed’.
- Any attempts to Reset DB Contents fails with ‘Invalid’
- Any attempts to start and initialize the vmware-vpxd service fails
- The only way I’ve found to fix this is to re-deploy the appliance and make sure to set the time zone and time before configuring the database. Also make sure you have the appliance deployed with at least 4 GB of RAM during the setup. You can reduce the RAM later after doing some tweaks discussed here.
The second type of issue I had has manifested itself recently as I begin to use the vCSA more. It has been increasingly common during backups from VDR. I suspect some of the memory tweaks may be responsible for this issue and I’ve continued to adjust these.
- vSphere Client will disconnect and not reconnect
- Looking at the status tab of the vCSA management web page shows the vCenter service is stopped.
- Starting the vcenter service or rebooting the appliance from the web management interface usually fixes the error.
UPDATE: I’ve re-tweaked some of my initial memory settings and the second issue has gone away. I think I was running the Tomcat JVM and vcenter services too lean on RAM and this was causing the service to hang or stop. Since I’ve increased the settings I’ve had no further issues.
Overall I think the vCSA shows a lot of promise and will continue to evolve with additional database support and features. If you have a home lab I think it is definately worth a look.
