Squishnet

vMotion fails at 9% on ESXi 5.0u1

Posted on 8 May 2012 | No responses

A collegue of mine sent along a solution to a problem he recently encountered (thanks Jason). A 6 node cluster of ESXi 5.0 u1 hosts experienced a power loss and went down hard. After the hosts were restarted and came back from the unexpected shut down he noticed that he was unable to vMotion virtual machines in that cluster. DRS was also unable to move any VMs around to balance load. The only way to move VMs between hosts was powering them off and using cold migration.

Any time a vMotion was atempted it would fail at 9% with the following error “A general system error occured: Failed to initialize migration at source. Error 0xbad0060. Necessary module isn’t loaded”. Attemps to disabled and re-enable vMotion in the host advanaced options or on the port group did not solve the issue. Here is the solution he came across:

SSH to the host

Check if migration module is running ‘vmkload_mod -l | grep migrate’ . You shouldn’t see anything loaded.

Disable vMotion on kernel port group.

Load the module ‘vmkload_mod migrate’

Verify module is now loaded ‘vmkload_mod -l | grep migrate’. You should seee the migrate module loaded.

Restart managment agents on host ‘services.sh restart’ and wait for host to reconnect in vCenter.

Enable vMotion on kernel port group.

Test vMotion and if working it is recommended you evacuate the host and reboot and check step 2 and vMotion functionality again.

As always all cautions apply and your milage may vary. Don’t do this if you don’t know what you’re doing. If you bork your host I don’t know you. Thanks again Jason for the information.

Windows 2008 R2 – Unidentifed Network

Posted on 21 April 2012 | No responses

I came across this issue two times in the past week on two different Windows 2008 R2 virtual machines running on vSphere. Ultimately I was able to fix these two boxes and decided to post what worked for me in hopes it might save someone else some time and frustration.

In one case this error occured out of the blue on a previously working server after a reboot. The second case was a POC server VM that I hadn’t connected to in a couple weeks. When I went back to finish my testing I couldn’t access the server from the network.

In both cases the network connections where now listed and “Unidentifed Network” and all traffic to and from the server was being blocked even though the Firewall was disabled. I could ping the server IP from the server itself, but I could not access anything else on the network. I tried removing and adding the NICs from the OS and VM shell. I also tried the NETSH INT IP RESET to reset the TCP stack. I tried different subnets, IPs, Port Groups, etc, all with no joy. Below is how I finally fixed the issue.

NOTE: These errors occured for me on two different lab servers in somewhat isolated test networks. As such the fix I employed is appropriate for those environments. In a production environment I might be reaching for my credit card or support agreement with MS and give them a call.

Run Group Policy Editior > gpedit.msc

Navigate to Computer Configuration > Windows Settings > Security Settings > Network List Manager and select the Unidentified Network entry.

Under User Permissions set it to ’User can change location’ . The setting under Location Type is proably optional and by default is set to ‘Not configured’

Open a command prompt and type ‘gpupdate /force’ to update the group policy.

I ended up having to restart the computer before I was able to change the location but I don’t know that this is a requirement.

Open the ‘Network and Sharing Center’ Under the view your active networks section click on Unidentified Network (shown boxed in Red )

When the Set Network Location box opens, select either Home or Work network. After making this selection I was able to access the network again.

A collegue also suggested disabling the IPSec Policy service as he had heard that this service may be responsible for causing the intial setting change to Unidentified. You may want to be careful with this because if this service is required due to security policies on your network you may cause yourself more problems.

QNAP replication with Amazon S3

Posted on 27 December 2011 | No responses

I’ve been backing up my important data on my QNAP using a couple different methods including synchronizing with another storage unit, Vmware VDR and the good old fashioned external drive. I figured it was time to get a better method figured out. After doing some research I decided to use Amazon Simple Storage Services (S3) to do the job. The service is dirt cheap and I feel I can pretty much trust the reliability. As of this writing the Free Usage Tier is 5GB, 20,000 Get Requests; 2000 Put Requests; and 15GB of data transfer a month for 1 year. Storage pricing is 0.140 per GB per month for the first 1 TB of data. Using their simple monthly calulator for 20GB of storage with 5 GB in / 5 GB out and 5000 Put / 5000 Get requests that is about $2.01 a month.

I signed up for an S3 account and was digging around for the best method to automate data synchronization. There are a variety of tools out there like s3sync, s3cmd, syncany, S3Fox for Firefox, JunleDisk, Backup Manager, Sync2S3, CrossFTP, GoodSync and probably a ton of others. Some require package installations or cron jobs to schedule and look like they would work well. However I stumbled on the fact that my QNAP 219P+ support Amazon S3 natively. Nice.

Setting up S3 with your QNAP

Sign up for an S3 account.
Create a bucket to use for the replication.
Retrieve your Access Key ID and Secret Access Key. From within you S3 account pull down Your Account and the select Security Credentials.
Login to your QNAP and go to the Backup > Remote Replication section.
Follow the directions over at QNAP to configure the replication and schedule.

Automating SQL Express Backup

Posted on 13 December 2011 | No responses

If you are using MS SQL Express for your vcenter or web development project, you may want to automate the backup of your databases. Unlike the full SQL Server, SQL Express doesn’t have the ability to schedule database backups or maintenance directly. However, this can be accomplished easily if you have SQL Server Management Studio installed. I have a 2 step process running in my lab. The first step uses a scheduled task that runs a saved sql job to back up the database. The second step is to copy the back-up to another location and add the date stamp to the file name. That way I can have multiple day’s of backup stored.

Step 1: Create SQL Backup Job

Open Microsoft SQL Server Management Studio and expand the databases section.
Right click on the database you want to backup and select tasks > backup.
Set the General backup settings like (Full, Differential), backup set name and backup destination and file name. I set my disk path as C:\Sql_Backup to keep it simple for my script in step 2.
Set the Options for the backup like apend or overwrite backups and whether to verify after backup.
When you have the settings the way you want them, from the scripts pull down menu choose Script Action to File (see screenshot).
Save the .sql file to your hard drive.
Open Task Scheduler and create a new task.
In the RUN box on the Task enter the following: “C:\Program Files\Microsoft SQL Server\100\Tools\Binn\SQLCMD.EXE” -S .\SQLINSTANCENAME -i “C:\Program Files\Microsoft SQL Server\vcenter-bu-script.sql” .
Schedule the task to repeat daily or however often you want.
When you are finished with the task, Run it and verify it works correctly. The file will be saved to the location set in step 3.

Step 2: Create a batch file to copy daily backups to a different location

Create a Windows batch file with the following code listed below.
Set up a scheduled task to run the batch file. Make sure the start time is sufficently delayed to allow the sql backup script to complete.

@Echo Off
NET USE T: \\backupserver\backupshare\
@For /F “tokens=2,3,4 delims=/ ” %%A in (‘Date /t’) do @(
Set Day=%%A
Set Month=%%B
Set Year=%%C
Set All=%%C_%%B_%%A
)
@For %%a in (“C:\Sql_backup\*.bak”) do copy %%a “T:\Sql_backups\%%~na_%All%.bak”
NET USE T: /delete

NOTE: When I run the date /t command from the server this runs on I get ‘Tue 12/13/2011′ so the tokens 2,3,4 represent the month, day, year and I’m leaving out the Tue. If running date /t returns a different result you may need to modify the tokens in the script.

vCenter Appliance Memory Tweaks

Posted on 11 December 2011 | 3 responses

I’ve been playing around with the Virtual Center Server Appliance and a friend of mine pointed me to a post that has some memory tweaks that you can use to reduce the memory foot print of the appliance.

See the post on vCSA memory tweaks by Merijntje Tak here (and thanks to him for the information). In his post he describes a method to reduce the amount of RAM DB2 takes and also point to two other files that can be edited.

The following 3 files can be edited to allow you to reduce the RAM allocated to the vCSA.

/etc/vmware-vpx/tomcat-java-opts.cfg
/usr/lib/vmware-vsphere-client/server/bin/dmk.sh
Change the RAM allocated to DB2 as posted by Merijntje

/etc/vmware-vpx/tomcat-java-opts.cfg: Sets the heap size for the JVM (scratch memory)
Default: Xmx1024m MaxPermSize=256m. The Xmx=max heap size; MaxPermSize=permanent heap isn’t subject to garbage collection.
Tweaked: I had changed the Xmx to 512m but experienced stability issues so I’ve set it back to 1024m.

/usr/lib/vmware-vsphere-client/server/bin/dmk.sh: Startup script for vCenter Service.
Default: JAVA_OPTS=”$JAVA_OPTS -Xmx1024m -Xms512m -XX:PermSize-128m”. Xmx=max heap; Xms=starting heap.
Tweaked: JAVA_OPTS=”$JAVA_OPTS -Xmx512m -Xms512m -XX:PermSize-128m”. This has been stable.

DB2 Memory Tweak

Login to the vCenter linux box

Change to user db2inst1 (SU db2inst1)

Start the db2 utility (/opt/db2/current/bin/db2)

Issue the following statements (without quotes):
“connect to vcdb”
“select name, value from sysibmadm.dbmcfg where name = ‘instance_memory’”
“update dbm cfg using instance_memory 65536 immediate”. NOTE: I doubled my instance_memory to 131072

VMware VCenter Server Appliance (vCSA)

Posted on 8 December 2011 | 5 responses

VMware Virtual Center Server Appliance was recently released as GA. A colleague and I decided to take a look at the vCSA in one of our POC labs at work and I decided to try it in my home lab as well. I work at a large enterprise virtualization shop and the vCSA isn’t geared towards that space yet. However it seems that VMware’s long term road map is pointing in that directions. Keep in mind that this is basically a 1.0 release at this point so put it in production at your own risk.

As has been widely published, there are a few features that vCSA doesn’t yet support.

Linked Mode
vCenter Heartbeat
IPv6
Microsoft SQL Database
Suggested limit of 5 Hosts and 50 VMs.

The vCSA is delivered as a OVF file that is deployed with 2 vCPU, 8 GB RAM and 82 GB of Disk. You have the option of using the embedded DB2 database or an external Oracle database (in case you have one lying around the lab). It should be noted that vCSA is a licensed product and will require a vCenter license. The same vCenter license will work on either the Windows based or appliance based virtual center.

Installation

Follow instructions from VMware here under the Configuring VMware vCenter Server Appliance section.

Installation Tips

Connect to the appliance console to configure some basic settings like IP address, time zone and proxy server information.
Then connect to the web management interface at http://ipaddress:5480 to configure the database settings, AD information and the like.

Set the time zone before you set up the database.
Set up NTP to keep time sync on the appliance before you set up the database if you want to do so.
If you select the embedded DB2 database, be sure to click the TEST button before saving the selection on the Database configuration screen. Be sure the test is successful.

Always restart the appliance from the System Tab. I ran into several instances where the embedded DB2 database became corrupted after rebooting using tools or from the command line. This is just a guess at this point but I’m sticking with it.

Stability Issues with vCSA

Now I need to preface this part of the post with the statement that the stability issues I’ve experienced so far could be due to my environment, set-up procedures, stupidity, sun spots and a variety of other issues that are not related to the actual product.

I have a couple different types of issues with vCSA. First was around setup. Several different times I’ve had the database appear to become corrupt. This has either happened during or shorty after setup or after making changes to the time zone. That is where the suggestions above come from. This has manifested itself with the following systems.

vSphere Client will not connect. Looking at the web management interface will show the vcenter service stopped. Attempting to manually start the service will fail with various errors.

Examining the boot process you will see the DB2 service start normally ‘waiting for embedded db2 database to start: success’ , then you will see the message ‘cleaning session lock table: failed’. After that the vmware-vpxd service will start but will fail to initialize with ‘waiting for vpxd to initialize: failed’.
Any attempts to Reset DB Contents fails with ‘Invalid’
Any attempts to start and initialize the vmware-vpxd service fails
The only way I’ve found to fix this is to re-deploy the appliance and make sure to set the time zone and time before configuring the database. Also make sure you have the appliance deployed with at least 4 GB of RAM during the setup. You can reduce the RAM later after doing some tweaks discussed here.

The second type of issue I had has manifested itself recently as I begin to use the vCSA more. It has been increasingly common during backups from VDR. I suspect some of the memory tweaks may be responsible for this issue and I’ve continued to adjust these.

vSphere Client will disconnect and not reconnect
Looking at the status tab of the vCSA management web page shows the vCenter service is stopped.
Starting the vcenter service or rebooting the appliance from the web management interface usually fixes the error.

UPDATE: I’ve re-tweaked some of my initial memory settings and the second issue has gone away. I think I was running the Tomcat JVM and vcenter services too lean on RAM and this was causing the service to hang or stop. Since I’ve increased the settings I’ve had no further issues.

Overall I think the vCSA shows a lot of promise and will continue to evolve with additional database support and features. If you have a home lab I think it is definately worth a look.

QNAP TS-219P+

Posted on 7 April 2011 | 2 responses

I’ve been looking to add some new network attached storage for my home network. I wanted something that would provide NFS and iSCSI access to use with my virtualization lab as well as simple CIFS/SMB for use as shared storage.

A collegue mentioned QNAP as one option and I began doing some research on them. The units have a very full feature set and a review at Smallnetbuilder.com showed good performance data . I went ahead and purchased the Qnap TS-219p+ along with 2 SAMSUNG EcoGreen F2 HD154UI 1.5TB 32MB Cache SATA 3.0Gb/s 3.5″ Internal Hard Drives. These are 5400 RPM drives with a manufacturer reported maximum transfer rate of 300 MB/sec. Tom’s Hardware lists this same drive as having 107 MB/s maximum transfer speed in their testing. The QNAP TS-219p+ is not on the Vmware HCL but I had no issues adding the NFS mount to my ESXi 4.1 host.

QNAP TS-219p+ Specs

CPU: Marvell 6282 1.6GHz

DRAM: 512MB DDRIII RAM

Flash Memory: 16MB

HDD: 2 x 3.5″ and 2.5” SATA II HDD

HDD Tray: 2 x hot-swappable tray

LAN Port: 1 x Gigabit RJ-45 Ethernet port

USB: 3 x USB 2.0 port (Front:1 Back: 2)
Supports USB printer, disk, pen drive, USB hub, and USB UPS, etc.

eSATA: 2 x eSATA port (Back)

Power Consumption (W)
Sleep mode: 7W
In operation: 19W (with 2 x 500GB HDD installed)

File System

Internal HDD: EXT3, EXT4
External HDD: EXT3, EXT4, NTFS, FAT32, HFS+

Networking

TCP/IP (IPv4 & IPv6: Dual Stack)
Jumbo Frames
DHCP Client, DHCP Server
Protocols: CIFS/SMB, AFP (3.1), NFS, FTP, HTTP, HTTPS, Telnet, SSH, iSCSI and SNMP.
iSCSI
Network Service Discovery (UPnP, Bonjour)
USB Wi-Fi 802.11n Adapter Support (Optional Purchase)

Network File Sharing

CIFS/SMB (Plus DFS Support)
AFP
NFS
FTP
WebDAV

Disk Management

Single Disk, JBOD, RAID 0 / 1

Performance Testing

I used Iometer to benchmark NFS and iSCSI performance of the QNAP TS-219P+. For a test setup I configured the unit as Raid 0 and used a Windows XP SP3 VM configured with 1 GB RAM and 1 vCPU. My initial results have been redacted due to an issue I discovered with my hyper-visor hardware that caused poor throughput and iops numbers. My colleague Jason does a lot of testing of storage and virtualiztion and offered to re-test my Qnap on his infrastructure. His results were much better and more in line with my expectations.

Conclusions

The Qnap 219p+ has a polished UI and an impressive feature set. Overall IOPS and throughput performance are quite good for a 2 disk soho NAS system (considering I was using 5400 RPM drives). Although the 219p+ is not on the VMware HCL, it integrated into my vmware lab seamlessly. I was able to attach both NFS and iSCSI datastores in ESXi 4.1 without problem. A comparison of performancevs. the Iomega Ix2-200 showed the QNAP besting it in most categories (interestingly the Iomega edged out the QNAP in iSCSI for the 60% Random, 65% Read and Random 8K 70% read tests). The Ix2-200 costs several hundred dollars less than the QNAP + drives so it is worth a look if you want to save a few dollars. The Ix2-200 is also listed on the VMware HCL, whereas the QNAP 219p+ is not.

One thing that I did not expect to see was the disparity between the NFS and iSCSI performance. In some cases, NFS performed at almost 1.5x that of NFS. If you are looking for a flexible, full featured and well built soho NAS, I would highly recommend looking at QNAP.

502 Proxy Error – Apache reverse proxy

Posted on 8 June 2010 | 1 response

I configured an Apache2 reverse proxy as described here. I could get the index page of the site to load but the css wasn’t getting applied. If I clicked any links on the site I get a 502 proxy error.

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /page2.html.

Reason: DNS lookup failure for: internal.domain3.compage2.html

I noticed that URL in the Reason line was missing the / between the domain and page2.html, but since it was present in the GET /page2.html I dismissed that clue. I launched into troubleshooting mode and checked DNS resolution, mod_proxy config, caching, etc.

Finally I took a shot in the dark and went back to the original error. I added a terminal / (shown in red below) to the mod_proxy statements in my httpd.conf file and if flipping worked. I didn’t see this required in any of the documents I read on configuring mod_proxy but who know’s. Maybe I’m a dumbass.

ProxyPass / http://internal.domain3.com/
ProxyPassReverse / http://internal.domain3.com/
ProxyHTMLURLMap http://internal.domain3.com/ /

Apache Reverse Proxy & Virtual Hosts

Posted on 8 June 2010 | 1 response

I’ve been working on a .NET project and needed to make a development box available on the web. I already have another server running that is serving up several other websites. Since I only have 1 IP from my ISP I needed to figure out how to pass port 80 traffic to multiple internal hosts.

You cannot NAT port 80 to two different internal or DMZ hosts with only 1 public IP. You can of course use PAT if your willing to use an alternate port for one of the servers. I did some quick research on reverse proxies originally intending to use Nginx. I’d worked with it on a project at one of my previous gigs. Then I discovered that Apache using mod_proxy can do the same thing.

Proxy Traffic Flow

I’m not going to go through all details, but you can find a good how to here at apache tutor. I’ll list a short run-down of the highlights for my situation.

Upgraded my apache2 installation to apache2-dev so I had access to apxs to build mod_proxy_html (note you may or may not need mod_proxy_html but you probably will).

Download and compile mod_proxy_html from here. http://apache.webthing.com/mod_proxy_html/

Build mod_proxy_html using apxs

# apxs -c -I/usr/include/libxml2 -I. -i mod_proxy_html.c

(make sure you have libxml2 installed. If you get errors building mod_proxy_html.c try upgrading your libxml2 – worked for me)

Now edit your httpd.conf to make sure you load the correct modules

LoadModule proxy_module         modules/mod_proxy.so
LoadModule proxy_http_module    modules/mod_proxy_http.so
LoadFile    /usr/lib/libxml2.so
LoadModule proxy_html_module    modules/mod_proxy_html.so

Configure the virtual hosts section of httpd.conf. I’ve got a FQDN of www.domain3.com that I want to reverse proxy to a different internal server which answers to internal.domain3.com.   The below is how I configured my httpd.conf.

<VirtualHost *:80>
ServerName www.domain3.com

ProxyRequests off
ProxyPass / http://internal.domain3.com/
ProxyPassReverse / http://internal.domain3.com/
ProxyHTMLURLMap http://internal.domain3.com/ /
</VirtualHost>

Restart apache #/etc/init.d/httpd restart

Citrix Receiver for iPhone Installation

Posted on 7 June 2010 | No responses

I guess the title of the post is a bit incorrect. I’m not here to tell you all the hows/whys of installing the Citrix Receiver backend or iPhone app. However I’m will tell you how I fixed my two vexing errors. The below is done on Citrix Presentation Server 4.5. I am not using the gateway product. Users access the Citrix web interface by going to https://citrix.mydomain.com . Nothing fancy.

I created the PNAgent site as described here (watch the video). I installed the Citrix iPhone Receiver app and attempted to access my site. I got an error “unable to load app list” . Eventually I got that fixed and then was able to load my app list. However any time I attempted to launch an app I’d get the black screen with “Please” wait and then eventually I’d get kicked back to my app list. No errors on the Citrix Server side.

I found lots of post telling you to change your gateway configuration or change the secure access settings similar to what you need to do to get the web interfaced working when the Citrix server is behind a NAT firewall. Well my PNAgent didn’t have the option to set the gateway access settings. Now what? Well the solution to both problems are listed below.

ERROR: Unable to Load App List

Check the Server Farm Settings: Citrix AMC > Citrix Resources > Configuration Tools > Web Interface > PNAgent Site (whatever this is called).

Right Click PNAgent Website > Manage Server Farms > Select Farm > Verify XML Service Port is 80 and Transport type is http. If mine was set to https or ssl relay I got the error.

The second, more vexing error was not being able to launch the apps once I got the list to load. If your Citrix Web access is working (and your not using the secure gateway product and are behind a NAT firewall) the solution is pretty easy. If you’re trying to get your web access to work check here. Anyway here is the solution.

ERROR: Clicking on an app takes your to a black screen that says “Please Wait”. Eventually the app never opens and you are kicked back to the app list.

Navigate to C:\Inetpub\wwwroot\Citrix\AccessPlatform\conf and copy the WebInterface.conf file to your desktop. Rename it so you know it’s from the Access Platform site.

Navigate to C:\Inetpub\wwwroot\Citrix\PNAgent\conf directory and copy the WebInterface.conf file to your desktop

You should now have the two webinterface.conf files. Open both using a text editior. You will need to copy 2 lines from the accessplateform webinterface file over to the PNAgent web interface file and then save it back to the PNAgent conf directory.

Look for the lines “ClientAddressMap=192.168.x.x/255.255.255.0,Normal,*,Alternate” in the acessplatform file and copy that to the PNAgent File. Note the IP listed may be different and is the internal subnet your Citrix server is on.

Look for “ServerAddressMap=192.168.x.x:1494,10.10.10.10:1494″ and copy it from the accessplatform file to the pnagent file and over write the existing setting.

Save the changes to the webinterface.conf file for the PNAgent site. Make a copy of the original and then over write the one in C:\Inetpub\wwwroot\Citrix\PNAgent\conf . That’s it. It should work now.

older posts »