Using Powershell to create many Hyper-V Virtual Machines in parallel on Gridstore array


In a prior post I put out a powershell script to create many Hyper-V virtual machines on multiple LUNs using the Create-VM.ps1 script. That script ran in a single thread creating one VM before moving on to the next. In this test of the Gridstore array I ran 8 scripts at the same time in parallel.

GS-013a

# Script to create 8 test VMs on a specific LUN on current HyperV Host at a fixed point in time.
# Uses Create-VM.ps1 from https://superwidgets.wordpress.com/category/powershell/
# Sam Boutros – 7/1/2014
#
$GoTime = 40 # minute
$VMPrefix = “V-2012R2-LABk”
$VMrootPath = “k:\VMs”
$NumberofVMs = 8 # Per target LUN
#
# Common variables
$VMG = 2
$VMMemoryType = “Dynamic”
$VMStartupRAM = 1GB
$VMminRAM = 512MB
$VMmaxRAM = 1GB
$vSwitch = “Gridstore_vSwitch”
$VMCores = 2
$VLAN = 19
$AdditionalDisksTotal = 2
$AdditionalDisksSize = 1TB
$GoldenImageDiskPath = “E:\Golden\V-2012R2-3-C.VHDX”
$CSV = (Get-Location).path + “\IOPS_” + (Get-Date -format yyyyMMdd_hhmmsstt) + “.csv”
#
Write-Output “Waiting to start at the $GoTime minute”
do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) # Wait until GoTime
For ($j=1; $j -lt $NumberofVMs+1; $j++) {
$VMName = $VMPrefix + $j
$VMFolder = $VMRootPath + “\” + $VMName
.\Create-VM.ps1 -VMName $VMName -VMFolder $VMFolder -VMG $VMG -VMMemoryType $VMMemoryType -VMStartupRAM $VMStartupRAM -VMminRAM $VMminRAM -VMmaxRAM $VMmaxRAM -vSwitch $vSwitch -VMCores $VMCores -VLAN $VLAN -AdditionalDisksTotal $AdditionalDisksTotal -AdditionalDisksSize $AdditionalDisksSize -GoldenImageDiskPath $GoldenImageDiskPath -CSV $CSV
Start-VM -Name $VMName
}

The other 7 scripts were identical, except for the last letter of the $VMPrefix and the drive letter of the $VMRootPath.

The line do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) ensures that all 8 scripts kick off within 1 second of each other, essentially starting at the same time.

GS-013b

GS-013c

The Hyper-V host was clearly stressed:

GS-013d

GS-013e

GS-013f

The GridControl snap-in showed CPU utilization on the Gridstore storage nodes was holding around 60%:

GS-013g

Write process on the storage nodes hits all 6 nodes:

GS-013h

Read requests hit 4 storage nodes:

GS-013i

Here’s a copy of the metrics summary script output.

One of the 8 scripts had an error:

GS-013j

GS-013k

The error was that the script tried to create VM too fast. The interesting thing here is the magenta error message in this screen shot. It’s from an error handling code in the Create-VM.ps1 script. The script log file Create-VM_V-2012R2-LABk2_20140701_034214PM.txt showed:

2014.07.01 03:42:14 PM: Starting with paramters:
2014.07.01 03:42:14 PM:
2014.07.01 03:42:14 PM: VMName: V-2012R2-LABk2
2014.07.01 03:42:14 PM: VMFolder: k:\VMs\V-2012R2-LABk2
2014.07.01 03:42:14 PM: vSwitch: Gridstore_vSwitch
2014.07.01 03:42:14 PM: GoldenImageDiskPath: E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:14 PM: VMG: 2
2014.07.01 03:42:14 PM: VMMemoryType: Dynamic
2014.07.01 03:42:14 PM: VMStartupRAM: 1073741824 – VMStartupRAM (calculated):
2014.07.01 03:42:14 PM: VMminRAM: 536870912 – VMminRAM (calculated):
2014.07.01 03:42:14 PM: VMmaxRAM: 1073741824 – VMmaxRAM (calculated):
2014.07.01 03:42:14 PM: VMCores: 2 – VMCores (calculated):
2014.07.01 03:42:14 PM: VLAN: 19 – VLAN (Calculated):
2014.07.01 03:42:14 PM: AdditionalDisksTotal: 2 – AdditionalDisksTotal (calculated):
2014.07.01 03:42:14 PM: AdditionalDisksSize: 1099511627776 – AdditionalDisksSize (Calculated): 1099511627776
2014.07.01 03:42:14 PM: .
2014.07.01 03:42:14 PM: Creating VM with the following paramters:
2014.07.01 03:42:14 PM: VM Name: ‘V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Folder: ‘k:\VMs\V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Generation: ‘2’
2014.07.01 03:42:14 PM: VM Memory: Dynamic, Startup memory: 1,024 MB, Minimum memory: 512 MB, Maximum memory: 1,024 MB
2014.07.01 03:42:14 PM: vSwitch Name: ‘Gridstore_vSwitch’
2014.07.01 03:42:14 PM: VM CPU Cores: ‘2’
2014.07.01 03:42:14 PM: Copying disk image k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX from goldem image E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:41 PM: Copy time = 0 : 27 : 762 (Minutes:Seconds:Milliseconds) – 00:00:26.7623983 Milliseconds
2014.07.01 03:42:41 PM: File k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX size: 9,148 MB (9592373248 bytes)
2014.07.01 03:42:41 PM: Throughput: 342 MB/s (341.822877660408 MB/s)
2014.07.01 03:42:41 PM: or: 2,735 Mbps (2734.58302128326 Mbps)
2014.07.01 03:43:32 PM:
2014.07.01 03:43:32 PM: Error: Failed to create VM ‘V-2012R2-LABk2’.. stopping
2014.07.01 03:43:32 PM: Error details: The operation cannot be performed while the virtual machine is in its current state.[The operation cannot be performed while the virtual machine is in its current state..count-1]

Windows event viewer search showed:

GS-013l

So it appears that this “k” script working to create 8 VMs on drive “K” sent New-VM request at 3:43:32 (line 303 of CreateVM.ps1). That request took 11 seconds to complete as shown in event ID 13002 above. In the mean time, the script’s subsequent error checking lines 304, 305 detected that the VM has not bean created and aborted execution as designed.

On another note, I’m sure there are better ways to run powershell commands or scripts in parallel. I’ve looked into The [RunspaceFactory] and [PowerShell] Accelerators while running powershell in Multithreaded Apartment (powerhsell.exe -mta). I’ve also tried Updated Type Accelerator functions for PowerShell 3.0 and 4.0. I have tried the Start-Job/Get-Job cmdlets with script blocks, but had hard time passing parameters to Create-VM.ps1 with $ScriptBlock and -ArgumentList. I’ve also looked into ForEach-Parallel powershell workflow command. I simply needed to get this testing done quickly and did not have time to find a better way to run tasks in parallel in powershell. I welcome any comments or suggestions on better ways to run powershell scripts in parallel.


Conclusion and important points to note:

  • This set of scripts ran tasks that attempted to create 64 VMs on 8 Gridstore vLUNs. It did so by running 8 scripts in parallel, each script attempted to create 8 VMs on a different vLUN. 55 VMs were created successfully. Script failures were tracked down to busy/overwhelmed physical host.
  • Similar to the last test, each of the 8 vLUNs is configured as an IOPS (2+1) LUN
  • Network bandwidth utilization was completely maxed out on the testing Hyper-V host (10 Gbps Ethernet) I will use NIC teams to provide 20 Gbps Ethernet bandwidth
  • Storage nodes’ CPU utilization was at around 60% in this test, which is not a bottleneck.
  • This test is essentially a disk copy test as well and not a Hyper-V virtual machine performance test.
  • Summary comparison between this and the last test:

GS-013m

The throughput values under Gbps and GB/Min. are meant to compare parallel versus serial execution testing and not as metrics for the storage array, because they include time to create and configure virtual machines in addition to time taken by disk operations.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s