Using Powershell to create many Hyper-V Virtual Machines in parallel on Gridstore array

In a prior post I put out a powershell script to create many Hyper-V virtual machines on multiple LUNs using the Create-VM.ps1 script. That script ran in a single thread creating one VM before moving on to the next. In this test of the Gridstore array I ran 8 scripts at the same time in parallel.


# Script to create 8 test VMs on a specific LUN on current HyperV Host at a fixed point in time.
# Uses Create-VM.ps1 from
# Sam Boutros – 7/1/2014
$GoTime = 40 # minute
$VMPrefix = “V-2012R2-LABk”
$VMrootPath = “k:\VMs”
$NumberofVMs = 8 # Per target LUN
# Common variables
$VMG = 2
$VMMemoryType = “Dynamic”
$VMStartupRAM = 1GB
$VMminRAM = 512MB
$VMmaxRAM = 1GB
$vSwitch = “Gridstore_vSwitch”
$VMCores = 2
$VLAN = 19
$AdditionalDisksTotal = 2
$AdditionalDisksSize = 1TB
$GoldenImageDiskPath = “E:\Golden\V-2012R2-3-C.VHDX”
$CSV = (Get-Location).path + “\IOPS_” + (Get-Date -format yyyyMMdd_hhmmsstt) + “.csv”
Write-Output “Waiting to start at the $GoTime minute”
do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) # Wait until GoTime
For ($j=1; $j -lt $NumberofVMs+1; $j++) {
$VMName = $VMPrefix + $j
$VMFolder = $VMRootPath + “\” + $VMName
.\Create-VM.ps1 -VMName $VMName -VMFolder $VMFolder -VMG $VMG -VMMemoryType $VMMemoryType -VMStartupRAM $VMStartupRAM -VMminRAM $VMminRAM -VMmaxRAM $VMmaxRAM -vSwitch $vSwitch -VMCores $VMCores -VLAN $VLAN -AdditionalDisksTotal $AdditionalDisksTotal -AdditionalDisksSize $AdditionalDisksSize -GoldenImageDiskPath $GoldenImageDiskPath -CSV $CSV
Start-VM -Name $VMName

The other 7 scripts were identical, except for the last letter of the $VMPrefix and the drive letter of the $VMRootPath.

The line do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) ensures that all 8 scripts kick off within 1 second of each other, essentially starting at the same time.



The Hyper-V host was clearly stressed:




The GridControl snap-in showed CPU utilization on the Gridstore storage nodes was holding around 60%:


Write process on the storage nodes hits all 6 nodes:


Read requests hit 4 storage nodes:


Here’s a copy of the metrics summary script output.

One of the 8 scripts had an error:



The error was that the script tried to create VM too fast. The interesting thing here is the magenta error message in this screen shot. It’s from an error handling code in the Create-VM.ps1 script. The script log file Create-VM_V-2012R2-LABk2_20140701_034214PM.txt showed:

2014.07.01 03:42:14 PM: Starting with paramters:
2014.07.01 03:42:14 PM:
2014.07.01 03:42:14 PM: VMName: V-2012R2-LABk2
2014.07.01 03:42:14 PM: VMFolder: k:\VMs\V-2012R2-LABk2
2014.07.01 03:42:14 PM: vSwitch: Gridstore_vSwitch
2014.07.01 03:42:14 PM: GoldenImageDiskPath: E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:14 PM: VMG: 2
2014.07.01 03:42:14 PM: VMMemoryType: Dynamic
2014.07.01 03:42:14 PM: VMStartupRAM: 1073741824 – VMStartupRAM (calculated):
2014.07.01 03:42:14 PM: VMminRAM: 536870912 – VMminRAM (calculated):
2014.07.01 03:42:14 PM: VMmaxRAM: 1073741824 – VMmaxRAM (calculated):
2014.07.01 03:42:14 PM: VMCores: 2 – VMCores (calculated):
2014.07.01 03:42:14 PM: VLAN: 19 – VLAN (Calculated):
2014.07.01 03:42:14 PM: AdditionalDisksTotal: 2 – AdditionalDisksTotal (calculated):
2014.07.01 03:42:14 PM: AdditionalDisksSize: 1099511627776 – AdditionalDisksSize (Calculated): 1099511627776
2014.07.01 03:42:14 PM: .
2014.07.01 03:42:14 PM: Creating VM with the following paramters:
2014.07.01 03:42:14 PM: VM Name: ‘V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Folder: ‘k:\VMs\V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Generation: ‘2’
2014.07.01 03:42:14 PM: VM Memory: Dynamic, Startup memory: 1,024 MB, Minimum memory: 512 MB, Maximum memory: 1,024 MB
2014.07.01 03:42:14 PM: vSwitch Name: ‘Gridstore_vSwitch’
2014.07.01 03:42:14 PM: VM CPU Cores: ‘2’
2014.07.01 03:42:14 PM: Copying disk image k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX from goldem image E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:41 PM: Copy time = 0 : 27 : 762 (Minutes:Seconds:Milliseconds) – 00:00:26.7623983 Milliseconds
2014.07.01 03:42:41 PM: File k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX size: 9,148 MB (9592373248 bytes)
2014.07.01 03:42:41 PM: Throughput: 342 MB/s (341.822877660408 MB/s)
2014.07.01 03:42:41 PM: or: 2,735 Mbps (2734.58302128326 Mbps)
2014.07.01 03:43:32 PM:
2014.07.01 03:43:32 PM: Error: Failed to create VM ‘V-2012R2-LABk2’.. stopping
2014.07.01 03:43:32 PM: Error details: The operation cannot be performed while the virtual machine is in its current state.[The operation cannot be performed while the virtual machine is in its current state..count-1]

Windows event viewer search showed:


So it appears that this “k” script working to create 8 VMs on drive “K” sent New-VM request at 3:43:32 (line 303 of CreateVM.ps1). That request took 11 seconds to complete as shown in event ID 13002 above. In the mean time, the script’s subsequent error checking lines 304, 305 detected that the VM has not bean created and aborted execution as designed.

On another note, I’m sure there are better ways to run powershell commands or scripts in parallel. I’ve looked into The [RunspaceFactory] and [PowerShell] Accelerators while running powershell in Multithreaded Apartment (powerhsell.exe -mta). I’ve also tried Updated Type Accelerator functions for PowerShell 3.0 and 4.0. I have tried the Start-Job/Get-Job cmdlets with script blocks, but had hard time passing parameters to Create-VM.ps1 with $ScriptBlock and -ArgumentList. I’ve also looked into ForEach-Parallel powershell workflow command. I simply needed to get this testing done quickly and did not have time to find a better way to run tasks in parallel in powershell. I welcome any comments or suggestions on better ways to run powershell scripts in parallel.

Conclusion and important points to note:

  • This set of scripts ran tasks that attempted to create 64 VMs on 8 Gridstore vLUNs. It did so by running 8 scripts in parallel, each script attempted to create 8 VMs on a different vLUN. 55 VMs were created successfully. Script failures were tracked down to busy/overwhelmed physical host.
  • Similar to the last test, each of the 8 vLUNs is configured as an IOPS (2+1) LUN
  • Network bandwidth utilization was completely maxed out on the testing Hyper-V host (10 Gbps Ethernet) I will use NIC teams to provide 20 Gbps Ethernet bandwidth
  • Storage nodes’ CPU utilization was at around 60% in this test, which is not a bottleneck.
  • This test is essentially a disk copy test as well and not a Hyper-V virtual machine performance test.
  • Summary comparison between this and the last test:


The throughput values under Gbps and GB/Min. are meant to compare parallel versus serial execution testing and not as metrics for the storage array, because they include time to create and configure virtual machines in addition to time taken by disk operations.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s