Posts tagged “Gridstore benchmarking

Using Powershell to create many Hyper-V Virtual Machines on many LUNs asynchronously


In a prior post I put out a powershell script to create many Hyper-V virtual machines on multiple LUNs using the Create-VM.ps1 script. That script ran in a single thread creating one VM before moving on to the next. In another post I manually ran 8 scripts to achieve the goal of running multiple jobs at the same time. In this script I use Start-Job with ScriptBlock and -ArgumentList to run multiple jobs at the same time.

GS-015j

The script can be downloaded from the Microsoft TechNet Gallery.

Hardware:

Compute node (Hyper-V host): 2x Xeon E5-2430L CPUs at 2 GHz with 6 cores each (12 Logical processors each) and 15 MB L3 cache, 96 GB RAM, 2x 10Gbps NICs that are not RDMA capable, setup in a NIC team (Teaming mode: Switch Independent, Load balancing mode: Hyper-V Port). Storage is 4x LUNs from a 3-node Gridstore array. Each LUN is configured as IOPS 2+1 LUN. Each Gridstore storage node has  1x Xeon E5-2403 processor at 1.8GHz with 4 cores (no hyper-threading) and 10 MB L3 cache, 32GB DDR3 1333 MHz DIMM, 4x 3TB 7200 RPM SAS disks, a 550GB PCIe Flash card, and 2x 10Gbps NICs that are not RDMA capable, setup in a NIC team (Broadcom Smart Load Balancing and Failover = switch-independent, no LACP support needed on switch).

The script took 9 minutes and 45 seconds to create the 40 VMs. During that time the Hyper-V host resource monitor showed:

GS-015a

GS-015b

GridControl snap-in showed:

GS-015c

GS-015d

GS-015e

I cal also see script logs piling up during script execution:

GS-015f

I started the 40 VMs manually after the script finished:

GS-015g

This excel file is based on the script output summary CSV file.

GS-015h

GS-015i


Conclusion and important points to note:

  • Hyper-V performance summary: No CPU or memory bottleneck observed during the test.
  • Array performance summary:
    • Files copies: 40
    • File size: 8.93GB
    • Concurrent operations: 4 copy operations at the same time
    • Total vLUNs used: 4
    • Average file copy duration: 10.18 seconds
    • Average throughput: 902.86 MB/s (6 Gbps)
    • Using the formula IOPS = BytesPerSec / TransferSizeInBytes
      LUNs are formatted as 64 KB blocks
      Average IOPS = (902.86*1024)/64 = 14.45k IOPS
  • Although 20 Gbps aggregate bandwidth is available to each of the compute node and the 3 storage nodes, I have not been able to produce network traffic above 10 Gbps.
  • CPU on the storage nodes was around 90% during the copy. The storage nodes can benefit from additional processing capacity.
Advertisements

Using Powershell to create 64 Hyper-V Virtual Machines on Gridstore array


This script uses another script Create-VM.PS1 from a prior post. It creates 64 Hyper-V virtual machines on 8 different LUNs. It creates them synchronously (one after the other). The 8 LUNs are on a 6 node GridStore array using H-nodes. Details of the Gridstore array setup are in this post.

GS-012i

# Script to create 64 test VMs on 8 diffent LUNs on current HyperV Host
# Uses Create-VM.ps1 from https://superwidgets.wordpress.com/category/powershell/
# Sam Boutros – 7/1/2014
#
$VMPrefix = “V-2012R2-LAB”
$VMG = 2
$VMMemoryType = “Dynamic”
$VMStartupRAM = 1GB
$VMminRAM = 512MB
$VMmaxRAM = 1GB
$vSwitch = “Gridstore_vSwitch”
$VMCores = 2
$VLAN = 19
$AdditionalDisksTotal = 2
$AdditionalDisksSize = 1TB
$GoldenImageDiskPath = “E:\Golden\V-2012R2-3-C.VHDX”
$CSV = (Get-Location).path + “\IOPS_” + (Get-Date -format yyyyMMdd_hhmmsstt) + “.csv”
$VMFirstNumber = 1 # Starting number
$TargetLUNs = @(“e”,”g”,”h”,”i”,”j”,”k”,”l”,”m”)
$NumberofVMs = 8 # Per target LUN
#
foreach ($LUN in $TargetLUNs) {
$VMrootPath = $LUN + “:\VMs”
For ($j=$VMFirstNumber; $j -lt $NumberofVMs+$VMFirstNumber; $j++) {
$VMName = $VMPrefix + $j
$VMFolder = $VMRootPath + “\” + $VMName
.\Create-VM.ps1 -VMName $VMName -VMFolder $VMFolder -VMG $VMG -VMMemoryType $VMMemoryType -VMStartupRAM $VMStartupRAM -VMminRAM $VMminRAM -VMmaxRAM $VMmaxRAM -vSwitch $vSwitch -VMCores $VMCores -VLAN $VLAN -AdditionalDisksTotal $AdditionalDisksTotal -AdditionalDisksSize $AdditionalDisksSize -GoldenImageDiskPath $GoldenImageDiskPath -CSV $CSV
Start-VM -Name $VMName
}
$VMFirstNumber += $NumberofVMs
}

During the script execution the Hyper-V resource monitor showed the following:

GS-012a

GS-012b

The GridControl snap-in showed:

GS-012c

GS-012d

GS-012e

I can also see the VMs popping up in Hyper-V Manager:

GS-012f

and their files in the file system:

GS-012g

GS-012h

This file is derived from the script’s CSV file output.. It shows:

GS-012j
9 GB file average copy time: 10.83 seconds
GS-012k

9 GB file copy average throughput 852.5 MB/s

GS-012l

9 GB file copy average throughput 6,820.1 Gbps


Conclusion and important points to note:

  • Script ran in serial, creating 1 VM on 1 LUN, and after that’s done moving on to the next VM on the same LUN, then moving on to the next LUN.
  • Each LUN is configured as an IOPS (2+1) LUN. So, every write process is writing to 3 disks out of the array 24 total disks (whereas a read process reads from 2 disks in the LUN). Additional throughput is likely to be achieved by testing scenarios where we’re writing to all 8 LUNs simultaneously hitting 18 of the array disks at the same time.  
  • Network bandwidth utilization is at about 68.2% capacity of the 10 Gbps Ethernet used in this test. for the next test (in parallel) I will use NIC teams to provide 20 Gbps Ethernet bandwidth
  • Storage nodes’ CPU utilization was at around 50% in this test, which is not a bottleneck.
  • This test is essentially a disk copy test and not a Hyper-V virtual machine performance test. Hyper-V testing with Gridstore will be shown in a future post.
  • Using the formula IOPS = BytesPerSec / TransferSizeInBytes
    LUNs are formatted as 64 KB blocks
    Average IOPS = (852.5*1024)/64 = 13.64k IOPS