Benchmarking

Benchmarking Azure VM storage


Azure Standard tier virtual machines come with an optional number of persistent page blob disks, up to 1 terabyte each. These disks are expected to deliver 500 IOPS or 60 MB/s throughput eachVeeam-Azure05

Let’s see if we can make that happen.

I started by creating a new Standard Storage account. This is to make sure I’m not hitting the limitation of 20k IOPS/Standard Storage account during testing.

I created a new Standard A3 VM.

Veeam-Azure06

From my on-premises management machine, I used Powershell to create and attach 8x 1TB disks to the VM. To get started with Powershell for Azure see this post.

# Input
$SubscriptionName = '###Removed###' 
$StorageAccount = 'testdisks1storageaccount'
$VMName = 'testdisks1vm'
$ServiceName = 'testdisks1cloudservice'
$PwdFile = ".\$VMName-###Removed###.txt"
$AdminName = '###Removed###'
# Initialize
Set-AzureSubscription -SubscriptionName $SubscriptionName -CurrentStorageAccount $StorageAccount
$objVM = Get-AzureVM -Name $VMName -ServiceName $ServiceName
$VMFQDN = (Get-AzureWinRMUri -ServiceName $ServiceName).Host
$Port = (Get-AzureWinRMUri -ServiceName $ServiceName).Port
# Create and attach 8x 1TB disks
0..7 | % { $_
  $objVM | Add-AzureDataDisk -CreateNew -DiskSizeInGB 1023 -DiskLabel "Disk$_" -LUN $_ | Update-AzureVM
}

Next I used Powershell remoting to create a Storage Space optimized for 1 MB writes. The intend is to use the VM with Veeam Cloud Connect. This particular workloads uses 1 MB write operations.

This code block connects to the Azure VM:

# Remote to the VM 
# Get certificate for Powershell remoting to the Azure VM if not installed already
Write-Verbose "Adding certificate 'CN=$VMFQDN' to 'LocalMachine\Root' certificate store.." 
$Thumbprint = (Get-AzureVM -ServiceName $ServiceName -Name $VMName | 
  select -ExpandProperty VM).DefaultWinRMCertificateThumbprint
$Temp = [IO.Path]::GetTempFileName()
(Get-AzureCertificate -ServiceName $ServiceName -Thumbprint $Thumbprint -ThumbprintAlgorithm sha1).Data | Out-File $Temp
$Cert = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 $Temp
$store = New-Object System.Security.Cryptography.X509Certificates.X509Store "Root","LocalMachine"
$store.Open([System.Security.Cryptography.X509Certificates.OpenFlags]::ReadWrite)
$store.Add($Cert)
$store.Close()
Remove-Item $Temp -Force -Confirm:$false
# Attempt to open Powershell session to Azure VM
Write-Verbose "Opening PS session with computer '$VMName'.." 
if (-not (Test-Path -Path $PwdFile)) { 
  Write-Verbose "Pwd file '$PwdFile' not found, prompting to pwd.."
  Read-Host "Enter the pwd for '$AdminName' on '$VMFQDN'" -AsSecureString | 
    ConvertFrom-SecureString | Out-File $PwdFile 
}
$Pwd = Get-Content $PwdFile | ConvertTo-SecureString 
$Cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $AdminName, $Pwd
$Session = New-PSSession -ComputerName $VMFQDN -Port $Port -UseSSL -Credential $Cred -ErrorAction Stop

Now I have an open PS session with the Azure VM and can execute commands and get output back.

Next I check/verify available disks on the VM:

$ScriptBlock = { Get-PhysicalDisk -CanPool $True }
$Result = Invoke-Command -Session $Session -ScriptBlock $ScriptBlock 
$Result | sort friendlyname | FT -a 

Then I create an 8-column simple storage space optimized for 1 MB stripe size:

$ScriptBlock = { 
  $PoolName = "VeeamPool3"
  $vDiskName = "VeeamVDisk3"
  $VolumeLabel = "VeeamRepo3"
  New-StoragePool -FriendlyName $PoolName -StorageSubsystemFriendlyName “Storage Spaces*” -PhysicalDisks (Get-PhysicalDisk -CanPool $True) |
    New-VirtualDisk -FriendlyName $vDiskName -UseMaximumSize -ProvisioningType Fixed -ResiliencySettingName Simple -NumberOfColumns 16 -Interleave 64KB |
      Initialize-Disk -PassThru -PartitionStyle GPT |
        New-Partition -AssignDriveLetter -UseMaximumSize | 
          Format-Volume -FileSystem NTFS -NewFileSystemLabel $VolumeLabel -AllocationUnitSize 64KB -Confirm:$false
  Get-VirtualDisk | select FriendlyName,Interleave,LogicalSectorSize,NumberOfColumns,PhysicalSectorSize,ProvisioningType,ResiliencySettingName,Size,WriteCacheSize
}
$Result = Invoke-Command -Session $Session -ScriptBlock $ScriptBlock 
$Result | FT -a 

Next I RDP to the VM, and run IOMeter using the following settings:

  • All 4 workers (CPU cores)
  • Maximum Disk Size 1,024,000 sectors (512 MB test file)
  • # of Outstanding IOs: 20 per target. This is make sure the target disk is getting plenty of requests during the test
  • Access specification: 64KB, 50% read, 50% write

I’m expecting to see 4K IOPS and 320 MB/s throughput. (8 disks x 500 IOPS or 60 MB/s each). But this is what I get:

Veeam-Azure02

I got about 1.3 K IOPS instead of 4K, and about 90 MB/s throughput instead of 480 MB/s.

This VM and storage account are in East US 2 region.

This is consistent with other people’s findings like this post.

I also tested using the same 8 disks as a striped disk in Windows. I removed the volume, vDisk, Storage Space, then provisioned a traditional RAID 0 striped disk in this Windows Server 2012 R2 VM. Results were slightly better:

Veeam-Azure03-Striped-Disk8

This is still far off the expected 4k IOPS or 480 MB/s I should be seeing here.


I upgraded the VM to Standard A4 tier, and repeated the same tests:

Veeam-Azure04

Standard A4 VM can have a maximum of 16x 1TB persistent page blob disks. I used powershell to provision and attach 16 disks. Then create a storage space with 16 columns optimized for 1 MB stripe:

Veeam-Azure07

Then I benchmarked storage performance on drive e: using same exact IOMeter settings as above:

Veeam-Azure08

Results are proportionate to the Standard A3 VM test, but they still fall far short.

I’m seeing 2.7 K IOPS instead of the expected 8K IOPS, and about 175 MB/s throughput instead of the expected 960 MB/s


The IOmeter ‘Maximum I/O response Time’ is extremely high (26+ seconds). This has been a consistent finding in all Azure VM testing. This leads me to suspect that the disk requests are being throttled (possibly by the hypervisor).

 


New ‘Test-SBDisk’ function added to SBTools module tests disk IO performance


SBTools module can be downloaded from the Microsoft Script Center Repository.

To install it, extract the .rar file, and run install-SBTools.ps1 in the folder where you extracted the .rar file.

Test-SBDisk:

This is a function to test disk IO performance. It uses other functions in the SBTools module like Log and New-SBSeed.

This function tests disk IO performance by creating random files on the target disk and measuring IO performance. It leaves 2 files in the WorkFolder:

  • A log file that lists script progress, and
  • a CSV file that has a record for each testing cycle

Parameters:

This function accepts 5 parameters:

  1. Parameter WorkFolder
    This is where the test and log files will be created.
    Must be on a local drive. UNC paths are not supported.
    The function will create the folder if it does not exist
    The function will fail if it’s run in a security context that cannot create files and folders in the WorkFolder
    Example: c:\support
  2. Parameter MaxSpaceToUseOnDisk
    Maximum Space To Use On Disk (in Bytes)
    Example: 10GB, or 115MB or 1234567890
  3. .Parameter Threads
    This is the maximum number of concurrent copy processes the script will spawn.
    Maximum is 16. Default is 1.
  4. Parameter Cycles
    The function generates random files in a subfolder under the WorkFolder.
    When the total WorkSubFolder size reaches 90% of MaxSpaceToUseOnDisk, the script deletes all test files and starts over.
    This is a cycle. Each cycle stats are recorded in the CVS and log files
    Default value is 3.
  5. Parameter SmallestFile
    Order of magnitude of the smallest file.
    The function uses the following 9 orders of magnitude: (10KB,100KB,1MB,10MB,100MB,1GB,10GB,100GB,1TB) referred to as 0..8
    For example, SmallestFile value of 4 tells the script to use smallest file size of 100MB
    The script uses a variable: LargestFile, it’s selected to be one order of magnitude below MaxSpaceToUseOnDisk
    To see higher IOPS select a high SmallestFile value
    Default value is 4 (100MB). If the SmallestFile is too high, the script adjusts it to be one order of magnitude below LargestFile

Examples

  • Test-SBDisk “i:\support” 3GB
    This example tests the i: drive, generates files under i:\support, uses a maximum of 3GB disk space on i: drive.
    It runs a single thread, runs for 3 cycles, uses largest file = 100MB (1 order of magnitude below 3GB entered), smallest file = 10MB (1 order of magnitude below largest file)
    Sample screen output:
    Test-SBDisk1Sample log file content:
    2014.08.06 09:20:30 AM: Computer: XHOST16, WorkFolder = i:\support, MaxSpaceToUseOnDisk = 3GB, Threads = 1, Cycles = 3, SmallestFile = 10MB, LargestFile = 100MB
    2014.08.06 09:20:46 AM: Cycle #1 stats:
    2014.08.06 09:20:46 AM: Duration 10.02 seconds
    2014.08.06 09:20:46 AM: Files copied 2.70 GB
    2014.08.06 09:20:46 AM: Number of files 25
    2014.08.06 09:20:46 AM: Average file size 110.70 MB
    2014.08.06 09:20:46 AM: Throughput 276.33 MB/s
    2014.08.06 09:20:46 AM: IOPS 8.84k (32KB block size)
    2014.08.06 09:20:59 AM: Cycle #2 stats:
    2014.08.06 09:20:59 AM: Duration 9.83 seconds
    2014.08.06 09:20:59 AM: Files copied 2.70 GB
    2014.08.06 09:20:59 AM: Number of files 25
    2014.08.06 09:20:59 AM: Average file size 110.70 MB
    2014.08.06 09:20:59 AM: Throughput 281.60 MB/s
    2014.08.06 09:20:59 AM: IOPS 9.01k (32KB block size)
    2014.08.06 09:21:15 AM: Cycle #3 stats:
    2014.08.06 09:21:15 AM: Duration 12.12 seconds
    2014.08.06 09:21:15 AM: Files copied 2.70 GB
    2014.08.06 09:21:15 AM: Number of files 33
    2014.08.06 09:21:15 AM: Average file size 83.87 MB
    2014.08.06 09:21:15 AM: Throughput 228.25 MB/s
    2014.08.06 09:21:15 AM: IOPS 7.30k (32KB block size)
    2014.08.06 09:21:15 AM: Testing completed successfully.
    Sample CSV file content:
    Cycle # Duration (sec) Files (GB) # of Files Avg. File (MB) Throughput (MB/s) IOPS (K) (32KB blocks) Machine Name Start Time End Time
    1 10.02 2.7 25 110.7 276.33 8.84 XHOST16 8/6/2014 9:20 8/6/2014 9:20
    2 9.83 2.7 25 110.7 281.6 9.01 XHOST16 8/6/2014 9:20 8/6/2014 9:20
    3 12.12 2.7 33 83.87 228.25 7.3 XHOST16 8/6/2014 9:20 8/6/2014 9:21
  • Test-SBDisk “i:\support” 11GB -Threads 8 -Cycles 5 -SmallestFile 4
    This example tests the i: drive, generates files under i:\support, uses a maximum of 11GB disk space on i: drive, uses a maximum of 8 threads, runs for 5 cycles, and uses SamllestFile 100MB.

New ‘New-SBSeed’ function added to SBTools module creates random files for disk IO performance


SBTools module can be downloaded from the Microsoft Script Center Repository.

To install it, extract the .rar file, and run install-SBTools.ps1 in the folder where you extracted the .rar file.

New-SBSeed:

This is a recursive function to create files for disk performance testing. Files will have random numbers as content. File name will have ‘Seed’ prefix and .txt extension.

Function will start with the smallest seed file of 10KB, and end with the Seed file specified in the -SeedSize parameter.

Function will create seed files in order of magnitude starting with 10KB and ending with ‘SeedSize’.

Files will be created in the current folder.

This function takes 1 parameter: SeedSize. This is the size of the largest seed file generated. Accepted values are:

10KB
100KB
1MB
10MB
100MB
1GB
10GB
100GB
1TB

Example

New-SBSeed -SeedSize 10MB -Verbose

This example creates seed files starting from the smallest seed 10KB to the seed size specified in the -SeedSize parameter 10MB.
To see the output you can type in:

Get-ChildItem -Path .\ -Filter *Seed*

Sample output:

Mode LastWriteTime Length Name
—- ————- —— —-
-a— 8/6/2014 8:26 AM 102544 Seed100KB.txt
-a— 8/6/2014 8:26 AM 10254 Seed10KB.txt
-a— 8/6/2014 8:39 AM 10254444 Seed10MB.txt
-a— 8/6/2014 8:26 AM 1025444 Seed1MB.txt


Test-Disk powershell script to test disk IO performance


disk-io-performanceThis script tests disk IO performance by creating random files on the target disk and measuring IO performance. The script can be downloaded from the Microsoft Script Center Repository.  It leaves 2 files in the WorkFolder:
A log file that lists script progress, and
a CSV file that has a record for each testing cycle.

Warning:

This script will delete all subfolders in the work-folder. Make sure to run it in a new folder.

For help type in;

help .\Test-Disk.ps1 -full

Example   .\Test-Disk.ps1 “i:\support” 3GB
This example tests the i: drive, generates files under i:\support, uses a maximum of 3GB disk space on i: drive.   It runs a single thread, runs for 3 cycles, uses largest file = 100MB (1 order of magnitude below 3GB entered), smallest file = 10MB (1 order of magnitude below largest file)

Example   .\Test-Disk.ps1 “i:\support” 11GB 8 5 4
This example tests the i: drive, generates files under i:\support, uses a maximum of 11GB disk space on i: drive, uses a maximum of 8 threads, runs for 5 cycles, and uses SamllestFile 100MB.

Script output looks like:

Test-Disk


Get-Busy.ps1 powershell script to create files on many PCs and collect metrics


This script uses Busy.ps1 which is a script that I posted earlier. This script can be downloaded from the Microsoft TechNet Gallery. To use this script you need to edit the 4 data entry lines on top:

GetBusy

  • $WorkFolder = “e:\support” # Folder on each VM where test files will be created
  • $MaxSpaceToUseOnDisk = 20GB # Maximum amount of disk space to be used on each VM’s $WorkFolder during testing
  • $VMPrefix = “V-2012R2-LAB” # We’ll use that to scope this script (only running VMs with name matching this Prefix will be processed)
  • $LocalAdmin = “administrator” # This is the local admin account on the VMs. You can also use a domain account here if you like.

The script requires 1 positional parameter to indicate whether you wish to start the testing on the VMs or stop it. For example to start:

.\get-busy.ps1 start

GS-017e40

 To end the testing, use:

 .\get-busy.ps1 stop

The script will reach out to the VMs being tested and stop the busy.ps1 script, collect the test results, and cleanup the $Workfolder on each VM.

The script generates 2 log files:

  • A general log file that contains the messages displayed on the screen about each action attempted.
  • A CSV file that contains the compiled test results from all the CSV files generated by each busy.ps1 script on each of the tested VMs

Here’s an example of the compiled CSV file Get-busy_20140714_071428PM.csv


Powershell script to generate file read/write activity to test storage performance


This script generates file read/write activity to test storage performance. Set the $WorkFolder and $MaxSpaceToUseOnDisk variables on top to indicate the folder where test files will be created and maximum amount of disk space to be used on $WorkFolder during testing respectively.

The script can be run with the -verbose switch to show more information during processing, like:

.\busy2.ps1 -verbose

As the $WorkFolder exceed 90% of the capacity indicated by the $MaxSpaceToUseOnDisk variable, the scripts deletes all test files and starts a new cycle. the script can be stopped by clicking CTRL-C or setting the registry key HKLM:\Software\Microsoft\Busy to 0

The script includes a recursive function to generate seed files of (decimal) exponential sizes to be used for file copy activities.

The script generates 2 log files. The first one has details on the files created. The second one is a CSV file that has a record for every cycle the script goes through, including cycle duration, sum of generated file sizes, number of files generated on this cycle, average file size, disk throughput, IOPS,…

Here’s an example CSV file output:

Cycle # Duration (sec) Files (GB) # of Files Avg. File (MB) Throughput (MB/s) IOPS (K) Machine Name Start Time End Time
1 337.67 95.35 19 5138.85 289.15 4.63 HV-LAB-05 07/10/2014 19:54 07/10/2014 20:00

The script can be downloaded from the Microsoft TechNet Gallery.

Busy2


7/15/2014:

Version 1.2 is here

The script displays output to the screen like:

It also logs the same to a text file like Busy_HV-LAB-05.txt and enters the statistics in a CSV file like Busy_HV-LAB-05.csv


Using Powershell to create many Hyper-V Virtual Machines on many LUNs asynchronously


In a prior post I put out a powershell script to create many Hyper-V virtual machines on multiple LUNs using the Create-VM.ps1 script. That script ran in a single thread creating one VM before moving on to the next. In another post I manually ran 8 scripts to achieve the goal of running multiple jobs at the same time. In this script I use Start-Job with ScriptBlock and -ArgumentList to run multiple jobs at the same time.

GS-015j

The script can be downloaded from the Microsoft TechNet Gallery.

Hardware:

Compute node (Hyper-V host): 2x Xeon E5-2430L CPUs at 2 GHz with 6 cores each (12 Logical processors each) and 15 MB L3 cache, 96 GB RAM, 2x 10Gbps NICs that are not RDMA capable, setup in a NIC team (Teaming mode: Switch Independent, Load balancing mode: Hyper-V Port). Storage is 4x LUNs from a 3-node Gridstore array. Each LUN is configured as IOPS 2+1 LUN. Each Gridstore storage node has  1x Xeon E5-2403 processor at 1.8GHz with 4 cores (no hyper-threading) and 10 MB L3 cache, 32GB DDR3 1333 MHz DIMM, 4x 3TB 7200 RPM SAS disks, a 550GB PCIe Flash card, and 2x 10Gbps NICs that are not RDMA capable, setup in a NIC team (Broadcom Smart Load Balancing and Failover = switch-independent, no LACP support needed on switch).

The script took 9 minutes and 45 seconds to create the 40 VMs. During that time the Hyper-V host resource monitor showed:

GS-015a

GS-015b

GridControl snap-in showed:

GS-015c

GS-015d

GS-015e

I cal also see script logs piling up during script execution:

GS-015f

I started the 40 VMs manually after the script finished:

GS-015g

This excel file is based on the script output summary CSV file.

GS-015h

GS-015i


Conclusion and important points to note:

  • Hyper-V performance summary: No CPU or memory bottleneck observed during the test.
  • Array performance summary:
    • Files copies: 40
    • File size: 8.93GB
    • Concurrent operations: 4 copy operations at the same time
    • Total vLUNs used: 4
    • Average file copy duration: 10.18 seconds
    • Average throughput: 902.86 MB/s (6 Gbps)
    • Using the formula IOPS = BytesPerSec / TransferSizeInBytes
      LUNs are formatted as 64 KB blocks
      Average IOPS = (902.86*1024)/64 = 14.45k IOPS
  • Although 20 Gbps aggregate bandwidth is available to each of the compute node and the 3 storage nodes, I have not been able to produce network traffic above 10 Gbps.
  • CPU on the storage nodes was around 90% during the copy. The storage nodes can benefit from additional processing capacity.

Using Powershell to create many Hyper-V Virtual Machines in parallel on Gridstore array


In a prior post I put out a powershell script to create many Hyper-V virtual machines on multiple LUNs using the Create-VM.ps1 script. That script ran in a single thread creating one VM before moving on to the next. In this test of the Gridstore array I ran 8 scripts at the same time in parallel.

GS-013a

# Script to create 8 test VMs on a specific LUN on current HyperV Host at a fixed point in time.
# Uses Create-VM.ps1 from https://superwidgets.wordpress.com/category/powershell/
# Sam Boutros – 7/1/2014
#
$GoTime = 40 # minute
$VMPrefix = “V-2012R2-LABk”
$VMrootPath = “k:\VMs”
$NumberofVMs = 8 # Per target LUN
#
# Common variables
$VMG = 2
$VMMemoryType = “Dynamic”
$VMStartupRAM = 1GB
$VMminRAM = 512MB
$VMmaxRAM = 1GB
$vSwitch = “Gridstore_vSwitch”
$VMCores = 2
$VLAN = 19
$AdditionalDisksTotal = 2
$AdditionalDisksSize = 1TB
$GoldenImageDiskPath = “E:\Golden\V-2012R2-3-C.VHDX”
$CSV = (Get-Location).path + “\IOPS_” + (Get-Date -format yyyyMMdd_hhmmsstt) + “.csv”
#
Write-Output “Waiting to start at the $GoTime minute”
do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) # Wait until GoTime
For ($j=1; $j -lt $NumberofVMs+1; $j++) {
$VMName = $VMPrefix + $j
$VMFolder = $VMRootPath + “\” + $VMName
.\Create-VM.ps1 -VMName $VMName -VMFolder $VMFolder -VMG $VMG -VMMemoryType $VMMemoryType -VMStartupRAM $VMStartupRAM -VMminRAM $VMminRAM -VMmaxRAM $VMmaxRAM -vSwitch $vSwitch -VMCores $VMCores -VLAN $VLAN -AdditionalDisksTotal $AdditionalDisksTotal -AdditionalDisksSize $AdditionalDisksSize -GoldenImageDiskPath $GoldenImageDiskPath -CSV $CSV
Start-VM -Name $VMName
}

The other 7 scripts were identical, except for the last letter of the $VMPrefix and the drive letter of the $VMRootPath.

The line do {start-sleep 1} until ((Get-Date).Minute -eq $GoTime) ensures that all 8 scripts kick off within 1 second of each other, essentially starting at the same time.

GS-013b

GS-013c

The Hyper-V host was clearly stressed:

GS-013d

GS-013e

GS-013f

The GridControl snap-in showed CPU utilization on the Gridstore storage nodes was holding around 60%:

GS-013g

Write process on the storage nodes hits all 6 nodes:

GS-013h

Read requests hit 4 storage nodes:

GS-013i

Here’s a copy of the metrics summary script output.

One of the 8 scripts had an error:

GS-013j

GS-013k

The error was that the script tried to create VM too fast. The interesting thing here is the magenta error message in this screen shot. It’s from an error handling code in the Create-VM.ps1 script. The script log file Create-VM_V-2012R2-LABk2_20140701_034214PM.txt showed:

2014.07.01 03:42:14 PM: Starting with paramters:
2014.07.01 03:42:14 PM:
2014.07.01 03:42:14 PM: VMName: V-2012R2-LABk2
2014.07.01 03:42:14 PM: VMFolder: k:\VMs\V-2012R2-LABk2
2014.07.01 03:42:14 PM: vSwitch: Gridstore_vSwitch
2014.07.01 03:42:14 PM: GoldenImageDiskPath: E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:14 PM: VMG: 2
2014.07.01 03:42:14 PM: VMMemoryType: Dynamic
2014.07.01 03:42:14 PM: VMStartupRAM: 1073741824 – VMStartupRAM (calculated):
2014.07.01 03:42:14 PM: VMminRAM: 536870912 – VMminRAM (calculated):
2014.07.01 03:42:14 PM: VMmaxRAM: 1073741824 – VMmaxRAM (calculated):
2014.07.01 03:42:14 PM: VMCores: 2 – VMCores (calculated):
2014.07.01 03:42:14 PM: VLAN: 19 – VLAN (Calculated):
2014.07.01 03:42:14 PM: AdditionalDisksTotal: 2 – AdditionalDisksTotal (calculated):
2014.07.01 03:42:14 PM: AdditionalDisksSize: 1099511627776 – AdditionalDisksSize (Calculated): 1099511627776
2014.07.01 03:42:14 PM: .
2014.07.01 03:42:14 PM: Creating VM with the following paramters:
2014.07.01 03:42:14 PM: VM Name: ‘V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Folder: ‘k:\VMs\V-2012R2-LABk2’
2014.07.01 03:42:14 PM: VM Generation: ‘2’
2014.07.01 03:42:14 PM: VM Memory: Dynamic, Startup memory: 1,024 MB, Minimum memory: 512 MB, Maximum memory: 1,024 MB
2014.07.01 03:42:14 PM: vSwitch Name: ‘Gridstore_vSwitch’
2014.07.01 03:42:14 PM: VM CPU Cores: ‘2’
2014.07.01 03:42:14 PM: Copying disk image k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX from goldem image E:\Golden\V-2012R2-3-C.VHDX
2014.07.01 03:42:41 PM: Copy time = 0 : 27 : 762 (Minutes:Seconds:Milliseconds) – 00:00:26.7623983 Milliseconds
2014.07.01 03:42:41 PM: File k:\VMs\V-2012R2-LABk2\V-2012R2-LABk2-C.VHDX size: 9,148 MB (9592373248 bytes)
2014.07.01 03:42:41 PM: Throughput: 342 MB/s (341.822877660408 MB/s)
2014.07.01 03:42:41 PM: or: 2,735 Mbps (2734.58302128326 Mbps)
2014.07.01 03:43:32 PM:
2014.07.01 03:43:32 PM: Error: Failed to create VM ‘V-2012R2-LABk2’.. stopping
2014.07.01 03:43:32 PM: Error details: The operation cannot be performed while the virtual machine is in its current state.[The operation cannot be performed while the virtual machine is in its current state..count-1]

Windows event viewer search showed:

GS-013l

So it appears that this “k” script working to create 8 VMs on drive “K” sent New-VM request at 3:43:32 (line 303 of CreateVM.ps1). That request took 11 seconds to complete as shown in event ID 13002 above. In the mean time, the script’s subsequent error checking lines 304, 305 detected that the VM has not bean created and aborted execution as designed.

On another note, I’m sure there are better ways to run powershell commands or scripts in parallel. I’ve looked into The [RunspaceFactory] and [PowerShell] Accelerators while running powershell in Multithreaded Apartment (powerhsell.exe -mta). I’ve also tried Updated Type Accelerator functions for PowerShell 3.0 and 4.0. I have tried the Start-Job/Get-Job cmdlets with script blocks, but had hard time passing parameters to Create-VM.ps1 with $ScriptBlock and -ArgumentList. I’ve also looked into ForEach-Parallel powershell workflow command. I simply needed to get this testing done quickly and did not have time to find a better way to run tasks in parallel in powershell. I welcome any comments or suggestions on better ways to run powershell scripts in parallel.


Conclusion and important points to note:

  • This set of scripts ran tasks that attempted to create 64 VMs on 8 Gridstore vLUNs. It did so by running 8 scripts in parallel, each script attempted to create 8 VMs on a different vLUN. 55 VMs were created successfully. Script failures were tracked down to busy/overwhelmed physical host.
  • Similar to the last test, each of the 8 vLUNs is configured as an IOPS (2+1) LUN
  • Network bandwidth utilization was completely maxed out on the testing Hyper-V host (10 Gbps Ethernet) I will use NIC teams to provide 20 Gbps Ethernet bandwidth
  • Storage nodes’ CPU utilization was at around 60% in this test, which is not a bottleneck.
  • This test is essentially a disk copy test as well and not a Hyper-V virtual machine performance test.
  • Summary comparison between this and the last test:

GS-013m

The throughput values under Gbps and GB/Min. are meant to compare parallel versus serial execution testing and not as metrics for the storage array, because they include time to create and configure virtual machines in addition to time taken by disk operations.


Using Powershell to create 64 Hyper-V Virtual Machines on Gridstore array


This script uses another script Create-VM.PS1 from a prior post. It creates 64 Hyper-V virtual machines on 8 different LUNs. It creates them synchronously (one after the other). The 8 LUNs are on a 6 node GridStore array using H-nodes. Details of the Gridstore array setup are in this post.

GS-012i

# Script to create 64 test VMs on 8 diffent LUNs on current HyperV Host
# Uses Create-VM.ps1 from https://superwidgets.wordpress.com/category/powershell/
# Sam Boutros – 7/1/2014
#
$VMPrefix = “V-2012R2-LAB”
$VMG = 2
$VMMemoryType = “Dynamic”
$VMStartupRAM = 1GB
$VMminRAM = 512MB
$VMmaxRAM = 1GB
$vSwitch = “Gridstore_vSwitch”
$VMCores = 2
$VLAN = 19
$AdditionalDisksTotal = 2
$AdditionalDisksSize = 1TB
$GoldenImageDiskPath = “E:\Golden\V-2012R2-3-C.VHDX”
$CSV = (Get-Location).path + “\IOPS_” + (Get-Date -format yyyyMMdd_hhmmsstt) + “.csv”
$VMFirstNumber = 1 # Starting number
$TargetLUNs = @(“e”,”g”,”h”,”i”,”j”,”k”,”l”,”m”)
$NumberofVMs = 8 # Per target LUN
#
foreach ($LUN in $TargetLUNs) {
$VMrootPath = $LUN + “:\VMs”
For ($j=$VMFirstNumber; $j -lt $NumberofVMs+$VMFirstNumber; $j++) {
$VMName = $VMPrefix + $j
$VMFolder = $VMRootPath + “\” + $VMName
.\Create-VM.ps1 -VMName $VMName -VMFolder $VMFolder -VMG $VMG -VMMemoryType $VMMemoryType -VMStartupRAM $VMStartupRAM -VMminRAM $VMminRAM -VMmaxRAM $VMmaxRAM -vSwitch $vSwitch -VMCores $VMCores -VLAN $VLAN -AdditionalDisksTotal $AdditionalDisksTotal -AdditionalDisksSize $AdditionalDisksSize -GoldenImageDiskPath $GoldenImageDiskPath -CSV $CSV
Start-VM -Name $VMName
}
$VMFirstNumber += $NumberofVMs
}

During the script execution the Hyper-V resource monitor showed the following:

GS-012a

GS-012b

The GridControl snap-in showed:

GS-012c

GS-012d

GS-012e

I can also see the VMs popping up in Hyper-V Manager:

GS-012f

and their files in the file system:

GS-012g

GS-012h

This file is derived from the script’s CSV file output.. It shows:

GS-012j
9 GB file average copy time: 10.83 seconds
GS-012k

9 GB file copy average throughput 852.5 MB/s

GS-012l

9 GB file copy average throughput 6,820.1 Gbps


Conclusion and important points to note:

  • Script ran in serial, creating 1 VM on 1 LUN, and after that’s done moving on to the next VM on the same LUN, then moving on to the next LUN.
  • Each LUN is configured as an IOPS (2+1) LUN. So, every write process is writing to 3 disks out of the array 24 total disks (whereas a read process reads from 2 disks in the LUN). Additional throughput is likely to be achieved by testing scenarios where we’re writing to all 8 LUNs simultaneously hitting 18 of the array disks at the same time.  
  • Network bandwidth utilization is at about 68.2% capacity of the 10 Gbps Ethernet used in this test. for the next test (in parallel) I will use NIC teams to provide 20 Gbps Ethernet bandwidth
  • Storage nodes’ CPU utilization was at around 50% in this test, which is not a bottleneck.
  • This test is essentially a disk copy test and not a Hyper-V virtual machine performance test. Hyper-V testing with Gridstore will be shown in a future post.
  • Using the formula IOPS = BytesPerSec / TransferSizeInBytes
    LUNs are formatted as 64 KB blocks
    Average IOPS = (852.5*1024)/64 = 13.64k IOPS

Benchmarking Gridstore enterprise storage array (2)


This is another post in a series of posts in the process of performance testing and benchmarking Gridstore enterprise storage array.

Gridstore Array components:

6x H-Nodes. Each has 1x Xeon E5-2403 processor at 1.8 GHz with 4 cores (no hyper-threading) and 10 MB L3 cache, 32 GB DDR3 1333 MHz DIMM, 4x 3TB 7200 RPM SAS disks and a 550 GB PCIe Flash card.

GS-009k

Testing environment:

One compute node with 2x Xeon E5-2430L CPUs at 2 GHz with 6 cores each (12 Logical processors) and 15 MB L3 cache, 96 GB RAM

Pre-test network bandwidth verification:

Prior to testing array disk IO, I tested the availability of bandwidth on the Force 10 switch used. I used NTttcp Version 5.28 tool. One of the array nodes was the receiver:

GS-002

The HV-LAB-01 compute node was the sender:

GS-003

I configured the tool to use 4 processor cores only since the Gridstore storage nodes have only 4 cores.

The result was usable bandwidth of 8.951 Gbps (1,18.9 MB/s) – Testing was done using standard 1,500 MTU frames not 9,000 MTU jumbo frames.


vLUNs:

8 vLUNs were configured for this test.

GS-009l

Each vLUN is configured as follows:

  • Protect Level: 1 (striped across 3 Gridstore nodes, fault tolerant to survive single node failure)
  • Optimized for: IOPS
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 64 KB
  • Size: 5 TB (3 segments, 2.5 TB each)

This configuration utilizes each all 24 disks in this grid. (6 nodes * 4 disks each = 24 disks = 8 vLUNs * 3 disks each). It provides optimum array throughput.


Testing tool:

Intel’s IOMeter version 2006.07.27

24 workers, each configured to target all 8 vLUNs – 32 outstanding I/Os

GS-009m

IO profile: 50% read/50% write, 10% random, 8k alignment:

GS-009n

Test duration: 10 minutes


Test result:

IOMeter showed upwards of 17.7k IOPS:

GS-009a

Disk performance details on the compute node:

GS-009b

CPU performance details on the compute node:

GS-009c

Network performance on one of the storage nodes:

GS-009d

Disk performance on one of the storage nodes:

GS-009e

CPU performance on one of the storage nodes:

GS-009f

Overall summary performance on one of the storage nodes:

GS-009g

CPU utilization on the storage nodes as shown from the GridControl snap-in:

GS-009h

Bytes received:

GS-009i

Final test result:

GS-009j

and test details.


Conclusion and important points to note:

  • Network utilization maxed out beyond the 10 Gbps single NIC used on both the compute and storage nodes. This suggests that the array is likely to deliver more IOPS if more network bandwidth is available. Next test will use 2 teamed NICs on the compute node as well as 3 storage nodes with teamed 10 Gbps NICs as well.
  • CPU is maxed on the storage nodes during the test. Storage nodes have 4 cores. This suggests that CPU may be a bottleneck on storage nodes. It also leads me to believe that a) more processing power is needed on the storage nodes, and b) RDMA NICs are likely to enhance performance greatly. The Mellanox ConnectX-3 VPI dual port PCIe8 card may be just what the doctor ordered. In a perfect environment, I would have that coupled with the Mellanox Infiniband MSX6036F-1BRR 56Gbps Switch.
  • Disk IO performance on the storage nodes during the test showed about 240 MB/s data transfer, or about 60 MB/s per each of the disks in the node. This corresponds to the native IO performance of the SAS disks. This suggests minimal/negligible boost from the 550 GB PCIe flash card in the storage node.