Posts tagged “Storage

Get-Busy.ps1 powershell script to create files on many PCs and collect metrics


This script uses Busy.ps1 which is a script that I posted earlier. This script can be downloaded from the Microsoft TechNet Gallery. To use this script you need to edit the 4 data entry lines on top:

GetBusy

  • $WorkFolder = “e:\support” # Folder on each VM where test files will be created
  • $MaxSpaceToUseOnDisk = 20GB # Maximum amount of disk space to be used on each VM’s $WorkFolder during testing
  • $VMPrefix = “V-2012R2-LAB” # We’ll use that to scope this script (only running VMs with name matching this Prefix will be processed)
  • $LocalAdmin = “administrator” # This is the local admin account on the VMs. You can also use a domain account here if you like.

The script requires 1 positional parameter to indicate whether you wish to start the testing on the VMs or stop it. For example to start:

.\get-busy.ps1 start

GS-017e40

 To end the testing, use:

 .\get-busy.ps1 stop

The script will reach out to the VMs being tested and stop the busy.ps1 script, collect the test results, and cleanup the $Workfolder on each VM.

The script generates 2 log files:

  • A general log file that contains the messages displayed on the screen about each action attempted.
  • A CSV file that contains the compiled test results from all the CSV files generated by each busy.ps1 script on each of the tested VMs

Here’s an example of the compiled CSV file Get-busy_20140714_071428PM.csv

Advertisements

Benchmarking Gridstore enterprise storage array (1)


Gridstore provides an alternative to traditional enterprise storage. Basic facts about Gridstore storage technology include:

  • It provides storage nodes implemented as 1 RU servers that function collectively as a single storage array.
  • Connectivity between the nodes and the storage consumers/compute nodes occurs over one or more 1 or 10 Gbps Ethernet connections.
  • NIC teaming can be setup on the Gridstore nodes to provide additional bandwidth and fault tolerance
  • It utilizes a virtual controller to present storage to Windows servers

IO testing tool and its settings is detailed in this post.

vLUNs can be easily created using the GridControl snap-in. This testing is done with a Gridstore array composed of 6 H-nodes. Click node details to see more.

Prior to testing array disk IO, I tested the availability of bandwidth on the Force 10 switch used. I used NTttcp Version 5.28 tool. One of the array nodes was the receiver:

GS-002

 

The HV-LAB-01 compute node was the sender:

GS-003

I configured the tool to use 4 processor cores only since the Gridstore storage nodes had only 4 cores.

The result was usable bandwidth of 8.951 Gbps (1,18.9 MB/s) – Testing was done using standard 1,500 MTU frames not 9,000 MTU jumbo frames.

Test details::

On the receiver Gridstore storage node:
C:\Support>ntttcp.exe -r -m 4,*,10.5.19.30 -rb 2M -a 16 -t 120
Copyright Version 5.28
Network activity progressing…
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 120.011 311727.158 60023.949
1 120.011 233765.293 53126.468
2 120.011 306670.676 56087.990
3 120.011 293592.705 52626.788
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
134280.569568 120.011 1457.709 1118.902
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
17902.435 3.864 2148489.113
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
17388.114 46.288 26563.098 30.300
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
4634562 96592255 599 0 62.960

On the sender compute node: HV-LAB-05
C:\Support>ntttcp.exe -s -m 4,*,10.5.19.30 -rb 2M -a 16 -t 120
Copyright Version 5.28
Network activity progressing…
Thread Time(s) Throughput(KB/s) Avg B / Compl
====== ======= ================ =============
0 120.003 311702.607 65536.000
1 120.003 233765.889 65536.000
2 120.003 306669.667 65536.000
3 120.003 293592.660 65536.000
##### Totals: #####
Bytes(MEG) realtime(s) Avg Frame Size Throughput(MB/s)
================ =========== ============== ================
134268.687500 120.004 1457.441 1118.868
Throughput(Buffers/s) Cycles/Byte Buffers
===================== =========== =============
17901.895 2.957 2148299.000
DPCs(count/s) Pkts(num/DPC) Intr(count/s) Pkts(num/intr)
============= ============= =============== ==============
25915.561 1.504 71032.291 0.549
Packets Sent Packets Received Retransmits Errors Avg. CPU %
============ ================ =========== ====== ==========
96601489 4677580 22698 1 7.228


Test 1:

Compute node(s): 1 physical machine with 2x Xeon E5-2430L CPUs at 2 GHz with 6 cores each (12 Logical processors) and 30 MB L3 cache, 96 GB RAM, 2x 10 Gbps NICs

GS-001

vLUN:

  • Protect Level: 0 (no fault tolerance, striped across 4 Gridstore nodes)
  • Optimized for: N/A
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (4 segments, 512 GB each)

GS-A05

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 10.43k IOPS

GS-A01

 

In the above image you can see the read/write activity to the 4 nodes that make up this vLUN listed under Network Activity in the Resource Monitor/Network tab.

GS-A02

At the same time, the 4 nodes that make up this vLUN showed average CPU utilization around 40%. This dropped down to 0% right after the test.

GS-A03

The 4 nodes’ memory utilization averaged around 25% during the test. It’s baseline is 20%

GS-A04

 


Test 2: The same single compute node above

vLUN:

  • Protect Level: 1 (striped across 3 Gridstore nodes, fault tolerant to survive single node failure)
  • Optimized for: IOPS
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (3 segments, 1 TB each)

GS-B04

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 11.32k IOPS

GS-B01

 

GS-B02

 

GS-B03

 


Test 3: The same single compute node above

vLUN:

  • Protect Level: 1 (striped across 5 Gridstore nodes, fault tolerant to survive single node failure)
  • Optimized for: Throughput
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (5 segments, 512 GB each)

GS-C01

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 9.28k IOPS
GS-C02

GS-C03

 

GS-C04


Test 4: The same single compute node above

vLUN:

  • Protect Level: 2 (striped across 6 Gridstore nodes, fault tolerant to survive 2 simultaneous node failures)
  • Optimized for: Throughput
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (6 segments, 512 GB each)

GS-D01

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 4.56k IOPS

GS-D02

GS-D03

GS-D04

 


Test 5: The same single compute node above

2 vLUNs:

1. The same Grid Protection Level 1 vLUN from test 1 above with Platinum QoS setting +

2. Identical 2nd vLUN except that QoS is set to Gold:

  • Protect Level: 1 (striped across 3 Gridstore nodes, fault tolerant to survive 1 node failure)
  • Optimized for: IOPS
  • QoS: Gold
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (3 segments,  1 TB each)

GS-E01

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 10.52k IOPS

GS-E02

GS-E03

GS-E04

 


 

Test 6: The same single compute node above

3 vLUNs:

All the same:

  • Protect Level: 1 (striped across 3 Gridstore nodes, fault tolerant to survive 1 node failure)
  • Optimized for: IOPS
  • QoS: Platinum
  • Unmasked: to 1 server
  • File system: NTFS
  • Block size: 32 KB
  • Size: 2 TB (3 segments,  1 TB each)

GS-F1

Result:

Testing with 24 vCores, 10 Gbps NIC, 1 compute node, 32k block size, 50% read/50% write IO profile => 9.94k IOPS

GS-F6

GS-F5

GS-F4

GS-F3

GS-F2


 Summary:

GS-004


Using Azure Storage with Powershell – Getting started


Az01

  • After installation, type in Azure in the search field, click the search icon, then click Install to see installed modules on top:

Az02

  • Close the Microsoft Web Platform Installer. To get back to it to add/remove modules you can use the icon:

Az04

  • Notice the new Azure set of icons added:

Az03

  • I’ve pinned the Azure Powershell icon to the task bar. Open it by right clicking on it, then right click on Windows Azure Powershell and click Run as administrator

Az05

  •  To connect to your Azure account, type in Add-AzureAccount

Az06

  • Azure Powershell displays a message similar to:

Az07

  • Check your subscription(s) using the command: Get-AzureSubscription

Az08

  • If you have more than one subscription under your Azure account, you may want to switch to a specific subscription. Use this command to switch a given subscription and make it the default subscription: Select-AzureSubscription -SubscriptionName “Visual Studio Premium with MSDN” -Default
    Substitute “Visual Studio Premium with MSDN” with SubscriptionName as shown from the Get-AzureSubscription command

Az09

If you don’t have a storage account setup under your Azure subscription, you can create one from your Azure portal. Click Storage on the left, then click New at the bottom:

AZ11

Type in a name for the new account you wish to create – must be lower case letter only. Pick an Azure data center – typically one that’s physically close to your location to get better latency. Pick a subscription. Pick a replication setting. Locally-redundant give you 3 copies of your data in the data center you selected. Geo-redundant gives you 3 additional copies in another Azure data center. Geo-redundant is typically twice the cost of locally-redundant storage account, and is the default option.

AZ10

In a minute or 2 Azure will finish creating the storage account. click on the account name:

AZ12

Next click Dashboard, and click Manage Access Keys at the bottom:

AZ131

Copy the account name and the primary access key. You will need them to use your storage account via Powershell later.

AZ14

Secure this information because it provides access to your Azure data. Data can be accessed by using either the primary or secondary keys. Each key is 88 characters long and is made up of alphanumeric upper and lower case letters and special characters. The availability of 2 keys allows us to change keys without losing access by applications or machines that use the account. For example in case of key compromise, and you’re using the primary key in an application or machine, you can:

  1. Regenerate the secondary key
  2. Replace the key in the script/application/machine using the storage account (no access interruption)
  3. Regenerate the primary key

Now you have changed your account keys without any service interruption


Migrating IP settings from one NIC to another using Powershell


Here’s an example scenario where the following script may be particularly useful:

GridStore array where each node currently uses one 1 Gbps NIC. After adding 10 Gbps NIC to each node, we’d like to migrate the IP settings from the 1 Gbps NIC to the 10 Gbps NIC on each node. GridStore utilizes commodity rack mount servers and hardware and a robust software driver to present scalable, high performance, fully redundant vLUNs. More detailed posts on GridStore will follow.

This diagram shows network connectivity before adding the 10 Gbps NICs:

Before

 

After adding the 10 Gbps NICs:

After

Steps:

  1. You will need administrative credentials to the nodes from GridStore technical support
  2. From Server1, using GridControl snapin, stop all vLUNs:
    Stop-vLUNs
  3. RDP to each node.
    logon
  4. Currently nodes run Windows 7 Embedded and the RDP session will bring up a command prompt.
  5. Run Control to show Control Panel, double-click Network and Sharing Center, click Change Adapter Settings to view/confirm that you have 2 connected NICs:
    NICs
  6. Start Powershell ISE:
    start-ps-ise
  7. Copy/paste the following script and run it on each node:

    GS-2

# Script to move network configuration from one NIC to another on a GridStore node
# Sam Boutros
# 6/16/2014
# Works with Powershell 2.0
#
Set-Location “c:\support”
$Loc = Get-Location
$Date = Get-Date -format yyyymmdd_hhmmsstt
$logfile = $Loc.path + “\Move-GSNIC_” + $env:COMPUTERNAME + “_” + $Date + “.txt”
function log($string) {
Write-Host $string; $temp = “: ” + $string
$string = Get-Date -format “yyyy.mm.dd hh:mm:ss tt”; $string += $temp
$string | out-file -FilePath $logfile -append
}
#
log “Switching NIC configuration on $env:COMPUTERNAME”
$ConnectedNICs = Get-WmiObject Win32_NetworkAdapterConfiguration -Filter ‘IPEnabled=”true”‘
If ($ConnectedNICs.Count -lt 2) {log “Error: Less than 2 connected NICs:”; log $ConnectedNICs}
else {
if ($ConnectedNICs.Count -gt 2) {log “Error: More than 2 connected NICs:”; log $ConnectedNICs}
else { # 2 connected NICs
Stop-Service GridStoreManagementService
# Storing NICs details in variables for later use
$NIC0Index = $ConnectedNICs[0].index; log “NIC 0 Index: $NIC0Index”
$NIC0Desc = $ConnectedNICs[0].description; log “NIC 0 Description: $NIC0Desc”
$NIC0IPv4 = $ConnectedNICs[0].IPAddress[0]; log “NIC 0 IPv4: $NIC0IPv4”
$NIC0Mask = $ConnectedNICs[0].IPSubnet[0]; log “NIC 0 Subnet Mask: $NIC0Mask”
$Nic0 = Get-WmiObject win32_networkadapter -filter “DeviceId = $NIC0Index”
$NIC0ConnID = $Nic0.NetConnectionID; log “NIC 0 NetConnectionID: $NIC0ConnID”
#
$NIC1Index = $ConnectedNICs[1].index; log “NIC 1 Index: $NIC1Index”
$NIC1Desc = $ConnectedNICs[1].description; log “NIC 1 Description: $NIC1Desc”
$NIC1IPv4 = $ConnectedNICs[1].IPAddress[0]; log “NIC 1 IPv4: $NIC1IPv4”
$NIC1Mask = $ConnectedNICs[1].IPSubnet[0]; log “NIC 1 Subnet Mask: $NIC1Mask”
$Nic1 = Get-WmiObject win32_networkadapter -filter “DeviceId = $NIC1Index”
$NIC1ConnID = $Nic1.NetConnectionID; log “NIC 1 NetConnectionID: $NIC1ConnID”
# Identify Target NIC and Source NIC
if ($ConnectedNICs[0].IPAddress[0] -match “169.254”) {$TargetNIC = 0} else {$TargetNIC = 1}
$SourceNIC = 1 – $TargetNIC; log “Source NIC: NIC $SourceNIC”
$SourceIP = $ConnectedNICs[$SourceNIC].IPAddress[0]
$SourceMask = $ConnectedNICs[$SourceNIC].IPSubnet[0]
log “Source IP: $SourceIP”
# Setting IP address for Source NIC to DHCP
log “Changing IP address of source NIC to DHCP”
if ($ConnectedNICs[$SourceNIC].EnableDHCP().ReturnValue -eq 0) {
log “==> IP setting change was successful”} else {log “==> IP setting change failed”}
$ConnectedNICs[$SourceNIC].SetDNSServerSearchOrder()
# Need to disable and re-enable the Source NIC for the settings to take effect (!?)
$Nic0.Disable(); Start-Sleep -s 2
$Nic0.Enable(); Start-Sleep -s 2
# Setting IP address for Target NIC to the same values previousely held by the Source NIC
$TargetIP = $ConnectedNICs[$TargetNIC].IPAddress[0]
log “Target IP: $TargetIP”
log “Changing IP address of Target NIC to $SourceIP with subnet mask $SourceMask”
if ($ConnectedNICs[$TargetNIC].EnableStatic($SourceIP,$SourceMask).ReturnValue -eq 0) {
log “==> IP address change was successful”} else {log “==> IP address change failed”}
Remove-Item -Path “HKLM:\SOFTWARE\Wow6432Node\Gridstore\NetworkAdapter”
Start-Sleep -s 2
Start-Service GridStoreManagementService
}
}
Invoke-Expression “$env:windir\system32\Notepad.exe $logfile”