Posts tagged “Azure Blob Storage

Using CloudBerry Drive to make Azure Block Blob Storage available to Azure Virtual Machines


There’s a number of ways to make Azure storage available to a VM in Azure:

  • Attach a number of local VHD disks. There’s a couple of issues with this approach:
  • Map drives to a number of Azure File SMB shares. There’s a couple of issues with this approach:
    • The shares are not persistent although we can use CMDKEY tool as a workaround.
    • There’s a maximum of 5TB capacity per share, and a maximum of 1TB capacity per file.
  • Use a 3rd party tool such as Cloudberry Drive to make Azure block blob storage available to the Azure VM. This approach has the 500TB Storage account limit which is adequate for use with Veeam Cloud Connect. Microsoft suggests that the maximum NTFS volume size is between 16TB and 256TB on Server 2012 R2 depending on allocation unit size. Using this tool we get 128TB disk suggesting an allocation unit size of 32KB.

To install CloudBerry Drive on an Azure VM:

– Install C++ 2010 x64 Redistributable pre-requisite:

VeeamCC09

– Run CloudBerryDriveSetup, accept the defaults, and reboot.

– In the Azure Management Portal, obtain your storage account access key (either one is fine):

VeeamCC11

– Back in the Azure VM, right-click on the Cloudberry icon in the system tray and select Options:

VeeamCC10

– Under the Storage Accounts tab, click Add, pick Azure Blob as your Storage Provider, enter your Azure Storage account name and key:

VeeamCC12

– Under the Mapped Drives tab, click Add, type-in a volume label, click the button next to Path, and pick a Container. This is the container we created in step 3 above:

VeeamCC13

– You can see the available volumes in Windows explorer or by running this command in Powershell:

Get-Volume | FT -AutoSize

VeeamCC14

Add VHD disks to the VM for the CloudBerry Drive cache:

We’ll add VHD disks to the VM for that cache folder to have sufficient disk space and IOPS for the cache.

VeeamCC27

Highlight the Azure VM, click Attach at the bottom, and click Attach empty disk. Enter a name for the disk VHD file, and a size. The maximum size allowed is 1023 GB (as of September 2014). Repeat this process to add as many disks as allowed by your VM size. For example, an A1 VM can have a maximum of 2 disks, A2 max is 4, A3 max is 8, and A4 max is 16 disks.

In the Azure VM, I created a 2TB disk using Storage Spaces on the VM as shown:

VeeamCC28

This is setup as a simple disk for maximum disk space and IOPS, but it can be setup as mirrored disks as well.

Create a folder for the CloudBerry Drive cache on the new disk, and configure CloudBerry Drive to use it:

CloudBerry2

It’s important to have enough disk space on the drive where CloudBerry Caching occurs. The amount of available space on the Caching drive puts a limit on the file size that can be handled through CloudBerry drive which could be much less than the 128TB available space on a CloudBerry Drive that has an Azure Block Blob back end.

 

Advertisements

Azure VM Storage limitation – using Azure blob storage as backup target


One of the frustrating limitations I’ve come across when using Azure Virtual Machines is the limited disk space amount you can have. This limitation is particularly a hurtle when considering Azure storage as a backup target. The maximum amount of disk space you can have on an Azure VM is 16TB. This limitation stems from 3 issues:

  1. Maximum 1,023 GB per disk. Azure VM disks are stored as Page Blobs in Azure storage accounts. A Page Blob has a maximum of 1TB.
  2. Maximum 16 disks per VM. This is in spite of the fact that the VM OS may support 256 disks per SCSI controller, such as Server 2012 R2.
  3. Maximum 2,040 GB per disk. Although this limitation is superseded by the 1TB page blob limitation, it’s worth noting. This limitation stems from the fact that Azure VMs must use VHD disk format which has a maximum of 2TB/disk, in spite of the fact that the VM operating system may support disks as large as 64TB each, such as Server 2012 R2.

Although we can store large files in Azure as Block Blobs, many backup applications require a VM in the cloud to do WAN acceleration, multi-site deduplication, and similar functions.

AzureMVs

Another annoying feature is the artificial coupling of VM CPU, memory, and disk space resources in Azure VMs. For example, to have a VM with with the maximum allowed 16TB disk space, one must use one of the 5 large VM sizes:

AZStor05

This is the output from this Powershell command:

Get-AzureRoleSize | where { $_.MaxDataDiskCount -gt 15 } | Select InstanceSize,Cores,MemoryInMB,MaxDataDiskCount | Out-GridView

For example, the Basic A4 VM annual cost is about $5,500 compared to about $700 for a Small A1 VM which is all is needed for backup.

A VM with 16TB disk space can be setup from the Azure web interface or via Powershell. I’ve used this script to create and attach 16 disks to an A4 sized VM:

$SubscriptionName = “Visual Studio Premium with MSDN”
$StorageAccountName = “portalvhds1brcsk975g4fq”
Set-AzureSubscription -SubscriptionName $SubscriptionName -CurrentStorageAccountName $StorageAccountName

for ($i=0; $i -lt 16; $i++) {
    $i
    Get-AzureVM -ServiceName “v2012r2-az1” -Name “v2012r2-az1” |
        Add-AzureDataDisk -CreateNew -DiskSizeInGB 1023 -DiskLabel “Disk$i” -LUN $i |
        Update-AzureVM
}

The output looked like:

AZStor06

Back to the Azure VM, I created a disk pool, 1 vDisk, and 1 volume using all the 16 TB space, formatted with 64KB blocks:

AZStor04

The 16 disks before and after setting them up as a single vDisk using Storage Spaces

There may be changes coming down the pipeline that would allow up to 30TB/VM or more. For example, the StorSimple 1100 virtual appliance, which is an Azure VM associated with a StorSimple series 8000 storage array has a maximum capacity of 30TB.



Using Azure Storage with Powershell – Initial computer setup


To be able to use Powershell to run commands, lookup file and container (folder ) lists, transfer files from your computer to your Azure storage account, you need some initial setup. This is a one time setup that uses certificates for authentication.

Azure Powershell cmdlets are part of the Azure PS module. Follow this link to get it.

Before this initial setup, if you try to use the Azure cmdlets like:

Get-AzureStorageAccount

You’ll get an error similar to:

Get-AzureStorageAccount : No current subscription has been designated. Use Select-AzureSubscription
-Current <subscriptionName> to set the current subscription.
At line:1 char:1
+ Get-AzureStorageAccount
+ ~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Get-AzureStorageAccount], Exception
+ FullyQualifiedErrorId : System.Exception,Microsoft.WindowsAzure.Commands.ServiceManagement.Stor
ageServices.GetAzureStorageAccountCommand

blob5

To get started, we need a certificate. A self signed certificate is OK.

  1. Follow this link to download and install SDK for Windows 8. We need this SDK because it contains the makecert.exe tool that we’ll use in the next step. Alternatively, you can make and export a certificate manually in certmgr.msc
  2. Use makecert.exe tool and the following Powershell commands to create a certificate. Change the first line to reflect the certificate name of your choice:
    $certName = “SamAzure1”
    $makecertPath = “C:\Program Files (x86)\Windows Kits\8.0\bin\x64”
    Set-Location $makecertPath
    .\makecert.exe -sky exchange -r -n “CN=$certName” -pe -a sha1 -len 2048 -ss My “$certName.cer” 
  3. Next, upload the resulting .cer file to your Azure account. In Azure Management Interface, click settings on the bottom left, Management Certificates tab, Upload a management certificate:
    blob4Browse to “C:\Program Files (x86)\Windows Kits\8.0\bin\x64” folder, and the .cer file created in step 2 above. It will be named after the $certName value above.
  4. Next, get your subscription name and ID, under settings:
    blob6

  5. Finally, run the following Powershell commands:
    $certName = “SamAzure1”
    $SubscriptionID = “91e711e4-1620-4e5b-a64b-92a190101010”
    $SubscriptionName = “Visual Studio Premium with MSDN”
    $Cert = Get-ChildItem -Path Cert:\CurrentUser\My | Where { $_.Subject -like “CN=$certName” }
    Set-AzureSubscription -SubscriptionName $SubscriptionName -SubscriptionId $SubscriptionID -Certificate $Cert
    Change the first 3 lines to match your subscription details. $certName must be the same name from step 2 above.

You’re done. This setup remains valid as long as your subscription is active and you haven;t deleted the certificate from your Azure account.

To test, run cmdlets like:

Get-AzureStorageAccount
Get-AzureSubscription
Get-AzureVMImage