Posts tagged “Azure Storage

Unmanaged Azure Disk Snapshot functions added to AZSBTools PowerShell module to View, Add, Delete


3 new functions have been added to the AZSBTools PowerShell module to view, add, and delete snapshots of unmanaged Azure disks

The AzureRM.Compute PowerShell module (v 5.9.1 as of 1 Jan 2019) provides the Get-AzureRmSnapshotNew-AzureRmSnapshot,  and Remove-AzureRmSnapshot cmdlets to handle disk snapshots of managed disks. Although the same snapshot capabilities are available for unmanaged disks, there are no PS cmdlets to provide similar functionality for unmanaged disk snapshots.

Managed versus unmanaged disks

Managed Disks handle the storage account creation/management to avoid hitting the 20k IOPS standard storage account limit for example (no more than 40 standard disks in the same Storage Account @ 500 IOPS each). This is a pretty nice feature but it comes at a steep price. Consider Standard LRS USEast region pricing for example. A 1 TB data disk (S30) is $40.96/month

If we use 10 GB of this disk we pay $41/month even if we power off and deallocate the VM. We still pay the $41/month for this 1 TB disk including the 990 GB unused space.

In comparison, the same 1 TB unmanaged disk using the same 500 IOPS with 10 GB of used space costs $0.45/month (10 GB * $0.045 per GB).

This is because we’re billed only for the 10 GB used space not the 1023 GB allocated space, whether the VM is up and running or powered off and deallocated.

In short, managed disks come at the cost of paying for allocated space not used space. Given that used space is often as little as 1% of the allocated space, I highly recommend against using managed disks at this time (1 Jan 2019)

Well, using my own advise I find it hard to use unmanaged disks via PowerShell to perform routine Azure infrastructure management tasks such as:

  • Determine the amount of used disk space (separate post on that)
  • Manage disk snapshots; list, create, delete (the subject of this post)
  • Convert managed to unmanaged disk (separate post on that)

The functions Get-AzureRMUnmanagedDiskSnapshot, New-AzureRMUnmanagedDiskSnapshot, and Remove-AzureRMUnmanagedDiskSnapshot have been added to the AZSBTools module to simplify management of unmanaged disk snapshots via PowerShell. These functions do not affect the VM lease on its disk(s), do not require VM shutdown, and do not interfere with VM operation.

Get-AzureRMUnmanagedDiskSnapshot

This function will list disk snapshots for a given unmanaged disk. This applies to unmanaged ARM disk snapshots only not classic ASM disks or managed ARM disks. This function depends on the AzureRM and Azure PowerShell modules available in the PowerShell Gallery. To install required module: Install-Module AzureRM, Azure 

Example:

$ParameterList = @{
    LoginName = 'sam@dmain.com'
    SubscriptionName = 'my subscription name'
    StorageAccountName = 'storfluxwidget3vm'
    ContainerName = 'vhds'
    BlobName = 'Widget3VM-20181226-093810.vhd'
}
Get-AzureRMUnmanagedDiskSnapshot @ParameterList

To list snapshots in a given time frame, we filter on the SnapshotTime property of the output provided by this function. This function returns objects of type Microsoft.WindowsAzure.Storage.Blob.CloudPageBlob for each snapshot found that matches the provided storageaccount/container/blob parameters. The SnapshotTime property is of type DateTimeOffset which cannot be compared directly to DateTime type. To do the required filtering/comparison, we use the [DateTimeOffset].ToLocalTime{() method as in:

Get-AzureRMUnmanagedDiskSnapshot @ParameterList | 
    where { [DateTime]$_.SnapshotTime.ToLocalTime().ToString() -GE [DateTime]'2019-01-02 8:45' }

This will list snapshots taken at or after 2 Jan 2019 8:45 (am local time)

New-AzureRMUnmanagedDiskSnapshot

This function will create new disk snapshot for a given unmanaged disk.

Example:

$ParameterList = @{
    LoginName = 'sam@dmain.com'
    SubscriptionName = 'my subscription name'
    StorageAccountName = 'storfluxwidget3vm'
    ContainerName = 'vhds'
    BlobName = 'Widget3VM-20181226-093810.vhd'
}
New-AzureRMUnmanagedDiskSnapshot @ParameterList

Remove-AzureRMUnmanagedDiskSnapshot

This function will remove one or more disk snapshots for a given unmanaged disk. In addition to the 5 parameters LoginName, SubscriptionName, StorageAccountName, ContainerName, and BlobName that this group of functions take, this function also takes 2 additional parameters; FromDate and ToDate. These 2 parameters allow us to delete snapshots taken duing a given time frame.

Example:

$ParameterList = @{
    LoginName = 'sam@dmain.com'
    SubscriptionName = 'my subscription name'
    StorageAccountName = 'storfluxwidget3vm'
    ContainerName = 'vhds'
    BlobName = 'Widget3VM-20181226-093810.vhd'
}
Remove-AzureRMUnmanagedDiskSnapshot @ParameterList

This deletes all disk snapshots for the provided unmanaged disk in the provided StorageAccount/Container

 


To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type

$PSVersionTable

To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type

Set-PSRepository -Name PSGallery -InstallationPolicy Trusted

To trust the Microsoft PowerShell Gallery repository, then

Install-Module AZSBTools,AzureRM -Force -AllowClobber

AZSBTools contains functions that depend on AzureRM modules, and they’re typically installed together.

To load the AZSBTools, and AzureRM modules type:

Import-Module AZSBTools,AzureRM -DisableNameChecking

To view a list of cmdlets/functions in AZSBTools, type

Get-Command -Module AZSBTools

To view the built-in help of one of the AZSBTools functions/cmdlets, type

help <function/cmdlet name> -show

such as

help New-SBAZServicePrincipal -show

Advertisements

Azure storage – features and pricing – June 2018


In the ever evolving Azure storage list of offerings, it may be hard to fully realize the available Azure storage offerings and their general cost structure at a given point in time. This post lays out a general summary of Azure storage offerings and costs, from the prospective of a consultant trying to make a recommendation to a large client as what storage type/options to use for which workload and why.

Storage account type

Classic storage

  • This is of type ‘Microsoft.ClassicStorage/storageAccounts’
  • This is considered legacy and should be migrated to GPv1 which provides backward compatibility to Azure classic (ASM) services

General purpose v1 (GPv1)

  • This is of type ‘Microsoft.Storage/storageAccounts’, kind ‘Storage’
  • Does not support Cool and Archive access tier attributes
  • Lower read/write transaction cost compared to GPv2
  • Can be used with Azure classic (ASM) services

General purpose v2 (GPv2)

  • This is of type ‘Microsoft.Storage/storageAccounts’, kind ‘StorageV2’
  • Supports Cool and Archive access tier attributes
  • Access Tier attribute (Hot/Cool) is exposed at the account level
  • Cannot be used with Azure classic (ASM) services
  • Compared to Blob Storage account: GPv2 charge for Cool early deletion but not Cool data writes (per GB)

Blob storage

  • This is of type ‘Microsoft.Storage/storageAccounts’, kind ‘BlobStorage’
  • This is a sub-type of GPv2 that supports only Block Blobs – not Page Blobs, Azure files, …
  • Minor pricing difference: charge for Cool data writes (per GB) but not Cool early deletion
  • Features similar to GPv2:
    • Supports Cool and Archive access tier attributes
    • Access Tier attribute (Hot/Cool) is exposed at the account level
    • Cannot be used with Azure classic (ASM) services

GPv1 and Blob Block accounts can be upgraded to a GPv2 account. This change cannot be reversed.

Set-AzureRmStorageAccount -ResourceGroupName <resource-group> -AccountName <storage-account> -UpgradeToStorageV2

Data Access Tiers

Data Access Tiers are Hot, Cool, and Archive. They represent 3 different physical storage platforms.

For the purpose of simplification, I will refer to LRS pricing in US East Azure region in US dollars for the next 450TB/month.

  • Data Access Tiers are supported in GPv2 accounts only (and the Block Blob storage sub-type) not GPv1
  • A blob cannot be read directly from the Archive tier. To read a blob in Archive, it must first be moved to Hot or Cool.
  • Data in Cool tier is subject to minimum stay period of 30 days. So, moving 1 TB of data to Cool tier for example, obligates the client to paying for that storage for 30 days at a minimum even if the data is moved or deleted before the end of the 30 days. This is implemented via the ‘Early deletion charge’ which is prorated.
  • Data in Archive tier is subject to minimum stay period of 180 days.

Hot

  • This is the regular standard tier where data can be routinely read and written
  • Storage cost is 2 cents per GB per month
  • Lowest read and write IO charges (5 cents per 10k write, 0.4 cents per 10k read)
  • No data retrieval charge (This is the cost to copy data to Hot tier from another tier before it can be read – already in Hot tier)

Cool

  • 25% cheaper storage cost down to 1.5 cents per GB per month
  • More costly read and write IO charges (10 cents per 10k write = 200% Hot, 1 cent per 10k read = 250% Hot)
  • 1 cent per GB data retrieval charge (this is the cost to copy the data to Hot tier, which is a required interim step to read data that resides in Cool tier)

Archive

  • 90% cheaper storage cost down to 0.2 cents per GB per month
  • Most costly read and write IO charges (10 cents per 10k write = 200% Hot, 5 dollars per 10k read = 125,000% Hot)
  • 2 cent per GB data retrieval charge (this is the cost to copy the data to Hot tier, which is a required interim step to read data that resides in Archive tier)
  • The archive tier can only be applied at the blob level, not at the storage account level
  • Blobs in the archive storage tier have several hours of latency (Is this tier using tape not disk!?)

Geo-replication

Geo-replication refers to automatically keeping a copy of a given Storage Account data in another Azure region that’s 400 miles or more away from the primary site. The primary Azure site is the Azure region where the Storage account resides and where the usual read/write transactions occur. The choice of a secondary site is a constant determined by Microsoft and is available if the client chooses the geo-replication feature of a Storage Account. The following is the list of the Azure region pairs as of 18 June 2018

Data in the secondary Azure region cannot be accessed. It is only used under the following conditions:

  • Microsoft declares a region-wide outage. For example East US. This is a Microsoft triggered – not client triggered event
  • Microsoft will first try to restore the service in that region. During that time data is not accessible for read or write. It’s unclear how long will Microsoft pursue this effort before moving to geo-failover.
  • Microsoft initiates geo-failover from the primary to secondary region. That’s all data of all tenants in the primary region.
    • This is essentially a DNS change to re-point the storage end points’ fully qualified domain names to the secondary region
    • The RPO (Recovery Point Objective) is 15 minutes. That’s to say up to 15 minutes worth of data may be lost.
    • The RTO (Recovery Time Objective) is unclear. That’s the time between primary site outage to data availability for read and write on the secondary site
    • At the end of the geo-failover, read/write access is restored (for geo-replicated storage accounts only of course)
    • At a later time, Microsoft will perform a geo-failback which is the same process in the reverse direction
  • This is a process that never happened before. No Azure data center ever sustained a complete loss.
  • It’s unclear when failback will be triggered, whether it will include down-time, or another 15 minute data loss.

A storage account can be configured in one of several choices with respect to geo-replication:

LRS (Locally redundant storage)

When a write request is sent to Azure Storage Account, the transaction is fully replicated on three different physical disks across three fault domains and upgrade domains inside the primary location, then success is returned back to the client.

GRS (Geo-redundant storage)

In addition to triple synchronous local writes, GRS adds triple asynchronous remote writes of each data block.  Data is asynchronously replicated to the secondary Azure site within 15 minutes.

ZRS (Zone-redundant storage)

ZRS is similar to LRS but it provides slightly more durability than LRS (12 9’s instead of 11 9’s for LRS over a given year).

It’s ony available to GPv2 accounts.

RA-GRS (Read access geo-redundant storage)

RA-GRS is similar to GRS but it provides read access to the data in the secondary Azure site.

 

 

 

 

 

 


Azure Blob Storage Scalability


The following is the current (September 2014) hierarchy of Azure Services with a focus on Azure Blob Storage:

store01This is where you can see your subscriptions, billing information, view and sign up for Preview features, and download usage reports.

store02

This a summary of Azure portals:

store04

  • You can have up to 50 100 (as of 12/2014) Storage accounts in an Azure Subscription. A storage account has a maximum capacity of 500TB. A Storage Account has the following properties: URL (name), location, replication:

store03

  • Pick a name for your storage account in the URL field. It must be lower case letters and numbers.
  • Pick a location from the drop down list of Azure data centers. If you’re going to use the files in this Storage Account with an Azure VM, pick a the same location where the VM resides. Location cannot be changed after a storage account is created. Moving files between Storage Accounts will incur costs.
  • Pick a replication level. Replication is done automatically by Microsoft among different Azure data centers. Locally redundant replication keeps 3 replicas of your data in a single data center. Geo-redundant replication keeps 6 copies in 2 geographically distant data centers. Zone redundant keeps 6 copies in 2 geographically clode data centers. Replication affects storage costs. Replication can be changed after Storage Account creation.

Within a Storage Account, we can create “Containers”. A Container is a similar to a folder. Within a container we can copy/delete/snapshot files. Files in an Azure Storage account can be copied as either:

  • Block Blobs: no file size limit, no snapshot feature
  • Page Blobs: a page blob file has a limit of 1TB (1023 GB). It has the benefit of snapshot feature. For example, Azure VM disks as saved as page blobs in Azure Storage.

In summary:

store05