Archive for November, 2015

StorSimple 8k Device Software Version 1.2 (17584)


In this post I will go over the changes in Powershell cmdlets made available in StorSimple Software update 1.2 (17584)

To connect to StorSimple Powershell you can use the script in this post.

First, I identify the software version running on the device by using the Get-HcsSystem cmdlet:

Version-1

Next, I get a list of available cmdlets:

Commands-1.2

As noted in prior posts, there are many more cmdlets in the Powershell Core, Utility, Management, HCS, and other PS modules. However, Microsoft has locked down the available cmdlets on this device.

Next, using this script I collect the commands’ information in an XML file.

65 cmdlets are exposed in StorSimple Update 1.2, compared to 58 in Update 0.2.

Comparing v0.2 to v1.2 shows 7 new cmdlets:

Compare-1.2b

See this link for more information on HCS cmdlets, and this link for new features in StorSimple Software Update 1.2

 

 

Advertisements


StorSimple 8k series software upgrade from version 0.2 to version 1.2 – phase 3 of 3


Originally published on 9 November, 2015

See upgrade process overview and phase 1 and phase 2 for important warnings and pre-requisites.

Phase 3 – SAS controller update:

We must allow 24 hours after Firmware update 1.2 before proceeding to phase 3 according to Microsoft. This phase takes about 30 minutes and is non-disruptive (no down time).

  • From a computer running Windows 8.1 or 2012 R2 or Windows 10, run this Powershell script.
  • The script will verify that StorSimple Software Update v1.2 is successfully installed from phase 1
    Update-38
  • Next, the script will download the bits
    The bits are available for download from KB3043005 by browsing to http://catalog.update.microsoft.com and searching for StorSimple. However, this requires Internet Explorer browser and a number of add-ons. In the script I’ve saved the bits to my OneDrive and the script downloads them from there.
  • Finally, the script will run the Start-HcsHotFix cmdlet with the necessary paramters to start the SAS controller update. It will display an initial update status similar to:Update-39

The current controller will reboot. Then StorSimple will apply the SAS update to the other controller automatically and reboot it as well. This phase is a non-disruptive process. iSCSI volumes will remain up and no down time is anticipated. This image shows the controller rebooting and the iSCSI volume data still available to client computers:Update-40

During the update, the Azure Management Interface shows an update job in progress:Update-41


Monitoring StorSimple Backups


Currently (10 November, 2015) StorSimple 8k provides email alerts regarding a range of error conditions. Email Alerts can be configured per device in the Azure Management Interface under the Device/Configure screen:

Monitor-StorSimple03

However, the Azure Interface does not currently provide for notification regarding successful backup jobs. Some clients have requested daily email notification of StorSimple backup jobs for both failed and successful jobs. This script does just that.

The top section of the script has the input region where you should enter the information specific to your environment:Monitor-StorSimple04

In the Azure Management Interface, under the StorSimple Manager/Dashboard page, you can see the Subscription Name, Subscription ID, and StorSimple Manager Name. Monitor-StorSimple05

On the same page, click Registration Key at the bottom center to copy it.Monitor-StorSimple06

The TargetDeviceName is your StorSimple Device Name as seen on the StorSimple Manager/Devices page:

Monitor-StorSimple07

‘Duration’ variable in the script is set to 7 (days) by default. This will report on backups that ran in the past 7 days from the script execution time.

‘LogFile’ and ‘HTMLFile’ variables should point to local or network paths. That’s where the script will create its log and HTML files (output). If/when this script is set to run as a scheduled job or task, you need to make sure that it runs under credentials that have read/write access to these 2 paths.

‘EmailSender’ is what the email notification would appear to come from. This may or may not be an existing email account.

‘EmailRecipients’ is where you can add the email addresses of who should receive this report. You can add more than one recipient, each on a line.

‘SMTPServer’ is your email relay server. You should configure your SMTP relay server to allow relay from the computer running the script. Details will depend on your environment specific email relay requirements.

You need to run this script in Powershell as Administrator NOT Powershell ISE.

Finally, setup a scheduled job or task to run this script daily or as frequently as needed.

Sample script output:

Monitor-StorSimple08

Sample email:

Monitor-StorSimple09


StorSimple 8k series software upgrade from version 0.2 to version 1.2 – phase 2 of 3


Originally published on 9 November, 2015

Phase 2 – Firmware update

See upgrade process overview and phase 1 for important warnings and pre-requisites.

In this phase we need to enter ‘Maintenance Mode’. As noted in the process overview post, the device iSCSI volumes will be inaccessible during this process. It is recommended that this update be scheduled in a planned downtime window. In maintenance mode, Powershell Remoting is not available and we must enter Powershell cmdlets directly in the device serial interface.

This phase takes about 1 hour.

In this phase we will:

  • From a computer running Windows 8.1 or 2012 R2 or Windows 10, run this Powershell script. This will guide you through the steps of this phase.
  • Verify that StorSimple Software Update v1.2 is successfully installed from phase 1
    Run this cmdlet in the device Serial Interface:
    Get-HcsSystem
    You should see the following versions:
    HcsSoftwareVersion: 6.3.9600.17584
    CisAgentVersion: 1.0.9049.0
    MdsAgentVersion: 26.0.4696.1433Update-23
  • Enter Maintenance Mode
    Run this cmdlet in the device Serial Interface:
    Enter-HcsMaintenanceMode
    The device will reboot
    In Azure Management Interface, under the device/configure page, you can also see both controllers in Maintenance modeUpdate-18
  • Download the bits
    The bits are available for download from KB3063416 by browsing to http://catalog.update.microsoft.com and searching for StorSimple. However, this requires Internet Explorer browser and a number of add-ons. In the script I’ve saved the bits to my OneDrive and the script downloads them from there. Update-24This is the part of the script that downloads the bits. By default, it retries 3 times.
  • Update the firmware
    By running the Start-HcsHotFix cmdlet in the device serial interface.
    The script provides the exact cmdlet and its parameters as inUpdate-25Copy the green text from the script and paste it in your device serial interface window. You will be prompted to enter the password for the listed userUpdate-19
  • Switch to the peer controller and update its firmware
    By selecting option ‘2’ and running the Start-HcsHotFix cmdlet
    Update-26
  • Exit Maintenance Mode
    By running the Exit-HcsMaintenanceMode cmdlet at the device serial interface window
    The device will reboot several timesUpdate-21

We must allow 24 hours after Firmware update 1.2 before proceeding to phase 3 according to Microsoft.

The Azure Management Interface will show an informational alert similar to: Update-35

 


StorSimple 8k series software upgrade from version 0.2 to version 1.2 – phase 1 of 3


Originally published on 9 November, 2015

See upgrade process overview for important warnings and pre-requisites.

Phase 1 – Software update:

This phase takes about 1 hour. In this phase we will:

  • Establish remote Powershell to the StorSimple device using this Powershell script. You should see output similar to Update-30
  • Download the bits
    The bits are available for download from KB3063418 by browsing to http://catalog.update.microsoft.com and searching for StorSimple. However, this requires Internet Explorer browser and a number of add-ons. In the script I’ve saved the bits to my OneDrive and the script downloads them from there. Update-31
  • Update the device software
    This runs the Start-HcsHotFix cmdlet on the device. The update will apply to the Active Controller which will reboot, then the other controller will be automatically updated as well.
    Allow about 25 minutes for the process to complete.
    Expect to see email alerts as the Controllers are rebooted and patched.

 


StorSimple 8k series software upgrade from version 0.2 to version 1.2 – Process Overview


Executive summary:

Update-42


From time to time Microsoft releases software updates to StorSimple 8k series Hybrid Storage devices. Most devices that shipped during 2014 and 2015 came with software version 0.2 – kernel version 6.3.9600.17361. This can be viewed in the Azure Management Interface under the device dashboard:

Update-12

The typical Microsoft recommendation here is to click Install Updates in the Azure Management Interface under the Device Maintenance page. Update-13

Due to ‘known issues’ this method has been observed to fail. Furthermore, it may cause the Active Controller to go into Recovery mode which requires a Support call with Microsoft to bring it back online.

One alternative is to use the Powershell to update the device software by manually installing the required hot fixes as outlined in the Install Update 1.2 on your StorSimple device document.


Warning:

  • This update requires down time during which the device’s iSCSI volumes are not available to file servers or client computers. It’s recommended that this be scheduled in a planned down time window.
  • You will need access to the device serial interface during this process.

The update process is as follows (in this order):

  1. Install StorSimple Software Update v1.2
  2. Install StorSimple Firmware Update 1.2
  3. Install StorSimple SAS Controller (LSI) Update 1.2

Pre-requisites:

  • Access to one of the device two serial interfaces. Preferably both of them.
  • Access to the device via Powershell Remoting.
  • Disable Data 1 interface in the Azure Management interface prior to starting the process, and enable it back when the update is complete. This is to avoid a known issue. This can be done in the Azure Management Interface at the device/configure page
    Update-28
  • It’s recommended to offline all volumes at the file servers and perform a cloud snapshot of each volume prior to starting the upgrade process.
  • Plan for a 2-4 hour down time window depending on how long it would take you to do the cloud snapshots.

 

 


StorSimple 8k series – troubleshooting ‘Controller Off’ issue


This post has originally been published on 4 November, 2015

You may encounter a situation with StorSimple where one of the controllers shows as ‘off’ in the Azure Management Interface.

Off-01

By default, all Service Administrators and co-administrators of the Azure Subscription where the device is managed from will receive email alerts in case of this and similar error conditions.

The cause of this error is typically a severe error that requires Microsoft Support intervention such as failed software update.Off-09

Attempting to bring the ‘off’ controller back online by changing controller settings in Azure does not work:

Off-03

Azure StorSimple Manager issues the ‘on’ command:

Off-05

But the controller remains ‘off’

The reason is that the ‘off’ controller is ‘unreachable’. This can also be seen when running the Get-HcsSystem Powershell cmdlet on the device either via the serial interface or via Powershell remoting.

Off-04

The 2 controllers in the StorSimple 8k device are 2 computers running Windows 2012 R2 ‘Storage Edition’ which seems to be a Core-only version stripped down from unneeded Windows components and has the HCS (Hybrid Cloud Storage) Powershell module. It appears to be locked down via JEA exposing a very small subset of Powershell cmdlets. The 2 controllers (computers) are setup in a failover cluster. In this case, the controller is actually ‘on’ and you can see lights on its ports on the back of the device. It’s however not reachable by the cluster. The repair will hinge on bringing the cluster back in sync.

I also tried to reboot the active controller (Restart-HCSController -Force) to initiate a failover to the passive controller, hoping that might bring it back online, but that did not work. The Active controller was rebooted but the ‘off’ controller remained ‘off’. In normal operation, rebooting the active controller will initiate a controller failover where the Passive controller becomes Active while the controller that was Active is rebooting. This should be a seamless no-down-time process.

Connecting to Controller 1 via the serial interface showed it’s status as ‘Recovery’:

Off-07

Get-HCSSystem on Controller 1 shows:

Off-08

Since we don’t have access to the Powershell cmdlets that we need to bring the cluster back in sync, we have to open a support ticket with Microsoft via this link. You’ll get a confirmation message similar to:

Off-06

In 2015 Microsoft have made significant improvements to StorSimple support. I typically get a call back from a knowledgeable engineer within 30 minutes.

Note:

We will need access to Controller 1 serial interface to run the necessary Powershell cmdlets to bring it back online. This is exactly why it’s a recommended best practice to have remote access to the serial interfaces of BOTH controllers at all times. A device like Lantronix EDS2100 2-Port Secure Device Server – CY1733 would do the job.

To fix this issue you simply need to run the Exit-HCSRecoveryMode cmdlet on Controller 1 which is in Recovery mode. After the controller reboots, all should be back in order.

Unfortunately, the Exit-HCSRecoveryMode cmdlet is only made available to Microsoft support. To access it, you need to run Enable-HCSSupport cmdlet, provide the encryption key to MS support on the phone/remote session, they would decrypt it and give you a password. Run the Enter-HCSSupport cmdlet and enter the provided password to gain access to all available Powershell cmdlets including the Exit-HCSRecoveryMode cmdlet that we need in this case.