Match-LargeArrays function added to AZSBTools PowerShell module
Sometimes one needs to match large arrays in PowerShell based on a common property. For example, matching Active Directory (AD) users to Azure users based on AD user’s ms-ds-ConsistencyGuid versus Azure user’s ImmutableId. The ImmutableId Azure user property is not actually immutable and can be modified. It is populated with the base 64 version of the corresponding AD user’s ObjectGuid, and is typically populated by the ADConnect software which uses a stripped down version of MIM.
Consider the following demo data:
$DesiredArraySize = 1000
$MatchPerCent = 93.7 # The sample data generated will have x% matching records
#region Generate Demo arrays
$DemoADUserList = $DemoAzureUserList = @()
$MatchRecordCount = [Math]::Round($MatchPerCent * $DesiredArraySize / 100)
foreach ($LoopCounter in (1..$DesiredArraySize)) {
$Guid = New-Guid
$EmployeeId = Get-Random -Minimum 1000000000 -Maximum 9999999999
$DemoADUserList += New-Object -TypeName PSObject -Property ([Ordered]@{
GivenName = 'Sam'
SurName = $Guid
DisplayName = "Sam $Guid"
Name = "Sam $Guid"
samAccountName = "Sam$Guid"
UserPrincipalName = "sam$Guid@mydomain.com"
Mail = "sam$Guid@mydomain.com"
EmployeeId = $EmployeeId
Enabled = $true
DistinguishedName = "CN=Sam $Guid,OU=US,DC=mydomain,DC=com"
'msDS-CloudExtensionAttribute5' = $EmployeeId
ObjectGuid = $Guid
'ms-ds-ConsistencyGuid' = Convert-ObjectGuid2ImmutableId -ObjectGuid $Guid
})
if ($LoopCounter -le $MatchRecordCount) {$Id = $Guid} else {$Id = New-Guid}
$DemoAzureUserList += New-Object -TypeName PSObject -Property ([Ordered]@{
GivenName = 'Sam'
SurName = $Id
DisplayName = "Sam $Id"
Mail = "sam$Id@mydomain.com"
UserPrincipalName = "sam$Id@mydomain.onmicrosoft.com"
AccountEnabled = $true
ObjectId = New-Guid
ImmutableId = Convert-ObjectGuid2ImmutableId -ObjectGuid $Id
CreationType = $null
UserState = $null
UserType = 'Member'
})
}
#endregion
This code generates two array: $DemoADUserList and $DemoAzureUserList. A record in the DemoADUserList looks like:
While a record in the DemoAzureUserList looks like:
Traditional matching algorithm may look like:
#region Match using the traditional method
$Duration = Measure-Command {
foreach ($ADUser in $DemoADUserList) {
if ($FoundInAzure = $DemoAzureUserList | where ImmutableId -EQ $ADUser.'ms-ds-ConsistencyGuid') {
$ADUser | Add-Member -MemberType NoteProperty -Name MatchingAzureObjectId -Value $FoundInAzure.ObjectId -Force
} else {
$ADUser | Add-Member -MemberType NoteProperty -Name MatchingAzureObjectId -Value 'Not Found in Azure' -Force
}
}
}
Write-log 'Using the traditional method','matched',('{0:N0}' -f $DesiredArraySize),'lists',"($('{0:N0}' -f ($DesiredArraySize*$DesiredArraySize)) records) in",
"$($Duration.Hours):$($Duration.Minutes):$($Duration.Seconds)",'hh:mm:ss',"($('{0:N0}' -f $Duration.TotalSeconds) seconds)" Yellow,Green,Cyan,Green,Cyan,Green,Cyan,DarkYellow
Write-Log ' Identified',('{0:N0}' -f ($DemoADUserList | where MatchingAzureObjectId -ne 'Not Found in Azure').Count),'matching records' Green,Cyan,Green
#endregion
The problem with traditional matching algorithm is that it takes a very long time for large data sets. For example, for 10k records data sets, this algorithm takes ~ 30 minutes. For 200k data sets, it takes over a week!!
The new Match-LargeArrays function leverages Hashtable indexing to reduce that matching time by upwards of 50,000% or 500 folds!!
$Result = Match-LargeArrays -Array1 $DemoADUserList -Property1 'ms-ds-ConsistencyGuid' -Array2 $DemoAzureUserList -Property2 'ImmutableId'
For a 200k data sets the matching time is reduced from over a week to under 2 minutes!!
The result for 10k data sets takes under 2 seconds:
Using the traditional matching algorithm with the same 10k data sets and on the same hardware takes 17 minutes and 31 seconds or over 586 times longer!!
The $Result is the function’s returned array. It is a copy of the input Array1 with an additional property “MatchingObject” that contains the matching record(s) from Array2. For example:
To use/update the AZSBTools PowerShell module which is available in the PowerShell Gallery, you can use the following code:
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 # PowerShellGallery dropped Ssl3 and Tls as of 1 April 2020 Remove-Module AZSBTools -Force -EA 0 Install-Module AZSBTools -Force -AllowClobber -SkipPublisherCheck # -Scope CurrentUser Import-Module AZSBTools -DisableNameChecking -Force Get-Command -Module AZSBTools
You need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help Get-DayOfMonth -show
Remove-AzureUserProxyAddresses function added to AZSBTools PowerShell module
Many organizations use ADConnect to replicate/synchronize some/all of their Active Directory users and/or computers to their Azure directory. A great deal of transformation occurs to objects as they get replicated from AD to Azure. The schema of the two databases is quite different although some object attributes carry the same names. For example, the ‘user’ object in AD has ‘GivenName’,’SurName’, and ‘DisplayName’ attributes in common with the ‘person’ object in Azure directory. 
One of the user AD attributes is proxyaddresses. Proxyaddresses is a multivalued attribute that is used on users, groups and contacts in order to facilitate mail delivery. It is subject to the following guidelines:
- The primary (sending) mail alias must be prefixed with upper case “SMTP:”.
- Only one value/alias is allowed to have the upper case “SMTP:” prefix.
- Secondary mail aliases must be prefixed with lower case “smtp:”.
- No duplicate values (across all AD objects) are allowed.
In Azure directory, the ‘person’ object also has a proxyaddresses property. However, Azure person proxyaddresses property is a calculated property. Microsoft uses the complex logic described in this article to calculate the Azure person proxyaddresses.
In large organizations with frequent mergers and acquisitions, it’s not uncommon for the AD user proxyaddresses attribute to change over time showing several additions and removals of ‘smtp:’ addresses. ADConnect and the logic Microsoft use to calculate the Azure person proxyaddresses fails to remove ‘smtp:’ addresses that have been removed from the AD user proxyaddresses attribute. This can manifest as end user problems such as failure to login to OneDrive for business, SharePoint Online sites, and the like.
The process to remove unwanted ‘smtp:’ addresses from the Azure person proxyaddresses is as follows:
- On the on-premises ADConnect server stop the ADSync Scheduler as in:
Set-ADSyncScheduler -SyncCycleEnabled $false - Soft delete the Azure person object using Remove-Msoluser PowerShell cmdlet.
- Create a temporary Azure person object for each smtp: address you wish removed, using New-AzureADUser
- Populate the temporary Azure person object proxyaddresses property. One way to do that is to assign it an O365 license using Set-MsolUserLicense which requires setting the person’s ‘usagelocation’ property using Set-AzureADUser cmdlet.
- Restore the deleted user with the cmdlet Restore-MsolUser and the -AutoReconcileProxyConflicts switch.
- Remove the temporary Azure user(s) created during this process.
- Finally, enable the ADSync Scheduler on the ADConnect server using:
Set-ADSyncScheduler -SyncCycleEnabled $true
The new Remove-AzureUserProxyAddresses function of the AZSBTools PowerShell module automates this process. It takes one mandatory parameter; being the samAccountName of the AD user. It does not reach out to the ADConnect server and disable/enable the ADSync Scheduler. You’ll need to do that separately.
To use/update the AZSBTools PowerShell module which is available in the PowerShell Gallery, you can use the following code:
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# PowerShellGallery dropped Ssl3 and Tls as of 1 April 2020
Remove-Module AZSBTools -Force -EA 0
Install-Module AZSBTools -Force -AllowClobber -SkipPublisherCheck # -Scope CurrentUser
Import-Module AZSBTools -DisableNameChecking -Force
Get-Command -Module AZSBTools
You need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help Get-DayOfMonth -show
Get-SBADUser function added to AZSBTools PowerShell module
Get-SBADUser function has been added to the AZSBTools PowerShell module to provide details on Active Directory user objects. This comes in handy when you need to list AD users but do not have Active Directory PowerShell module or do not have the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- Other than console output, the function will return no output if the provided group does not exist
- If a user samaccountname is specified as a parameter the function will return output similar to:
- If the function is used without any parameters, it will return information on all AD users in the current domain
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
Get-SBADGroupMembers function added to AZSBTools PowerShell module
Get-SBADGroupMembers function has been added to the AZSBTools PowerShell module to provide member list information for Active Directory group objects including members of sub-groups. This function does not depend on or require Active Directory PowerShell module or the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- The function returns output similar to:
So this function’s emphasis is not on the provided group information such as it’s DN (Distinguished Name), OU (Organizational Unit), … Group properties can be obtained via the Get-SBADGroup function. The emphasis of Get-SBADGroupMembers is on a group’s member users, and whether a user is a direct member of the given group, or a member of a subgroup.
The ‘MemberOf’ field provides that visibility by listing the group hierarchy of each member user separated by dots. In the example above, testuser2 is member of testgroup2.testgroup1 which indicates that he’s a member of testgroup2 AD group which is a member of testgroup1 AD group. In the same example above, testuser1 is a direct member of testgroup1 AD group.
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
Get-SBADGroup function added to AZSBTools PowerShell module
Get-SBADGroup function has been added to the AZSBTools PowerShell module to provide details on Active Directory group objects including its members. This comes in handy when you need to list AD group members but do not have Active Directory PowerShell module or do not have the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- The function will return no output if the provided group does not exist
- If a group is specified as a parameter the function will return output similar to:
- If the function is used without any parameters, it will return information on all AD groups in the current domain:
To see group members including sub-groups use the Get-SBADGroupMembers function.
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
PowerShell module with functions to Get AD FSMO roles, Get and Set NTP server setting
This module can be downloaded from the Microsoft Script Center Repository.
The module does not require ActiveDirectory PS module, and includes 3 functions:
Get-ADRole: This is a function to return one or all DC FSMO role holders in the current AD forest
Get-NTPDCs: This is a function to return NTP server settings for one or all DCs in current AD forest
Set-NTP: This is a function to change NTP server settings for one or more DCs in current AD forest
Example:
$DCNames = ((([DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()).Sites).Servers).Name
$DCNames -notmatch (Get-ADRole PdcRole).DCName | % { Set-NTP $_ time-c.nist.gov }
This example will change NTP server setting on all DCs except PDC emulator to ‘time-c.nist.gov’
Of course for this to work, it needs to to be run under an AD user that has permission to write to the DCs’ registry.
Getting started:
- To use the functions/cmdlets in this module, download this module, extract the files using Winrar for example.
- Open PowerShell ISE as Administrator
- Open and execute the Install-Module.ps1 script
Powershell script to provide a PS Credential object, saving password securely
Have you ever been in the situation where you need to execute a cmdlet like
Disable-ADAccount -Identity ‘Someone’ -Server ‘MyDomainController’
To disable a user account, but it fails because your account does not have permission to disable users?
You can use another account that have permissions to disable users by using the -Credential parameter of the Disable-ADAccount cmdlet as in
Disable-ADAccount -Identity ‘Someone’ -Server ‘MyDomainController’ -Credential (Get-Credential)
The Get-Credential cmdlet prompts for a user name and password, which is fine if you need to run it once or a few times. However, we often come across situations where we need to use several credentials to automate tasks in Active Directory, Exchange, SharePoint,… You will rarely have a single account that has permission to do all these tasks, or across multiple directories. In an automation script, the Get-SBCredntial function can make this easy.
Here’s an example:
$SourceADCred = Get-SBCredential 'domain1\MyADAdmin' $TargetADCred = Get-SBCredential 'domain2\MyADAdmin' $ExCred = Get-SBCredential 'domain1\MyExchangeAdmin' Disable-ADAccount -Identity 'Someone' -Server 'MyDomainController1' -Credential $SourceADCred Disable-ADAccount -Identity 'Sometwo' -Server 'MyDomainController2' -Credential $TargetADCred Get-Mailbox -Identity 'someone@domain.com' -Credential $ExCred
Sam's Corner 