Match-LargeArrays function added to AZSBTools PowerShell module
Sometimes one needs to match large arrays in PowerShell based on a common property. For example, matching Active Directory (AD) users to Azure users based on AD user’s ms-ds-ConsistencyGuid versus Azure user’s ImmutableId. The ImmutableId Azure user property is not actually immutable and can be modified. It is populated with the base 64 version of the corresponding AD user’s ObjectGuid, and is typically populated by the ADConnect software which uses a stripped down version of MIM.
Consider the following demo data:
$DesiredArraySize = 1000
$MatchPerCent = 93.7 # The sample data generated will have x% matching records
#region Generate Demo arrays
$DemoADUserList = $DemoAzureUserList = @()
$MatchRecordCount = [Math]::Round($MatchPerCent * $DesiredArraySize / 100)
foreach ($LoopCounter in (1..$DesiredArraySize)) {
$Guid = New-Guid
$EmployeeId = Get-Random -Minimum 1000000000 -Maximum 9999999999
$DemoADUserList += New-Object -TypeName PSObject -Property ([Ordered]@{
GivenName = 'Sam'
SurName = $Guid
DisplayName = "Sam $Guid"
Name = "Sam $Guid"
samAccountName = "Sam$Guid"
UserPrincipalName = "sam$Guid@mydomain.com"
Mail = "sam$Guid@mydomain.com"
EmployeeId = $EmployeeId
Enabled = $true
DistinguishedName = "CN=Sam $Guid,OU=US,DC=mydomain,DC=com"
'msDS-CloudExtensionAttribute5' = $EmployeeId
ObjectGuid = $Guid
'ms-ds-ConsistencyGuid' = Convert-ObjectGuid2ImmutableId -ObjectGuid $Guid
})
if ($LoopCounter -le $MatchRecordCount) {$Id = $Guid} else {$Id = New-Guid}
$DemoAzureUserList += New-Object -TypeName PSObject -Property ([Ordered]@{
GivenName = 'Sam'
SurName = $Id
DisplayName = "Sam $Id"
Mail = "sam$Id@mydomain.com"
UserPrincipalName = "sam$Id@mydomain.onmicrosoft.com"
AccountEnabled = $true
ObjectId = New-Guid
ImmutableId = Convert-ObjectGuid2ImmutableId -ObjectGuid $Id
CreationType = $null
UserState = $null
UserType = 'Member'
})
}
#endregion
This code generates two array: $DemoADUserList and $DemoAzureUserList. A record in the DemoADUserList looks like:
While a record in the DemoAzureUserList looks like:
Traditional matching algorithm may look like:
#region Match using the traditional method
$Duration = Measure-Command {
foreach ($ADUser in $DemoADUserList) {
if ($FoundInAzure = $DemoAzureUserList | where ImmutableId -EQ $ADUser.'ms-ds-ConsistencyGuid') {
$ADUser | Add-Member -MemberType NoteProperty -Name MatchingAzureObjectId -Value $FoundInAzure.ObjectId -Force
} else {
$ADUser | Add-Member -MemberType NoteProperty -Name MatchingAzureObjectId -Value 'Not Found in Azure' -Force
}
}
}
Write-log 'Using the traditional method','matched',('{0:N0}' -f $DesiredArraySize),'lists',"($('{0:N0}' -f ($DesiredArraySize*$DesiredArraySize)) records) in",
"$($Duration.Hours):$($Duration.Minutes):$($Duration.Seconds)",'hh:mm:ss',"($('{0:N0}' -f $Duration.TotalSeconds) seconds)" Yellow,Green,Cyan,Green,Cyan,Green,Cyan,DarkYellow
Write-Log ' Identified',('{0:N0}' -f ($DemoADUserList | where MatchingAzureObjectId -ne 'Not Found in Azure').Count),'matching records' Green,Cyan,Green
#endregion
The problem with traditional matching algorithm is that it takes a very long time for large data sets. For example, for 10k records data sets, this algorithm takes ~ 30 minutes. For 200k data sets, it takes over a week!!
The new Match-LargeArrays function leverages Hashtable indexing to reduce that matching time by upwards of 50,000% or 500 folds!!
$Result = Match-LargeArrays -Array1 $DemoADUserList -Property1 'ms-ds-ConsistencyGuid' -Array2 $DemoAzureUserList -Property2 'ImmutableId'
For a 200k data sets the matching time is reduced from over a week to under 2 minutes!!
The result for 10k data sets takes under 2 seconds:
Using the traditional matching algorithm with the same 10k data sets and on the same hardware takes 17 minutes and 31 seconds or over 586 times longer!!
The $Result is the function’s returned array. It is a copy of the input Array1 with an additional property “MatchingObject” that contains the matching record(s) from Array2. For example:
To use/update the AZSBTools PowerShell module which is available in the PowerShell Gallery, you can use the following code:
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 # PowerShellGallery dropped Ssl3 and Tls as of 1 April 2020 Remove-Module AZSBTools -Force -EA 0 Install-Module AZSBTools -Force -AllowClobber -SkipPublisherCheck # -Scope CurrentUser Import-Module AZSBTools -DisableNameChecking -Force Get-Command -Module AZSBTools
You need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help Get-DayOfMonth -show
Get-SBADUser function added to AZSBTools PowerShell module
Get-SBADUser function has been added to the AZSBTools PowerShell module to provide details on Active Directory user objects. This comes in handy when you need to list AD users but do not have Active Directory PowerShell module or do not have the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- Other than console output, the function will return no output if the provided group does not exist
- If a user samaccountname is specified as a parameter the function will return output similar to:
- If the function is used without any parameters, it will return information on all AD users in the current domain
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
Get-SBADGroupMembers function added to AZSBTools PowerShell module
Get-SBADGroupMembers function has been added to the AZSBTools PowerShell module to provide member list information for Active Directory group objects including members of sub-groups. This function does not depend on or require Active Directory PowerShell module or the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- The function returns output similar to:
So this function’s emphasis is not on the provided group information such as it’s DN (Distinguished Name), OU (Organizational Unit), … Group properties can be obtained via the Get-SBADGroup function. The emphasis of Get-SBADGroupMembers is on a group’s member users, and whether a user is a direct member of the given group, or a member of a subgroup.
The ‘MemberOf’ field provides that visibility by listing the group hierarchy of each member user separated by dots. In the example above, testuser2 is member of testgroup2.testgroup1 which indicates that he’s a member of testgroup2 AD group which is a member of testgroup1 AD group. In the same example above, testuser1 is a direct member of testgroup1 AD group.
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
Get-SBADGroup function added to AZSBTools PowerShell module
Get-SBADGroup function has been added to the AZSBTools PowerShell module to provide details on Active Directory group objects including its members. This comes in handy when you need to list AD group members but do not have Active Directory PowerShell module or do not have the necessary permissions to login to a Domain Controller.
- This function must be run from a domain-joined computer
- This function does not require or depend on the Active Directory PowerShell module
- This function does not require permission/rights to login or connect to a Domain Controller
- The function will return no output if the provided group does not exist
- If a group is specified as a parameter the function will return output similar to:
- If the function is used without any parameters, it will return information on all AD groups in the current domain:
To see group members including sub-groups use the Get-SBADGroupMembers function.
To use the AZSBTools PowerShell module which is available in the PowerShell Gallery, you need PowerShell 5. To view your PowerShell version, in an elevated PowerShell ISE window type
$PSVersionTable
To download and install the latest version of AZSBTools from the PowerShell Gallery and its dependencies, type
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
To trust the Microsoft PowerShell Gallery repository, then
Install-Module AZSBTools,Az -Force -AllowClobber -Scope CurrentUser
AZSBTools contains functions that depend on Az module, and they’re typically installed together.
To load the AZSBTools, and Az modules type:
Import-Module AZSBTools,Az -DisableNameChecking
To view a list of cmdlets/functions in AZSBTools, type
Get-Command -Module AZSBTools
To view the built-in help of one of the AZSBTools functions/cmdlets, type
help <function/cmdlet name> -show
such as
help New-SBAZServicePrincipal -show
PowerShell module with functions to Get AD FSMO roles, Get and Set NTP server setting
This module can be downloaded from the Microsoft Script Center Repository.
The module does not require ActiveDirectory PS module, and includes 3 functions:
Get-ADRole: This is a function to return one or all DC FSMO role holders in the current AD forest
Get-NTPDCs: This is a function to return NTP server settings for one or all DCs in current AD forest
Set-NTP: This is a function to change NTP server settings for one or more DCs in current AD forest
Example:
$DCNames = ((([DirectoryServices.ActiveDirectory.Forest]::GetCurrentForest()).Sites).Servers).Name
$DCNames -notmatch (Get-ADRole PdcRole).DCName | % { Set-NTP $_ time-c.nist.gov }
This example will change NTP server setting on all DCs except PDC emulator to ‘time-c.nist.gov’
Of course for this to work, it needs to to be run under an AD user that has permission to write to the DCs’ registry.
Getting started:
- To use the functions/cmdlets in this module, download this module, extract the files using Winrar for example.
- Open PowerShell ISE as Administrator
- Open and execute the Install-Module.ps1 script
Using Powershell to get list of all Domain Controllers with static IPs and email it as CSV file attachment
This is an update of a prior post. It adds the functionality of exporting the output to CSV format that can be opened in Excel, and emailing it out.
# Script to list domain controllers with static IP addresses
# Sam Boutros
# 6/13/2014 – Rev 1
# 6/16/2014 – Rev 2 – Write output to CSV file, email it out
#
$MailRelay = “192.168.xxx,yyy” # Put the FQDN or IP address of your SMTP relay server here
$SMTPPort = 25 # Replace port number if needed
$Sender = “Sam Boutros <abcd@bcde.com>” # Edit this line to update the email sender
$Recipients = “Recipient1 <name1@domain2.com>”, “R2 <name3@domain4.com>” # Put email recipients here
$EmailSubject = “List of domain controllers with static IPv4 addresses”
$EmailBody = “Please see attached list of domain controllers with static IPv4 addresses”
# End data entry section
#
$Loc = Get-Location
$Date = Get-Date -format yyyyMMdd_hhmmsstt
$CSVfile = $Loc.path + “\DCsWithStaticIPs_” + $Date + “.csv”
write-output “Listing all domain controllers with static IPs:”
$DCs = Get-ADDomainController -Filter *
write-output ‘DC Name,Static IPv4’ | out-file -Filepath $CSVfile -append -encoding ASCII
foreach ($DC in $DCs.Hostname)
{
$ints = Get-WmiObject -ComputerName $DC -Query “select IPAddress from Win32_NetworkAdapterConfiguration where IPEnabled=TRUE and DHCPEnabled=FALSE”
ForEach ($int in $ints.IPAddress)
{
if ($int.Contains(“.”)) # This is IPv4, not IPv6
{
write-output “DC Name: $DC, Static IPv4: $int”
$csvline = $DC + ‘,’ + $int
write-output $CSVline | out-file -Filepath $CSVfile -append -encoding ASCII}
}
}
send-mailmessage -from $Sender -to $Recipients -subject $EmailSubject -body $EmailBody -Attachments $CSVfile -smtpServer $MailRelay -Port $SMTPPort
Sam's Corner 