Private/Core/Invoke-MainProcessingLogic.ps1
|
#Requires -Version 5.1 <# .SYNOPSIS Core SID discovery and processing logic for Find-UnknownSID .DESCRIPTION Handles the main processing workflow including AD object discovery, SID analysis, and optional removal operations. Focused solely on the core business logic of orphaned SID processing. .NOTES Author: Jeffrey Stuhr Blog: https://www.techbyjeff.net LinkedIn: https://www.linkedin.com/in/jeffrey-stuhr-034214aa/ Last Updated: 2025-07-04 Version: 2.0.0 TROUBLESHOOTING: - For processing failures: .\Troubleshooting\Common\SID-Processing-Orchestration-Issues.md - For AD connectivity: .\Troubleshooting\Common\Common-Issues.md #> function Invoke-MainProcessingLogic { <# .SYNOPSIS Execute the main SID discovery and processing logic .DESCRIPTION Orchestrates the main processing workflow including AD object discovery, SID analysis, and optional removal operations with comprehensive error handling and performance monitoring. BUSINESS VALUE: - Identifies orphaned SIDs that pose security risks - Enables cleanup of obsolete access control entries - Provides comprehensive audit trail for compliance - Supports large-scale AD security remediation - Delivers actionable security intelligence PROCESSING FEATURES: - Parallel processing for improved performance - Memory-efficient streaming results management - Comprehensive retry logic for resilient operations - Real-time progress reporting and monitoring - Integration with backup systems for safe removal .PARAMETER SearchBase [String[]] (Mandatory) Distinguished Names of search bases for processing. Each DN must be valid and accessible in Active Directory. VALIDATION RULES: - Must pass Test-ValidDistinguishedName validation - Each DN must exist in Active Directory - User must have read permissions to the specified containers BUSINESS CONTEXT: Defines the scope of orphaned SID discovery operation. Consider organizational structure when selecting search bases. .PARAMETER Remove [Switch] (Optional) Whether to perform SID removal operations. Requires additional permissions and triggers backup creation. SECURITY IMPLICATIONS: - Requires elevated privileges for ACL modification - Should be tested thoroughly in non-production environment - Backup creation is strongly recommended .PARAMETER IncludeInherited [Switch] (Optional) Whether to include inherited ACEs in processing. Affects scope and performance of SID analysis. PERFORMANCE IMPACT: Including inherited ACEs significantly increases processing time but provides more comprehensive security analysis. .PARAMETER MaxRetries [Int] (Optional) Retries after the first attempt for each directory read during discovery and each SID lookup during detection, when the failure looks transient (Default: 3). Passed to Get-ADObjectsSequential and Find-OrphanedSIDsInObject. .PARAMETER BackupPath [String] (Optional) Directory path for storing ACL backups. Required for removal operations in production environments. .PARAMETER CorrelationId [String] (Optional) Correlation ID for tracking this operation. Auto-generated if not provided for audit purposes. .OUTPUTS [Hashtable] Processing results containing: - TotalObjectsProcessed: Number of AD objects analyzed - OrphanedSIDsFound: Count of orphaned SIDs discovered - UnverifiedSIDsFound: Count of SIDs a failed lookup left unverified (never removed) - ProcessingErrors: Number of errors encountered - ProcessingDuration: Total execution time - ObjectsPerSecond: Processing performance metric - PeakMemoryUsageMB: Maximum memory consumption - StreamingSummary: Results summary from streaming manager - RemovalResults: Details of removal operations (if performed) - StreamingManager: Reference for accessing detailed results .EXAMPLE PS> Invoke-MainProcessingLogic -SearchBase "OU=Users,DC=contoso,DC=com" DESCRIPTION: Execute discovery mode on Users OU OUTPUT: Hashtable with discovered orphaned SIDs and statistics USE CASE: Standard orphaned SID discovery operation DURATION: Varies based on object count (typically 1-5 seconds per 100 objects) .EXAMPLE PS> Invoke-MainProcessingLogic -SearchBase @("OU=Users,DC=contoso,DC=com") -Remove -BackupPath "C:\Backups" DESCRIPTION: Execute removal operation with backup creation OUTPUT: Results including removal statistics and backup location BUSINESS CASE: Production cleanup of orphaned SIDs with safety measures COMPLIANCE: Supports change management and audit requirements .EXAMPLE PS> Invoke-MainProcessingLogic -SearchBase "DC=contoso,DC=com" -IncludeInherited DESCRIPTION: Comprehensive domain-wide analysis with inherited permissions OUTPUT: Complete security analysis results for entire domain ENTERPRISE USE: Large-scale security assessment and remediation PERFORMANCE: Optimized for reliable sequential processing .NOTES Author: Jeffrey Stuhr Blog: https://www.techbyjeff.net LinkedIn: https://www.linkedin.com/in/jeffrey-stuhr-034214aa/ SECURITY CONSIDERATIONS: - Removal operations require elevated AD permissions - Backup creation is essential for production safety - All operations are logged for security audit - ShouldProcess support enables safe testing with -WhatIf PERFORMANCE CHARACTERISTICS: - Processing rate: 50-200 objects per second (depending on complexity) - Memory usage: Optimized with streaming results management - Parallel processing scales with available system resources - Progress reporting every 50 objects for user feedback TROUBLESHOOTING: - For processing failures: .\Troubleshooting\Common\SID-Processing-Orchestration-Issues.md - For AD connectivity: .\Troubleshooting\Common\Common-Issues.md - For memory issues: .\Troubleshooting\Performance\Memory-Management.md KNOWN LIMITATIONS: - Large datasets may require memory management tuning - Network latency affects processing speed - Some ACL operations may require domain administrator privileges - Streaming results are temporarily stored on disk #> [CmdletBinding(SupportsShouldProcess)] [OutputType([hashtable])] param( [Parameter(Mandatory)] [ValidateScript({ Test-ValidDistinguishedName $_ })] [string[]]$SearchBase, [Parameter()] [switch]$Remove, [Parameter()] [switch]$IncludeInherited, [Parameter()] [ValidateRange(1, 10)] [int]$MaxRetries = 3, [Parameter()] [string]$BackupPath, [Parameter()] [string]$CorrelationId = [System.Guid]::NewGuid().ToString() ) begin { Write-StructuredLog "Starting main processing logic" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId # StreamingResultsManager class is loaded via SecureClassImporter in main script # Verify it's available before proceeding if (-not ([System.Management.Automation.PSTypeName]'StreamingResultsManager').Type) { throw "StreamingResultsManager class not available - ensure SecureClassImporter loaded all classes successfully" } # Initialize streaming results manager instead of in-memory arrays $tempDir = Join-Path ([System.IO.Path]::GetTempPath()) "OrphanedSid-$CorrelationId" # -WhatIf on the public command arrives here as $WhatIfPreference. Read it directly: asking # ShouldProcess would print a "What if" line about the temp directory, which is not what the # caller is rehearsing. $whatIfMode = [bool]$WhatIfPreference # Batch size comes from the BatchSize configuration key (default 50) $batchSize = 50 if ($script:Config -and $script:Config.BatchSize -ge 1) { $batchSize = [int]$script:Config.BatchSize } # The domain SID lets Get-SIDAnalysis tell a SID from this domain apart from a foreign or # unknown one. Discovery still works without it; every domain SID is then rated Unknown Domain. $script:CurrentDomainSID = $null try { $script:CurrentDomainSID = (Get-ADDomain -ErrorAction Stop).DomainSID.Value } catch { Write-StructuredLog "Could not determine the current domain SID: $($_.Exception.Message)" -Level Warning -Component 'MainProcessing' -CorrelationId $CorrelationId } # Create StreamingResultsManager with 2-argument constructor (doesn't create directory yet) $script:StreamingResults = [StreamingResultsManager]::new($tempDir, $batchSize) # Configure WhatIf mode and complete initialization (this will handle directory creation properly) $script:StreamingResults.ConfigureWhatIfMode($whatIfMode) $script:RemovalResults = @() Write-StructuredLog "Initialized streaming results manager with temp directory: $tempDir (WhatIf: $whatIfMode)" -Level Debug -Component 'MainProcessing' -CorrelationId $CorrelationId } process { try { # Process each search base foreach ($searchPath in $SearchBase) { Write-StructuredLog "Processing search base: $searchPath" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId # Log security audit for AD discovery operation attempt Write-ADOperationSecurityLog -OperationName 'AD-ObjectDiscovery' -Outcome 'Attempt' -SecurityContext @{ SearchBase = $searchPath Operation = 'Get-ADObjectsSequential' Mode = if ($Remove) { 'RemovalMode' } else { 'DiscoveryMode' } } -CorrelationId $CorrelationId # Always perform discovery phase - ShouldProcess only applies to removal operations # Get AD objects for processing using refactored sequential processing try { $adObjects = Get-ADObjectsSequential -SearchBase @($searchPath) -MaxRetries $MaxRetries -CorrelationId $CorrelationId # Log successful AD discovery Write-ADOperationSecurityLog -OperationName 'AD-ObjectDiscovery' -Outcome 'Success' -SecurityContext @{ SearchBase = $searchPath ObjectsFound = @($adObjects).Count Operation = 'Get-ADObjectsSequential' } -CorrelationId $CorrelationId } catch { # Log failed AD discovery Write-ADOperationSecurityLog -OperationName 'AD-ObjectDiscovery' -Outcome 'Failure' -SecurityContext @{ SearchBase = $searchPath ErrorMessage = $_.Exception.Message Operation = 'Get-ADObjectsSequential' } -CorrelationId $CorrelationId throw } # Ensure $adObjects is always treated as an array for consistent counting $objectArray = @($adObjects) # Categorize objects for meaningful user feedback $objectStats = @{ Users = 0 Computers = 0 Groups = 0 OUs = 0 Other = 0 } foreach ($obj in $objectArray) { switch ($obj.ObjectClass) { 'user' { $objectStats.Users++ } 'computer' { $objectStats.Computers++ } 'group' { $objectStats.Groups++ } 'organizationalUnit' { $objectStats.OUs++ } default { $objectStats.Other++ } } } # Report meaningful discovery summary $summaryParts = @() if ($objectStats.Users -gt 0) { $summaryParts += "$($objectStats.Users) users" } if ($objectStats.Computers -gt 0) { $summaryParts += "$($objectStats.Computers) computers" } if ($objectStats.Groups -gt 0) { $summaryParts += "$($objectStats.Groups) groups" } if ($objectStats.OUs -gt 0) { $summaryParts += "$($objectStats.OUs) OUs" } if ($objectStats.Other -gt 0) { $summaryParts += "$($objectStats.Other) other objects" } $summaryText = if ($summaryParts.Count -gt 0) { $summaryParts -join ', ' } else { "0 objects" } Write-StructuredLog "Discovered: $summaryText (Total: $($objectArray.Count) objects)" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId Write-StructuredLog "Found $($objectArray.Count) AD objects to process" -Level Debug -Component 'MainProcessing' -CorrelationId $CorrelationId $script:Statistics.TotalObjects += $objectArray.Count # Process objects for orphaned SIDs - this always happens regardless of WhatIf $processedCount = 0 $orphanedFoundCount = 0 $null = $objectArray | ForEach-Object { # Captured once: inside a catch block $_ is the error record, not the object $currentObject = $_ try { # Pinned: housekeeping, not the operation -WhatIf rehearses Invoke-MemoryCheck -MemoryManager $script:MemoryManager -WhatIf:$false -CorrelationId $CorrelationId $script:Statistics.ProcessedObjects++ $processedCount++ # Show progress every 50 objects or on last object if (($processedCount % 50 -eq 0) -or ($processedCount -eq $objectArray.Count)) { $percentComplete = [Math]::Round(($processedCount / $objectArray.Count) * 100, 1) Write-StructuredLog "Processing: $processedCount/$($objectArray.Count) objects ($percentComplete%) - Orphaned SIDs found: $orphanedFoundCount" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId } # Active Directory returns an object without nTSecurityDescriptor when the # account may not read it. Such an object cannot be examined; say so at # Warning level and count it, so a clean run is not mistaken for a clean # directory. if (-not $currentObject.nTSecurityDescriptor) { $script:Statistics.UnexaminedObjects++ Write-StructuredLog "Not examined: the security descriptor of $($currentObject.DistinguishedName) could not be read, usually because the running account lacks permission to read it" -Level Warning -Component 'MainProcessing' -CorrelationId $CorrelationId return } # -MaxRetries applies to each directory lookup inside detection, where a # transient failure happens. Detection itself never throws: a failed rule is # logged and skipped, and a lookup that still fails gives an Unverified # result. Removal below runs once; a second attempt would take a second # backup and act on an ACL the first attempt may already have changed. $orphanedResults = @(Find-OrphanedSIDsInObject -ADObject $currentObject -IncludeInherited:$IncludeInherited -CurrentDomainSID $script:CurrentDomainSID -MaxRetries $MaxRetries -CorrelationId $CorrelationId) # An Unverified result is a SID that no lookup found but at least one lookup # failed to check. It is reported, counted apart, and never removed. $verifiedResults = @($orphanedResults | Where-Object { $_.VerificationStatus -ne 'Unverified' }) $unverifiedResults = @($orphanedResults | Where-Object { $_.VerificationStatus -eq 'Unverified' }) # Log security audit for orphaned SID analysis if ($orphanedResults.Count -gt 0) { Write-ADOperationSecurityLog -OperationName 'OrphanedSID-Detection' -Outcome 'Success' -SecurityContext @{ ObjectDN = $currentObject.DistinguishedName ObjectClass = $currentObject.ObjectClass OrphanedSIDsFound = $verifiedResults.Count OrphanedSIDs = ($verifiedResults | ForEach-Object { $_.OrphanedSID }) -join ',' UnverifiedSIDsFound = $unverifiedResults.Count UnverifiedSIDs = ($unverifiedResults | ForEach-Object { $_.OrphanedSID }) -join ',' } -CorrelationId $CorrelationId } # Additional memory management for large operations if ($processedCount % 100 -eq 0) { # Use centralized cleanup for large operations # Pinned: housekeeping, not the operation -WhatIf rehearses $cleanupResult = Invoke-Cleanup -CorrelationId $CorrelationId -WhatIf:$false Write-Verbose "Memory usage at $processedCount objects: $($cleanupResult.MemoryAfterMB) MB (Freed: $($cleanupResult.MemoryFreedMB) MB)" } if ($orphanedResults.Count -gt 0) { $script:Statistics.OrphanedSIDsFound += $verifiedResults.Count $script:Statistics.UnverifiedSIDsFound += $unverifiedResults.Count $orphanedFoundCount += $verifiedResults.Count # Stream results to disk instead of accumulating in memory. Unverified # results go too, marked by VerificationStatus, so they can be re-checked. foreach ($orphanedResult in $orphanedResults) { $script:StreamingResults.AddResult($orphanedResult) } if ($Remove) { foreach ($unverifiedResult in $unverifiedResults) { Write-StructuredLog "Not removing unverified SID $($unverifiedResult.OrphanedSID) from $($unverifiedResult.ObjectDN): a directory lookup failed, so it may belong to a live principal ($($unverifiedResult.VerificationError))" -Level Warning -Component 'MainProcessing' -CorrelationId $CorrelationId } # One removal per object with all of its verified orphaned SIDs, so the # object is backed up once, before any entry is removed. Removing them one # call at a time backed it up again after each removal, and a restore from # the newest backup could not bring back the SIDs removed first. # An inherited entry is set on a parent object and can only be removed # there; on this object it is reported, not removed or backed up. $inheritedOnly = @($verifiedResults | Where-Object { $_.IsInherited } | ForEach-Object { $_.OrphanedSID } | Select-Object -Unique | Where-Object { $_ -notin @($verifiedResults | Where-Object { -not $_.IsInherited } | ForEach-Object { $_.OrphanedSID }) }) foreach ($inheritedSid in $inheritedOnly) { Write-StructuredLog "Not removing inherited entry for orphaned SID $inheritedSid on $($currentObject.DistinguishedName): it is inherited from a parent object, so remove it there" -Level Warning -Component 'MainProcessing' -CorrelationId $CorrelationId } $sidsToRemove = @($verifiedResults | Where-Object { -not $_.IsInherited } | ForEach-Object { $_.OrphanedSID } | Select-Object -Unique) if ($sidsToRemove.Count -gt 0) { $objectDN = $currentObject.DistinguishedName $sidList = $sidsToRemove -join ', ' $target = "$objectDN - SIDs: $sidList" if ($PSCmdlet.ShouldProcess($target, "Remove Orphaned SIDs")) { Write-StructuredLog "Removing orphaned SIDs: $sidList from $objectDN" -Level Verbose -Component 'MainProcessing' -CorrelationId $CorrelationId # Log security audit for SID removal attempt Write-ADOperationSecurityLog -OperationName 'OrphanedSID-Removal' -Outcome 'Attempt' -SecurityContext @{ ObjectDN = $objectDN OrphanedSIDs = $sidList BackupPath = $BackupPath Operation = 'Remove-OrphanedSidEntry' } -CorrelationId $CorrelationId $removalParams = @{ ObjectDN = $objectDN OrphanedSIDs = $sidsToRemove CorrelationId = $CorrelationId } if ($BackupPath) { $removalParams.BackupPath = $BackupPath } try { $removalResult = Remove-OrphanedSidEntry @removalParams $script:RemovalResults += $removalResult # A failed removal comes back as a result, not an exception, # so the outcome is read from it $auditContext = @{ ObjectDN = $objectDN OrphanedSIDs = $sidList ActualRemovals = $removalResult.ActualRemovals FailedRemovals = $removalResult.FailedRemovals BackupCreated = [bool]$removalResult.BackupFile BackupFile = $removalResult.BackupFile } $auditOutcome = 'Success' if (-not $removalResult.Success) { $auditOutcome = 'Failure' $auditContext.ErrorMessage = $removalResult.ErrorMessage } Write-ADOperationSecurityLog -OperationName 'OrphanedSID-Removal' -Outcome $auditOutcome -SecurityContext $auditContext -CorrelationId $CorrelationId } catch { # Log failed SID removal Write-ADOperationSecurityLog -OperationName 'OrphanedSID-Removal' -Outcome 'Failure' -SecurityContext @{ ObjectDN = $objectDN OrphanedSIDs = $sidList ErrorMessage = $_.Exception.Message } -CorrelationId $CorrelationId throw } } else { Write-StructuredLog "WhatIf: Would remove orphaned SIDs: $sidList from $objectDN" -Level Verbose -Component 'MainProcessing' -CorrelationId $CorrelationId # Log WhatIf simulation for audit trail Write-ADOperationSecurityLog -OperationName 'OrphanedSID-Removal' -Outcome 'Attempt' -SecurityContext @{ ObjectDN = $objectDN OrphanedSIDs = $sidList SimulationMode = $true WhatIfMode = $true } -CorrelationId $CorrelationId # The preview is the same type a real removal returns: nothing # was removed, and RemovedSIDs lists what would have been $previewResult = [RemovalOperationResult]::new() $previewResult.ObjectDN = $objectDN $previewResult.IntendedRemovals = $sidsToRemove.Count $previewResult.ActualRemovals = 0 $previewResult.RemovedSIDs = $sidsToRemove $previewResult.Success = $true $previewResult.CorrelationId = $CorrelationId $previewResult.SecurityValidation = 'WhatIf - Would Remove' $script:RemovalResults += $previewResult } } } return $orphanedResults } } catch { # Detection retries are exhausted, or removal failed. Either way the object # is counted as an error and the run moves on to the next one. $script:Statistics.ProcessingErrors++ Write-StructuredLog "Error processing object $($currentObject.DistinguishedName): $($_.Exception.Message)" -Level Error -Component 'MainProcessing' -CorrelationId $CorrelationId } } # Show completion summary for this search base Write-StructuredLog "Completed $searchPath - Objects: $($objectArray.Count), Orphaned SIDs: $orphanedFoundCount" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId } # Complete statistics $script:Statistics.Complete() # Get summary from streaming results manager $streamingSummary = $script:StreamingResults.GetSummary() # Generate processing summary with streaming results $processingResults = @{ TotalObjectsProcessed = $script:Statistics.ProcessedObjects OrphanedSIDsFound = $script:Statistics.OrphanedSIDsFound UnverifiedSIDsFound = $script:Statistics.UnverifiedSIDsFound UnexaminedObjects = $script:Statistics.UnexaminedObjects ProcessingErrors = $script:Statistics.ProcessingErrors ProcessingDuration = $script:Statistics.Duration ObjectsPerSecond = $script:Statistics.ObjectsPerSecond PeakMemoryUsageMB = $script:MemoryManager.GetPeakMemoryUsage() StreamingSummary = $streamingSummary RemovalResults = $script:RemovalResults CorrelationId = $CorrelationId # Store reference to streaming manager for later access StreamingManager = $script:StreamingResults } Write-StructuredLog "Main processing completed successfully" -Level Information -Component 'MainProcessing' -CorrelationId $CorrelationId return $processingResults } catch { $script:Statistics.CriticalErrors++ Write-StructuredLog "Critical error in main processing: $($_.Exception.Message)" -Level Error -Component 'MainProcessing' -CorrelationId $CorrelationId throw } } } |