src/pipeline/Invoke-ScoutDeclarativeCollector.ps1
|
#Requires -Version 7.0 Set-StrictMode -Version Latest $ErrorActionPreference = 'Stop' <# .SYNOPSIS Run one declarative collector definition for the Processing or Reporting task (AB#5657/AB#5659). .DESCRIPTION The one interpreter every converted `.psd1` collector definition shares. It contains no collector-specific knowledge: it does not know what a "retirement lookup" or a "tag" is, it just filters `$Resources`, runs the definition's own `Preamble` once per matched resource (verbatim source lifted from the original collector -- see `docs/design/decisions/declarative-collectors.md`), evaluates each `Fields` entry's expression in the scope that preamble just populated, and exports the result the same way `Invoke-ScoutCollector` would for a hand-written `.ps1`. PROCESSING builds one script per matched resource: <Preamble, verbatim> foreach ($<AdditionalRowLoop.Variable> in <AdditionalRowLoop.Source>) { # 0 or more, nested <AdditionalRowLoop.Preamble, verbatim> foreach ($<TagLoop.Variable> in <TagLoop.Source>) { # omitted when TagLoop is $null @{ '<Field.Name>' = <Field.Expression>; ... } if ($ResUCount -eq 1) { $ResUCount = 0 } } } run with `RowLoopVariable`, `SUB`, `Retirements` and `Unsupported` pre-bound via `ScriptBlock.InvokeWithContext` -- the same technique that keeps the audit's AST extraction (AB#5658) safe, applied here to execution instead of parsing. One execution per RESOURCE, not per row, matches the original collectors' own granularity; running Preamble once per emitted row instead would still be correct (it is a pure function of the resource) but wastefully re-derives the same locals for every tag. REPORTING rebuilds the `$Exc` column list, conditional-formatting rules and Excel style from the definition's `Export` section and calls `Export-Excel` exactly as the original collector's Reporting branch did. .PARAMETER Definition A definition object from `Get-ScoutCollectorDefinition`. .PARAMETER Context The same context hashtable `Invoke-ScoutCollector` accepts: Resources, Subscriptions, InTag, Retirements, Unsupported, Task, File, SmaResources, TableStyle. .OUTPUTS Processing: an array of row hashtables. Reporting: nothing (writes to Context.File). .NOTES Tracks ADO AB#5657/AB#5659/AB#5656 (Feature AB#5656, Epic AB#5638). THIS IS NOW THE LIVE PATH. Up to and including v2.9.0 this function ran only under `tests/DeclarativeCollectorEquivalence.Tests.ps1` -- 124 definitions existed and nothing executed them. `Invoke-ScoutCollector` routes to it for every collector that has a `.psd1`, so a defect here is a defect in a customer's report. The kill switch (`AZURESCOUT_FORCE_IMPERATIVE_COLLECTORS`) puts every collector back on its `.ps1`. #> function Invoke-ScoutDeclarativeCollector { [CmdletBinding()] Param( [Parameter(Mandatory, Position = 0)] [PSCustomObject]$Definition, [Parameter(Mandatory, Position = 1)] [hashtable]$Context ) if ($Context['Task'] -eq 'Processing') { return Invoke-ScoutDeclarativeProcessing -Definition $Definition -Context $Context } return Invoke-ScoutDeclarativeReporting -Definition $Definition -Context $Context } function Build-ScoutDeclarativeRowScript { <# Compose the per-resource script text described in the function help above. #> param([PSCustomObject]$Definition) # Expressions are emitted UNWRAPPED, exactly as the original collector's own `$obj = @{ ... }` # hashtable literal wrote them. Two separate reasons, each of which broke a real collector when # the first draft wrapped them in `( ... )`: # # * A field's source text is not always an EXPRESSION. `Containers/ARO.ps1` and # `Containers/ContainerRegistries.ps1` (among others) write a field as a multi-line # `if (...) { ... } else { ... }` STATEMENT. A hashtable literal accepts a statement as a # value; `( ... )` accepts only a pipeline, so `('Name' = (if (...) ...))` is a PARSE error # -- "The term 'if' is not recognized as a name of a cmdlet". Every field of every such # collector was unreachable. # * `$( ... )` is not a safe substitute either: a subexpression collects the output STREAM, # so `$(@(1))` is the scalar 1 and `$(@())` is `$null`, where the original produced a # one-element array and an empty array respectively. Silently changing the shape of an # array-valued field ('Zone' and friends) is exactly the class of difference the # equivalence proof exists to catch. # # Not wrapping at all is therefore both the compatible AND the faithful choice: the interpreter # reproduces the original hashtable literal, character for character, per field. $FieldLines = (@($Definition.Fields) | ForEach-Object { $QuotedName = $_.Name -replace "'", "''" " '$QuotedName' = $($_.Expression)" }) -join "`n" # Each extra loop may carry its OWN preamble -- the statements the original collector runs once # per item of the fanned-out collection, before it starts emitting rows for that item. Without # this, `Security/Vault.ps1` (which derives its three permission strings per access policy) and # `Networking/NATGateway.ps1` and `Networking/VirtualNetwork.ps1` (per subnet) silently produced # $null for every field computed inside the loop, where the original produced ''. The row-level # Preamble cannot cover it: those statements depend on the loop variable, which does not exist # yet when the row preamble runs. # The tag loop is the LAST entry of the loop nest, not a fixed frame around the row -- 25 # collectors have no tag loop at all and RouteTables calls its variable $TagKey, so both the # existence and the naming come from the definition. $Nest = [System.Collections.Generic.List[object]]::new() foreach ($Loop in @($Definition.AdditionalRowLoops)) { $Nest.Add($Loop) } if ($Definition.TagLoop) { $Nest.Add($Definition.TagLoop) } $OpenLoops = '' $CloseLoops = '' foreach ($Loop in $Nest) { $OpenLoops += "foreach (`$$($Loop.Variable) in $($Loop.Source)) {`n" # Get-ScoutCollectorDefinition normalises every entry to Variable/Source/Preamble, so the # key is always present here even when the original loop body had no setup statements. if (-not [string]::IsNullOrWhiteSpace($Loop.Preamble)) { $OpenLoops += "$($Loop.Preamble)`n" } $CloseLoops = "}`n" + $CloseLoops } @" $($Definition.Preamble) $OpenLoops @{ $FieldLines } if (`$ResUCount -eq 1) { `$ResUCount = 0 } $CloseLoops "@ } function Invoke-ScoutDeclarativeProcessing { [CmdletBinding()] Param( [Parameter(Mandatory)] [PSCustomObject]$Definition, [Parameter(Mandatory)] [hashtable]$Context ) Set-StrictMode -Off $ErrorActionPreference = 'Continue' $Resources = $Context['Resources'] $Sub = $Context['Subscriptions'] $Retirements = $Context['Retirements'] $Unsupported = $Context['Unsupported'] # ROW ORDER IS THE SHEET'S ORDER, and how a multi-type collector builds its resource set decides # it. There are two shapes in the estate and they are NOT interchangeable: # # 'Grouped' -- one filtered pass per type, appended: # $RedisCache = $Resources | Where-Object { $_.TYPE -eq 'microsoft.cache/redis' } # $RedisCache += $Resources | Where-Object { $_.TYPE -eq 'microsoft.cache/redisenterprise' } # Every redis row precedes every redisenterprise row however the two interleave # in $Resources. # 'SinglePass' -- ONE pass whose condition admits several types: # $arcSites = $Resources | Where-Object { $_.TYPE -in @('microsoft.azurestackhci/sites', ...) } # Rows come out in $Resources order, types interleaved. # # `Hybrid/ArcSites.ps1` is the second shape, and interpreting it as the first reordered its # worksheet -- caught by the equivalence proof, which is why the mode is declared per collector # instead of assumed. For a single-type collector the two are identical. $Matched = if ($Definition.ResourceTypeMatching -eq 'SinglePass') { @($Resources | Where-Object { @($Definition.ResourceTypes) -contains $_.TYPE }) } else { @(foreach ($Type in @($Definition.ResourceTypes)) { $Resources | Where-Object { $_.TYPE -eq $Type } }) } if ($Definition.AdditionalFilter) { # The filter may need locals the Processing branch set up before it -- `AI/AppliedAIServices.ps1` # tests `$appliedAIKinds -contains $_.KIND` against an array literal defined two lines earlier. # FilterPreamble carries those statements verbatim, prepended INSIDE the Where-Object block: # assignments emit nothing, so the block's only output is still the condition itself. $FilterText = if ([string]::IsNullOrWhiteSpace($Definition.FilterPreamble)) { $Definition.AdditionalFilter } else { "$($Definition.FilterPreamble)`n$($Definition.AdditionalFilter)" } $ExtraFilter = [scriptblock]::Create($FilterText) $Matched = @($Matched | Where-Object $ExtraFilter) } if (@($Matched).Count -eq 0) { return @() } # ONCE-PER-COLLECTOR SETUP (AB#5659). The 15 collectors the audit called 'CrossResourceJoin' # hoist one or more secondary `$X = $Resources | Where-Object { $_.TYPE -eq '<other type>' }` # passes above their row loop and correlate against them per resource. Those statements are # lifted verbatim into SetupPreamble and run HERE -- once -- rather than inside the row script, # so a 5,000-NIC estate does not re-filter all 5,000 public IPs 5,000 times. # # The declared names are recovered by appending a `Get-Variable` per name to the lifted source # and reading the PSVariable objects back out of the invocation's output stream: there is no # other way to see the scope InvokeWithContext created. A name the preamble never assigned # comes back missing and THROWS -- it does not bind $null and carry on, because a join variable # that quietly became $null is a whole column of blanks in a shipped report. $SetupBindings = [System.Collections.Generic.List[psvariable]]::new() if (-not [string]::IsNullOrWhiteSpace($Definition.SetupPreamble)) { # -ErrorAction Ignore, not SilentlyContinue: SilentlyContinue still appends to $Error, and # Invoke-ScoutProcessing's AB#402 error detection reads $Error. $Harvest = (@($Definition.SetupVariables) | ForEach-Object { "Get-Variable -Name '$($_ -replace "'", "''")' -ErrorAction Ignore" }) -join "`n" $SetupScript = [scriptblock]::Create("$($Definition.SetupPreamble)`n$Harvest") $SetupContext = [System.Collections.Generic.List[psvariable]]::new() $SetupContext.Add([psvariable]::new('Resources', $Resources)) $SetupContext.Add([psvariable]::new('SUB', $Sub)) $SetupContext.Add([psvariable]::new('Retirements', $Retirements)) $SetupContext.Add([psvariable]::new('Unsupported', $Unsupported)) $Produced = @($SetupScript.InvokeWithContext($null, $SetupContext)) $ByName = @{} foreach ($Item in $Produced) { # The lifted source may legitimately emit its own output (a bare expression statement), # so select the PSVariables out rather than assuming the stream is only the harvest. # # Tested with `-is` and used AS IS -- deliberately NOT unwrapped first. Items returned # through the output stream are PSObject-wrapped, but `.BaseObject` is not reachable by # plain member access on a wrapped object: member lookup resolves against the BASE type, # PSVariable has no BaseObject, and with StrictMode off the read returns $null. Every # harvested variable was silently discarded that way, which is worth spelling out # because the symptom was not a blank column -- the declared-name check below turned it # into a throw naming all 13 collectors at once, which is the only reason it took # minutes to find rather than a release. if ($Item -is [psvariable]) { $ByName[$Item.Name] = $Item } } $MissingSetup = @(@($Definition.SetupVariables) | Where-Object { -not $ByName.ContainsKey($_) }) if (@($MissingSetup).Count -gt 0) { throw "Collector definition '$($Definition.Path)' declares SetupVariables its SetupPreamble never assigned: $($MissingSetup -join ', ')." } foreach ($Name in @($Definition.SetupVariables)) { $SetupBindings.Add($ByName[$Name]) } } $RowScriptText = Build-ScoutDeclarativeRowScript -Definition $Definition $RowScriptBlock = [scriptblock]::Create($RowScriptText) $Rows = foreach ($Resource in $Matched) { $Variables = [System.Collections.Generic.List[psvariable]]::new() $Variables.Add([psvariable]::new($Definition.RowLoopVariable, $Resource)) $Variables.Add([psvariable]::new('SUB', $Sub)) $Variables.Add([psvariable]::new('Retirements', $Retirements)) $Variables.Add([psvariable]::new('Unsupported', $Unsupported)) # Bound unconditionally, matching the original collectors' own scope: Networking/NetworkWatchers # does its three sub-resource joins INSIDE the row loop rather than hoisting them, and lifting # those statements without $Resources in scope produced three empty columns. $Variables.Add([psvariable]::new('Resources', $Resources)) foreach ($Binding in $SetupBindings) { $Variables.Add($Binding) } try { $RowScriptBlock.InvokeWithContext($null, $Variables) } catch [System.Management.Automation.MethodInvocationException] { # InvokeWithContext is a .NET method call, so SOME of what the row script throws comes # back with an extra layer: `Exception calling "InvokeWithContext" with "2" # argument(s): "<the real message>"`. The imperative collector -- same failure, same # data -- reports the inner message on its own, and that text is what reaches the # customer's run log through Invoke-ScoutCollector's warning. Leaving the layer on # would change the wording of every contained-collector failure on the release that # switches paths, for no reason connected to what actually failed. # # Unwrap ONE layer, and only when the inner exception is a PowerShell RuntimeException # -- measured, that is exactly the case where the layer was added. The two shapes: # # [datetime]$null -> MethodInvocationException("...InvokeWithContext...") # -> RuntimeException("Cannot convert null...") <- relayed # 'x'.Substring(9) -> MethodInvocationException("Exception calling "Substring"...") # -> ArgumentOutOfRangeException(...) <- NOT relayed # # In the second, InvokeWithContext adds nothing: the MethodInvocationException IS the # script's own error and the imperative path reports it verbatim, so unwrapping there # would strip a layer the imperative path keeps -- swapping one message difference for # another. Typed rather than matched on the word "InvokeWithContext", which is a # localisable string. $Inner = $_.Exception.InnerException if ($Inner -is [System.Management.Automation.RuntimeException]) { throw $Inner } throw } } @($Rows) } function Invoke-ScoutDeclarativeReporting { [CmdletBinding()] Param( [Parameter(Mandatory)] [PSCustomObject]$Definition, [Parameter(Mandatory)] [hashtable]$Context ) Set-StrictMode -Off $ErrorActionPreference = 'Continue' $SmaResources = $Context['SmaResources'] if (-not $SmaResources -or @($SmaResources).Count -eq 0) { return } $InTag = $Context['InTag'] $File = $Context['File'] $TableStyle = $Context['TableStyle'] $Export = $Definition.Export $RowUnits = @($SmaResources | ForEach-Object { $_.'Resource U' } | Measure-Object -Sum).Sum $TableName = "$($Export.TableNamePrefix)$RowUnits" $Style = New-ExcelStyle -HorizontalAlignment Center -AutoSize -NumberFormat $Export.NumberFormat $ConditionalText = @(foreach ($Line in @($Export.ConditionalText)) { & ([scriptblock]::Create($Line)) }) # Column order is the sheet's contract. The original collectors build $Exc by calling # $Exc.Add(...) in source order with the two Tag columns added inside `if ($InTag)` -- which # is very often NOT at the end (all 13 Databases collectors add 'Resource U' afterwards). So # the tag block is INSERTED at the recorded position, not appended. $Columns = [System.Collections.Generic.List[string]]::new() foreach ($Col in @($Export.Columns)) { $Columns.Add($Col) } if ($InTag) { $InsertAt = $Columns.Count if (-not [string]::IsNullOrWhiteSpace($Export.TagColumnsBefore)) { $Found = $Columns.IndexOf($Export.TagColumnsBefore) if ($Found -ge 0) { $InsertAt = $Found } } foreach ($Col in @($Export.TagColumns)) { $Columns.Insert($InsertAt, $Col) $InsertAt++ } } $ExportParams = @{ Path = $File WorksheetName = $Export.WorksheetName AutoSize = $true MaxAutoSizeRows = 100 TableName = $TableName TableStyle = $TableStyle Style = $Style } if (@($ConditionalText).Count -gt 0) { $ExportParams['ConditionalText'] = $ConditionalText } $SmaResources | ForEach-Object { [PSCustomObject]$_ } | Select-Object $Columns | Export-Excel @ExportParams } |