Modules/businessdev.ALbuild.RuntimePackages/Public/Get-BcRuntimeWorkSet.ps1
|
function Get-BcRuntimeWorkSet { <# .SYNOPSIS Plans outstanding runtime-package work, grouped by BC platform version across all products. .DESCRIPTION Turns "which (product, app version, platform version) runtime packages are still missing?" into an ordered, sliceable plan. It is a pure function - no feed, blob or artifact calls - so the whole planning policy is unit-testable and a run can be inspected before a single container starts. WHY GROUPING BY PLATFORM VERSION IS THE WHOLE POINT A runtime package is produced server-side by a running BC service tier, and Microsoft only guarantees it on the exact platform version it was produced on - so the (product x platform) matrix is irreducible. What IS reducible is the container count. Measured on build 27197, provisioning a container costs 238-564 s depending on the major, while the value-adding work per product is a constant ~80 s. Iterating platform-versions-per-product, as one pipeline per product does, pays that provisioning cost once per product; iterating products-per-platform-version pays it once, full stop. For a 12-product catalogue over ~192 versions that is ~2 300 container starts against 192. It also removes a failure mode rather than working around it: several products depend on other products in the same catalogue (Extension License, 365 business API). Per-product pipelines had to fetch those dependencies' runtime packages from blob storage and SKIP the platform version when they were not there yet. Built in one container in dependency order, the dependency is simply present - it was built moments earlier, in the same container. LANES A product release invalidates the entire matrix at once, and a full rebuild is hours of work no matter how it is scheduled. Customers, however, pull the newest majors first. -FastLaneMinMajor splits the plan so those versions are built and published first, and the long tail follows behind without holding the release up. SLICES Each slice is the unit of work for one agent job. -MaxVersionsPerSlice bounds how long a single job runs, which matters because a job holds one of the organisation's few parallel slots for its whole lifetime - and because a job that dies loses only its in-flight version, not the run. .PARAMETER Product The catalogue, as objects with: Name - display name (used in logs and the per-app result). AppId - app.json 'id'; keys the blob layout <appId>/<platformVersion>/<file>. AppVersion - the released version to build runtime packages for. MinimumVersion - app.json 'application'; platform versions below it are not applicable. Compared as a whole version, so a '17.1.0.0' minimum really does exclude 17.0.x. DependsOn - OPTIONAL names of other catalogue products this one needs installed first. Projects - OPTIONAL, passed through untouched for the worker. Country - OPTIONAL, passed through untouched. .PARAMETER PlatformArtifact Candidate platform versions as objects with PlatformVersion, ArtifactUrl and Country. Country is part of the grouping key, not decoration: a BC artifact carries ONE localisation, so a container built from the 'de' artifact cannot produce a runtime package for a product that ships against 'w1'. In the current catalogue Banking, ERiC and Sanction Screen build against 'de' and the rest against 'w1', so a platform version yields one container per country in use - still one per country instead of one per product. .PARAMETER ExistingPackage Relative blob paths that already exist ('<appId>/<platformVersion>/<file>.app'). A (product, platform version) pair whose file is already there is reported as skipped rather than rebuilt, which is what makes a re-run after a failure cheap and a scheduled sweep close to free. .PARAMETER SkipPlatformVersion Platform versions to exclude outright - Microsoft occasionally publishes an artifact that cannot produce a working container, and one bad version must not stall the catalogue. .PARAMETER FastLaneMinMajor Platform majors >= this go into the FastLane; everything else into the LongTail. 0 (default) disables the split and puts everything in a single 'All' lane. .PARAMETER Lane Which lane to return: All (default), FastLane or LongTail. .PARAMETER MaxVersionsPerSlice Maximum platform versions per slice. Default 12. .EXAMPLE $plan = Get-BcRuntimeWorkSet -Product $catalogue -PlatformArtifact $artifacts ` -ExistingPackage $blobNames -FastLaneMinMajor 27 -Lane FastLane $plan.Slices | ForEach-Object { Invoke-BcRuntimeFactory -WorkItem $_.Items ... } .OUTPUTS PSCustomObject with Slices (Index, Lane, Items[]) and Summary (per-product counts). #> [CmdletBinding()] [OutputType([PSCustomObject])] param( [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [object[]] $Product, [Parameter(Mandatory)] [ValidateNotNull()] [object[]] $PlatformArtifact, [string[]] $ExistingPackage = @(), [string[]] $SkipPlatformVersion = @(), [ValidateRange(0, [int]::MaxValue)] [int] $FastLaneMinMajor = 0, [ValidateSet('All', 'FastLane', 'LongTail')] [string] $Lane = 'All', [ValidateRange(1, [int]::MaxValue)] [int] $MaxVersionsPerSlice = 12 ) $ordered = Get-BcRuntimeProductOrder -Product $Product # Case-insensitive: blob listings and app.json ids differ in casing across products. $existing = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) foreach ($p in $ExistingPackage) { if (-not [string]::IsNullOrWhiteSpace($p)) { [void]$existing.Add(($p -replace '\\', '/').Trim('/')) } } $skip = [System.Collections.Generic.HashSet[string]]::new([System.StringComparer]::OrdinalIgnoreCase) foreach ($s in $SkipPlatformVersion) { if (-not [string]::IsNullOrWhiteSpace($s)) { [void]$skip.Add($s.Trim()) } } # Newest platform version first, in both lanes: whichever lane a version lands in, the versions # customers are most likely to ask for should exist soonest. $candidates = @($PlatformArtifact | Where-Object { -not $skip.Contains("$($_.PlatformVersion)") } | Sort-Object -Property @{ Expression = { [version](ConvertTo-BcVersion $_.PlatformVersion) }; Descending = $true }) $summary = [System.Collections.Generic.List[object]]::new() $counts = @{} foreach ($p in $ordered) { $counts[$p.Name] = [PSCustomObject]@{ Name = $p.Name; Planned = 0; SkippedExisting = 0; NotApplicable = 0 } } $items = [System.Collections.Generic.List[object]]::new() foreach ($artifact in $candidates) { $pv = "$($artifact.PlatformVersion)" $artifactCountry = "$(Get-BcRuntimeProperty -InputObject $artifact -Name 'Country' -Default '')" $pvMajor = ([version](ConvertTo-BcVersion $pv)).Major # $itemLane, NOT $lane: PowerShell variable names are case-insensitive, so a local '$lane' IS the # '$Lane' parameter. Assigning it would overwrite the requested lane on the first iteration and # the filter below would then always match - the fast lane would silently build the whole matrix. $itemLane = if ($FastLaneMinMajor -le 0) { 'All' } elseif ($pvMajor -ge $FastLaneMinMajor) { 'FastLane' } else { 'LongTail' } if ($Lane -ne 'All' -and $itemLane -ne $Lane) { continue } $pending = [System.Collections.Generic.List[object]]::new() foreach ($p in $ordered) { # A BC artifact carries one localisation. Only products built against THIS artifact's country # can be produced in a container started from it - the rest get their own container from # their own country's artifact. $productCountry = "$(Get-BcRuntimeProperty -InputObject $p -Name 'Country' -Default '')" if ($artifactCountry -and $productCountry -and $productCountry -ne $artifactCountry) { continue } # Compared as a WHOLE version, not just the major. app.json 'application' is a minimum # platform version and its minor part carries real meaning: run 27339 spent a container # and a red app on Extension License at 17.0.16993.0, because its source calls # Environment Information.IsSaaSInfrastructure - which the BC standard source shows # arriving in BC17.1, not 17.0. A major-only test cannot express that, and raising the # minimum to 18 to work around it would throw away BC17.1 through 17.5, where the app # compiles perfectly well. # # A minimum with zeros in the last two parts (the normal '17.1.0.0' shape) therefore # admits every 17.1 build and rejects every 17.0 one. A minimum that names a full build # is honoured literally - that is what the manifest asked for, and the plan's 'n/a' count # makes an over-tight one visible. $minimum = "$(Get-BcRuntimeProperty -InputObject $p -Name 'MinimumVersion' -Default '')" if (-not [string]::IsNullOrWhiteSpace($minimum)) { $minVersion = [version](ConvertTo-BcVersion $minimum) if (([version](ConvertTo-BcVersion $pv)) -lt $minVersion) { $counts[$p.Name].NotApplicable++; continue } } $publisher = "$(Get-BcRuntimeProperty -InputObject $p -Name 'Publisher' -Default '')" $file = Get-BcRuntimePackageFileName -Publisher $publisher -Name $p.Name -Version "$($p.AppVersion)" $key = "$($p.AppId)/$pv/$file" if ($existing.Contains($key)) { $counts[$p.Name].SkippedExisting++; continue } $counts[$p.Name].Planned++ $pending.Add([PSCustomObject]@{ Name = $p.Name Publisher = $publisher AppId = $p.AppId AppVersion = "$($p.AppVersion)" Country = Get-BcRuntimeProperty -InputObject $p -Name 'Country' -Default '' Projects = Get-BcRuntimeProperty -InputObject $p -Name 'Projects' -Default @() # Transitive chain, in install order. The worker installs exactly this before the # product and tears it down again afterwards. DependencyChain = @(Get-BcRuntimeDependencyChain -Product $ordered -Name $p.Name) TargetPath = "$($p.AppId)/$pv" FileName = $file }) } # A platform version with nothing pending is not worth a container. if ($pending.Count -eq 0) { continue } $items.Add([PSCustomObject]@{ PlatformVersion = $pv Country = $artifactCountry ArtifactUrl = $artifact.ArtifactUrl Lane = $itemLane Products = $pending.ToArray() }) } $slices = [System.Collections.Generic.List[object]]::new() for ($i = 0; $i -lt $items.Count; $i += $MaxVersionsPerSlice) { $end = [Math]::Min($i + $MaxVersionsPerSlice, $items.Count) - 1 $chunk = @($items[$i..$end]) $slices.Add([PSCustomObject]@{ Index = $slices.Count + 1 Lane = $chunk[0].Lane Items = $chunk }) } foreach ($p in $ordered) { $summary.Add($counts[$p.Name]) } return [PSCustomObject]@{ Slices = $slices.ToArray() Summary = $summary.ToArray() TotalVersions = $items.Count TotalPackages = @($items | ForEach-Object { $_.Products }).Count Lane = $Lane } } |