Modules/businessdev.ALbuild.RuntimePackages/Public/Invoke-BcRuntimeFactory.ps1
|
function Invoke-BcRuntimeFactory { <# .SYNOPSIS Builds runtime packages for a slice of platform versions, several containers at a time. .DESCRIPTION Runs the work items from Get-BcRuntimeWorkSet through a pool of worker threads, each owning one BC container for one platform version, and checkpoints every version as soon as it is done. WHY A POOL INSIDE ONE JOB Throughput here is capped by the organisation's four parallel self-hosted jobs, not by the build host - which has 36 cores and 192 GB of RAM and was measured with ~72 GB still free while four containers ran. Adding pipeline stages cannot get past that cap; adding workers inside a job does, and costs nothing. WHY CHECKPOINTING Build 27197 built 186 runtime packages and shipped 130 of them. Upload ran once, after the whole loop, so when the run was cancelled ten hours of finished work was thrown away. -OnVersionComplete runs after each platform version, so a cancelled or crashed run loses at most the version still in flight. The callback runs on the DISPATCHER thread, deliberately. Uploading to blob storage and pushing to a feed needs an Azure PowerShell context, and those context objects are not safe to share across runspaces. Workers produce files; the dispatcher ships them. PER-APP VISIBILITY Azure DevOps renders a stage as a single icon, and giving every app its own stage would mean a container per (app, platform version) - the arrangement the factory exists to remove. So the per-app result is carried in the artefacts a job can publish instead: * the log, one collapsible '##[group]' per platform version with one line per app; * -JUnitPath, one test case per app per version for PublishTestResults@2, which is what puts a green/red/skipped entry per app into the Tests tab; * -SummaryPath, the app x version matrix for '##vso[task.uploadsummary]'. Worker output is captured to a per-version file and replayed by the dispatcher when the version finishes, so the log reads as ordered blocks instead of interleaved lines from parallel workers. .PARAMETER WorkItem Work items from Get-BcRuntimeWorkSet (PlatformVersion, ArtifactUrl, Products[]). .PARAMETER WorkRoot Root for the private per-worker working copies. .PARAMETER OutputFolder Root of the '<appId>/<platformVersion>/' output layout. .PARAMETER Throughput Requested worker count. Capped by available memory and disk - see Get-BcRuntimeWorkerCount. .PARAMETER MemoryLimit Per-container memory limit. .PARAMETER ReserveHostGb Host memory to leave for the agent, the compiler and the OS. .PARAMETER ReserveDiskGb Disk to leave free on the drive Docker stores containers on. .PARAMETER DiskPerContainerGb What one container is assumed to cost on that drive. Default 8, from measurement rather than estimate: run 27372 logged the free space at each of 12 container creations across five hours with two containers live throughout, and it stayed between 102.4 and 98.2 GB - about 4 GB for BOTH containers including their installed apps. The generic default in Get-BcRuntimeWorkerCount is 25 GB, which is right for a container that copies the artifact into its own writable layer; the factory mounts a shared host artifact cache instead, so it never pays that. 8 GB keeps a factor of four over what was measured. This is what capped the fast lane at two workers on a host with 138 GB of free memory: the old figure allowed (102.4 - 40) / 25 = 2, so requesting 3 changed nothing. .PARAMETER Credential Container admin credential. .PARAMETER LegacyLicenseFile The .flf license for platform majors up to 19; BC20 and newer take -LicenseFile. Both are needed because one run spans majors on either side of that change. .PARAMETER LicenseFile BC licence for the containers. .PARAMETER Signing Splat for Invoke-BcAppSigning. .PARAMETER TeardownMode Full (default) removes each app and its dependency chain after its runtime package is produced. .PARAMETER UseImageCache Start containers from the cached version-specific image. .PARAMETER OnVersionComplete Scriptblock invoked on the dispatcher thread with the finished version result - the checkpoint. .PARAMETER LogFolder Where per-version worker logs are written. Defaults to a folder under -WorkRoot. .PARAMETER JUnitPath Write JUnit results here for PublishTestResults@2. .PARAMETER SummaryPath Write the Markdown matrix here for task.uploadsummary. .PARAMETER DockerExecutable Docker executable. .PARAMETER NoAzureDevOpsLogging Suppress the '##vso[...]' logging commands. Tests set this: they run inside a real Azure DevOps job, where a logged issue would mark that job rather than the run under test. .OUTPUTS PSCustomObject: Produced, Failed, Skipped, Results[], WorkerCount. #> [CmdletBinding(SupportsShouldProcess)] [OutputType([PSCustomObject])] param( [Parameter(Mandatory)] [AllowEmptyCollection()] [object[]] $WorkItem, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $WorkRoot, [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [string] $OutputFolder, [ValidateRange(1, 64)] [int] $Throughput = 3, [string] $MemoryLimit = '8G', [ValidateRange(0, 512)] [int] $ReserveHostGb = 16, [ValidateRange(0, 4096)] [int] $ReserveDiskGb = 40, [ValidateRange(1, 512)] [int] $DiskPerContainerGb = 8, [pscredential] $Credential, [string] $LicenseFile, [string] $LegacyLicenseFile, [hashtable] $Signing, [ValidateSet('Full', 'AppOnly')] [string] $TeardownMode = 'Full', [switch] $UseImageCache, [scriptblock] $OnVersionComplete, [string] $LogFolder, [string] $JUnitPath, [string] $SummaryPath, [string] $DockerExecutable = 'docker', # Suppress the '##vso[...]' logging commands. Set by tests: they run inside a real Azure # DevOps job, where those commands would apply to the job running the tests. [switch] $NoAzureDevOpsLogging ) Assert-ALbuildLicensed -Feature 'RuntimePackages' $items = @($WorkItem) if ($items.Count -eq 0) { Write-ALbuildLog 'Nothing to build - every planned runtime package already exists.' if ($JUnitPath) { Set-Content -LiteralPath $JUnitPath -Value (ConvertTo-BcRuntimeJUnit -Result @()) -Encoding UTF8 } if ($SummaryPath) { Set-Content -LiteralPath $SummaryPath -Value (ConvertTo-BcRuntimeSummaryMarkdown -Result @()) -Encoding UTF8 } return [PSCustomObject]@{ Produced = 0; Failed = 0; Skipped = 0; Results = @(); WorkerCount = 0 } } foreach ($folder in @($WorkRoot, $OutputFolder)) { if (-not (Test-Path -LiteralPath $folder)) { New-Item -ItemType Directory -Force -Path $folder | Out-Null } } if (-not $LogFolder) { $LogFolder = Join-Path $WorkRoot 'logs' } if (-not (Test-Path -LiteralPath $LogFolder)) { New-Item -ItemType Directory -Force -Path $LogFolder | Out-Null } $workers = Get-BcRuntimeWorkerCount -Requested $Throughput ` -MemoryLimitGb ([int]($MemoryLimit -replace '[^0-9]', '')) ` -FreeMemoryGb (Get-BcRuntimeFreeMemoryGb) ` -ReserveMemoryGb $ReserveHostGb ` -FreeDiskGb (Get-BcRuntimeFreeDiskGb -DockerExecutable $DockerExecutable) ` -ReserveDiskGb $ReserveDiskGb -DiskPerContainerGb $DiskPerContainerGb # Thread jobs need PowerShell 7; on Windows PowerShell 5.1 the factory still works, just serially. $canRunParallel = $workers.Count -gt 1 -and $null -ne (Get-Command -Name 'Start-ThreadJob' -ErrorAction SilentlyContinue) if (-not $canRunParallel -and $workers.Count -gt 1) { Write-ALbuildLog -Level Warning 'Start-ThreadJob is unavailable (Windows PowerShell 5.1); running one container at a time.' } Write-ALbuildLog "Worker pool: $($workers.Reason)$(if (-not $canRunParallel) { ' (serial)' })." $results = [System.Collections.Generic.List[object]]::new() $checkpoint = { param([object] $Result) $log = Join-Path $LogFolder "$($Result.PlatformVersion).log" Write-Host "##[group]BC $($Result.PlatformVersion) - $($Result.Status) ($([Math]::Round($Result.Seconds / 60, 1)) min)" if (Test-Path -LiteralPath $log) { Get-Content -LiteralPath $log | ForEach-Object { Write-Host $_ } } foreach ($p in @($Result.Products)) { $mark = switch ($p.Status) { 'Succeeded' { ' +' } 'Skipped' { ' -' } default { ' x' } } Write-Host "$mark $($p.Name) [$($p.Status)] $($p.Seconds)s$(if ($p.Error) { " - $($p.Error)" })" } Write-Host '##[endgroup]' # Issues are raised from the dispatcher only, after the replay, so they attribute to the right # version instead of landing in whatever block happened to be open. foreach ($p in @($Result.Products | Where-Object { $_.Status -eq 'Failed' })) { # Suppressible for the same reason: a unit test that drives the failure path would # otherwise raise real warnings on the job running it. if (-not $NoAzureDevOpsLogging) { Write-Host "##vso[task.logissue type=warning]BC $($Result.PlatformVersion): $($p.Name) - $($p.Error)" } } $results.Add($Result) if ($OnVersionComplete) { try { & $OnVersionComplete $Result } catch { Write-ALbuildLog -Level Warning "Checkpoint for BC $($Result.PlatformVersion) failed: $($_.Exception.Message)" } } } if ($PSCmdlet.ShouldProcess("$($items.Count) platform version(s)", 'Build runtime packages')) { $buildArgs = @{ WorkRoot = $WorkRoot OutputFolder = $OutputFolder Credential = $Credential MemoryLimit = $MemoryLimit LicenseFile = $LicenseFile LegacyLicenseFile = $LegacyLicenseFile Signing = $Signing TeardownMode = $TeardownMode UseImageCache = [bool]$UseImageCache DockerExecutable = $DockerExecutable } if (-not $canRunParallel) { foreach ($item in $items) { & $checkpoint (Invoke-BcRuntimeVersionBuild @buildArgs -Item $item) } } else { $manifest = Join-Path (Split-Path -Parent (Split-Path -Parent $PSScriptRoot)) '..' | Join-Path -ChildPath 'businessdev.ALbuild.psd1' $manifest = [System.IO.Path]::GetFullPath($manifest) $queue = [System.Collections.Generic.Queue[object]]::new() foreach ($item in $items) { $queue.Enqueue($item) } $running = [System.Collections.Generic.List[object]]::new() $start = { param([object] $Item) $job = Start-ThreadJob -ArgumentList $manifest, $Item, $buildArgs, $LogFolder -ScriptBlock { param($ManifestPath, $Item, $BuildArgs, $LogFolder) $logFile = Join-Path $LogFolder "$($Item.PlatformVersion).log" $resultFile = Join-Path $LogFolder "$($Item.PlatformVersion).result.xml" try { Import-Module $ManifestPath -Force -DisableNameChecking # RuntimePackages is NESTED inside the root manifest, so 'Get-Module -Name' never # returns it - a fresh runspace only ever sees 'businessdev.ALbuild'. The module # object is needed because Invoke-BcRuntimeVersionBuild is private and only # reachable through its own module scope. Reach it via the root's NestedModules; # the by-name lookup stays first for the case where the submodule was imported # standalone, which the tests do. $module = Get-Module -Name 'businessdev.ALbuild.RuntimePackages' if (-not $module) { $module = @((Get-Module -Name 'businessdev.ALbuild').NestedModules | Where-Object { $_.Name -eq 'businessdev.ALbuild.RuntimePackages' })[0] } # Fail with something a human can act on. Without this the next line dies as # "The expression after '&' ... produced an object that was not valid", which # says nothing about the module that failed to resolve. if (-not $module) { throw "businessdev.ALbuild.RuntimePackages could not be resolved after importing '$ManifestPath'. Loaded modules: $(@(Get-Module | ForEach-Object { $_.Name }) -join ', ')." } # Every stream is captured to the worker's own file. Nothing writes to the host # from a worker: parallel writes would interleave into an unreadable log. $result = $null $output = & { $splat = @{} foreach ($k in $BuildArgs.Keys) { if ($null -ne $BuildArgs[$k]) { $splat[$k] = $BuildArgs[$k] } } $result = & $module { param($S, $I) Invoke-BcRuntimeVersionBuild @S -Item $I } $splat $Item $result | Export-Clixml -LiteralPath $resultFile -Depth 6 } *>&1 $output | ForEach-Object { "$_" } | Set-Content -LiteralPath $logFile -Encoding UTF8 } catch { "Worker for BC $($Item.PlatformVersion) failed: $($_.Exception.Message)" | Set-Content -LiteralPath $logFile -Encoding UTF8 [PSCustomObject]@{ PlatformVersion = "$($Item.PlatformVersion)" Status = 'Failed' Seconds = 0 Products = @($Item.Products | ForEach-Object { [PSCustomObject]@{ Name = "$($_.Name)"; Status = 'Failed'; Seconds = 0; Error = $_.Exception.Message; File = $null } }) } | Export-Clixml -LiteralPath $resultFile -Depth 6 } return "$($Item.PlatformVersion)" } $running.Add([PSCustomObject]@{ Job = $job; Item = $Item }) } while ($running.Count -lt $workers.Count -and $queue.Count -gt 0) { & $start $queue.Dequeue() } while ($running.Count -gt 0) { $finished = Wait-Job -Job @($running.Job) -Any foreach ($done in @($finished)) { $entry = @($running | Where-Object { $_.Job.Id -eq $done.Id })[0] Receive-Job -Job $done -ErrorAction SilentlyContinue | Out-Null Remove-Job -Job $done -Force -ErrorAction SilentlyContinue [void]$running.Remove($entry) $resultFile = Join-Path $LogFolder "$($entry.Item.PlatformVersion).result.xml" $result = if (Test-Path -LiteralPath $resultFile) { Import-Clixml -LiteralPath $resultFile } else { [PSCustomObject]@{ PlatformVersion = "$($entry.Item.PlatformVersion)" Status = 'Failed' Seconds = 0 Products = @($entry.Item.Products | ForEach-Object { [PSCustomObject]@{ Name = "$($_.Name)"; Status = 'Failed'; Seconds = 0; Error = 'The worker produced no result.'; File = $null } }) } } & $checkpoint $result if ($queue.Count -gt 0) { & $start $queue.Dequeue() } } } } } $all = @($results | ForEach-Object { $_.Products }) $produced = @($all | Where-Object { $_.Status -eq 'Succeeded' }).Count $failed = @($all | Where-Object { $_.Status -eq 'Failed' }).Count $skipped = @($all | Where-Object { $_.Status -eq 'Skipped' }).Count if ($JUnitPath) { Set-Content -LiteralPath $JUnitPath -Value (ConvertTo-BcRuntimeJUnit -Result $results.ToArray()) -Encoding UTF8 } if ($SummaryPath) { Set-Content -LiteralPath $SummaryPath -Value (ConvertTo-BcRuntimeSummaryMarkdown -Result $results.ToArray()) -Encoding UTF8 } Write-ALbuildLog -Level Success "Runtime factory: $produced package(s) built, $failed failed, $skipped skipped across $($results.Count) platform version(s)." # NO '##vso[task.complete ...]' here. A cmdlet must not decide the result of the pipeline task that # happens to be running it - the caller owns that, and does it from the returned counts. When the # unit tests exercised the failure path inside a real Azure DevOps job, this line marked the CI job # itself SucceededWithIssues (build 27294). # # A failed platform version still never fails the run: Microsoft occasionally publishes an artifact # that cannot produce a working container, and one bad version must not stall the catalogue. return [PSCustomObject]@{ Produced = $produced Failed = $failed Skipped = $skipped Results = $results.ToArray() WorkerCount = $workers.Count } } |