Modules/businessdev.ALbuild.Apps/Public/Invoke-BcContainerTest.ps1
|
function Invoke-BcContainerTest { <# .SYNOPSIS Runs AL tests in a Business Central container and collects/parses the results. .DESCRIPTION ALbuild's built-in AL test runner. It drives the AL Test Tool page inside the container (no BcContainerHelper) and produces a JUnit/XUnit result file that is copied to the host, parsed into a summary, and optionally used to fail the build. Test apps are taken from -ProjectFolder (every AL test app found is run) or from an explicit -ExtensionId list. For folder-based discovery the optional ALbuild 'pipeline.config' beside each app.json is honoured: its 'alTestRunnerId' selects the test-runner codeunit. The runner codeunit otherwise defaults to 130450 (test isolation) or, with -DisableIsolation, 130451 (isolation disabled); an explicit -TestRunnerCodeunitId overrides everything. For advanced scenarios a custom in-container invocation can still be supplied via -TestExecution (a script block that writes the result file to $ResultPathInContainer); when given, ALbuild only collects and parses the result and does not use the built-in runner. .PARAMETER Name Container name. .PARAMETER ProjectFolder Folder searched recursively for AL test apps to run. Mutually exclusive with -ExtensionId. .PARAMETER ExtensionId One or more app ids to run tests for, instead of discovering them from a project folder. .PARAMETER Credential Credentials of a SUPER user (NavUserPassword). If omitted, the containerUsername / containerPassword environment variables are used when present. .PARAMETER TestSuite Test suite name. When omitted, the project's albuild.json 'testSuite' applies, else 'DEFAULT'. .PARAMETER TestRunnerCodeunitId Explicit test-runner codeunit id, overriding pipeline.config and the isolation default. .PARAMETER DisableIsolation Use the isolation-disabled test runner (130451) as the default instead of 130450. .PARAMETER Tenant Tenant to use. Default 'default'. .PARAMETER CompanyName Company to run tests in. Default: the server's default company. .PARAMETER Culture Culture for the test run. Default 'en-US' (Microsoft tests target en-US). .PARAMETER Auth Client services credential type: NavUserPassword (default), Windows or AAD. .PARAMETER AzureDevOps Emit Azure DevOps warning log issues for failing tests. .PARAMETER ResultPath Host path to write the JUnit result file to. Default: ./TestResults.xml. .PARAMETER ResultPathInContainer Container path the run writes results to. Default: C:\run\my\TestResults.xml. A path under C:\run\my is read from the host share; any other path is copied out with 'docker cp'. .PARAMETER CodeCoverageTrackingType Code coverage tracking: Disabled (default), PerRun, PerCodeunit or PerTest. PerRun is captured as PerCodeunit, because Business Central keeps only the last codeunit under PerRun; the merged result covers the whole run. .PARAMETER CodeCoverageMap Test-to-code map to record with coverage: Disabled (default), PerCodeunit or PerTest. .PARAMETER CodeCoveragePath Host folder to copy the raw coverage .dat files into. Created when missing. Without it no coverage files are collected. .PARAMETER TestCodeunit Run only these test codeunits: an id, a range or an AL filter ('50100', '50100..50150', '50100|50200'; '50100-50150' is read as a range). Selects across every installed test app, so a filtered run is one pass, not one per app. A single test codeunit takes minutes instead of the whole suite. .PARAMETER TestProcedure Run only the test procedures whose name matches this AL filter ('PostInvoice', 'Post*|Release*'). Combines with -TestCodeunit or with the extension selection. .PARAMETER MinimumTests Fail when fewer tests ran than this. Guards against a suite that silently lost tests. .PARAMETER ExpectDeclaredTests Fail when fewer tests ran than the test apps under -ProjectFolder declare: every [Test] procedure, narrowed by -TestCodeunit / -TestProcedure. Catches a codeunit that never ran (a broken subscriber, a wrong suite, a run that broke off) while the rest passed. .PARAMETER TestExecution Optional custom in-container script block (advanced). When supplied, the built-in runner is not used; the block must write the result file to $ResultPathInContainer. .PARAMETER FailOnTestFailure Throw if any test failed. .PARAMETER DockerExecutable The Docker executable to use (default 'docker'). .EXAMPLE Invoke-BcContainerTest -Name bld -ProjectFolder $env:BUILD_REPOSITORY_LOCALPATH -FailOnTestFailure .EXAMPLE Invoke-BcContainerTest -Name bld -ExtensionId '5e8c2f...' -Credential $cred -AzureDevOps .OUTPUTS PSCustomObject with Passed, Failed, Skipped, Total, Failures, FailureSources (each failure with its callstack resolved to file and line where the source is in the workspace), Success, ResultPath, CoveragePath/CoverageFiles and ClientErrors. A non-empty ClientErrors throws instead of returning: the test client hit a server error dialog, so the suite did not run as requested - the results are read and copied first, so nothing collected is lost. #> [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSAvoidUsingConvertToSecureStringWithPlainText', '', Justification = 'Rebuilds the container credential from the containerPassword variable to authenticate the in-container test client; never persisted.')] [CmdletBinding(DefaultParameterSetName = 'ProjectFolder')] [OutputType([PSCustomObject])] param( [Parameter(Mandatory)] [ValidateNotNullOrEmpty()] [Alias('ContainerName')] [string] $Name, [Parameter(ParameterSetName = 'ProjectFolder')] [string] $ProjectFolder, [Parameter(Mandatory, ParameterSetName = 'ExtensionId')] [ValidateNotNullOrEmpty()] [string[]] $ExtensionId, [Parameter(Mandatory, ParameterSetName = 'Custom')] [ValidateNotNull()] [scriptblock] $TestExecution, [pscredential] $Credential, [string] $TestSuite = 'DEFAULT', [int] $TestRunnerCodeunitId, [switch] $DisableIsolation, [string] $Tenant = 'default', [string] $CompanyName = '', [string] $Culture = 'en-US', [ValidateSet('NavUserPassword', 'Windows', 'AAD')] [string] $Auth = 'NavUserPassword', [switch] $AzureDevOps, [string] $ResultPath = (Join-Path (Get-Location) 'TestResults.xml'), # Default into the shared mount (C:\run\my) so the result can be read on the host without # 'docker cp', which is unsupported against a running hyperv-isolated container. [string] $ResultPathInContainer = 'C:\run\my\TestResults.xml', [switch] $FailOnTestFailure, [ValidateSet('Disabled', 'PerRun', 'PerCodeunit', 'PerTest')] [string] $CodeCoverageTrackingType = 'Disabled', [ValidateSet('Disabled', 'PerCodeunit', 'PerTest')] [string] $CodeCoverageMap = 'Disabled', [string] $CodeCoveragePath = '', [string] $TestCodeunit, [string] $TestProcedure, [ValidateRange(0, [int]::MaxValue)] [int] $MinimumTests = 0, [switch] $ExpectDeclaredTests, [string] $DockerExecutable = 'docker' ) # Fall back to the containerUsername/containerPassword variables (set by the Create BC Container # task) when no credential was passed - the in-container test client must authenticate, otherwise # it fails with 'InvalidCredentialsError' and the session never initialises. if (-not $Credential -and $env:containerUsername -and $env:containerPassword) { $Credential = [System.Management.Automation.PSCredential]::new( $env:containerUsername, (ConvertTo-SecureString -String $env:containerPassword -AsPlainText -Force)) } # BC's 'Per Run' does not survive this runner. The client drives the AL Test Tool one codeunit at a # time (RunNextTest), and every call is a complete "Test Runner - Mgt".RunTests: it fires # OnRunTestSuite / OnAfterRunTestSuite, and the Per Run subscriber saves into the same # "Test Code Coverage Result" record (codeunit 0, the suite name) each time. Every codeunit overwrites # the one before, so the run reported the coverage of its LAST test codeunit - Banking CI 28539 showed # 3.37% for code the quality gate measured at 70.96%. Per Codeunit keeps one result per codeunit and # the conversion merges them into exactly what Per Run promises: the whole run. if ($CodeCoverageTrackingType -eq 'PerRun') { Write-ALbuildLog ("Code coverage: capturing 'PerCodeunit' for the requested 'PerRun'. The runner executes one " + "codeunit per call and Business Central keeps only the last one under 'PerRun'; the merged result covers the whole run.") $CodeCoverageTrackingType = 'PerCodeunit' } $clientRunErrors = @() if ($PSCmdlet.ParameterSetName -eq 'Custom') { Invoke-BcContainerCommand -ContainerName $Name -DockerExecutable $DockerExecutable ` -Variables @{ ResultPathInContainer = $ResultPathInContainer } -ScriptBlock $TestExecution | Out-Null } else { $nativeArgs = @{ Name = $Name ResultPathInContainer = $ResultPathInContainer DisableIsolation = $DisableIsolation Tenant = $Tenant CompanyName = $CompanyName Culture = $Culture Auth = $Auth AzureDevOps = $AzureDevOps DockerExecutable = $DockerExecutable } # Only when given: unbound, the runner takes albuild.json's testSuite, then DEFAULT. if ($PSBoundParameters.ContainsKey('TestSuite')) { $nativeArgs['TestSuite'] = $TestSuite } if ($PSBoundParameters.ContainsKey('ProjectFolder')) { $nativeArgs['ProjectFolder'] = $ProjectFolder } if ($PSBoundParameters.ContainsKey('ExtensionId')) { $nativeArgs['ExtensionId'] = $ExtensionId } if ($Credential) { $nativeArgs['Credential'] = $Credential } if ($PSBoundParameters.ContainsKey('TestRunnerCodeunitId')) { $nativeArgs['TestRunnerCodeunitId'] = $TestRunnerCodeunitId } if ($TestCodeunit) { $nativeArgs['TestCodeunit'] = ConvertTo-BcTestCodeunitFilter -Filter $TestCodeunit } if ($TestProcedure) { $nativeArgs['TestProcedure'] = $TestProcedure.Trim() } if ($TestCodeunit -or $TestProcedure) { Write-ALbuildLog ("Test filter: codeunit(s) '{0}', procedure(s) '{1}'." -f $(if ($TestCodeunit) { $nativeArgs['TestCodeunit'] } else { 'all' }), $(if ($TestProcedure) { $TestProcedure } else { 'all' })) } if ($CodeCoverageTrackingType -ne 'Disabled') { $nativeArgs['CodeCoverageTrackingType'] = $CodeCoverageTrackingType $nativeArgs['CodeCoverageMap'] = $CodeCoverageMap # Write into the C:\run\my host share so the raw .dat files can be read back on the host. $nativeArgs['CodeCoveragePathInContainer'] = 'C:\run\my\albuild-coverage' } $clientRunErrors = @(Invoke-BcNativeContainerTest @nativeArgs) } # Prefer reading the result from the host side of the shared folder (the container wrote it # there); fall back to 'docker cp' for a process-isolated container or a custom path. $hostResult = $null if ($ResultPathInContainer -like 'C:\run\my\*') { $hostResult = Join-Path (Get-BcContainerHostShare -Name $Name) (Split-Path -Path $ResultPathInContainer -Leaf) } if ($hostResult -and (Test-Path -LiteralPath $hostResult)) { Copy-Item -LiteralPath $hostResult -Destination $ResultPath -Force } else { Invoke-BcDocker -DockerExecutable $DockerExecutable -Quiet -Arguments @('cp', "$($Name):$ResultPathInContainer", $ResultPath) | Out-Null } if (-not (Test-Path -LiteralPath $ResultPath)) { throw "Test result file '$ResultPath' was not produced by the test run." } # Copy the raw code coverage .dat files out of the container's host share to the requested folder. $coverageFiles = @() if ($CodeCoverageTrackingType -ne 'Disabled' -and $CodeCoveragePath) { if (-not (Test-Path -LiteralPath $CodeCoveragePath)) { New-Item -ItemType Directory -Force -Path $CodeCoveragePath | Out-Null } $covHostDir = Join-Path (Get-BcContainerHostShare -Name $Name) 'albuild-coverage' if (Test-Path -LiteralPath $covHostDir) { foreach ($dat in @(Get-ChildItem -LiteralPath $covHostDir -Filter '*.dat' -File -ErrorAction SilentlyContinue)) { Copy-Item -LiteralPath $dat.FullName -Destination $CodeCoveragePath -Force $coverageFiles += (Join-Path $CodeCoveragePath $dat.Name) } } Write-ALbuildLog "Code coverage: collected $($coverageFiles.Count) raw file(s) into '$CodeCoveragePath'." } [xml] $doc = Get-Content -LiteralPath $ResultPath -Raw -Encoding UTF8 $summary = Get-BcTestResultSummary -Document $doc Write-ALbuildLog ("Tests: $($summary.Passed) passed, $($summary.Failed) failed, $($summary.Skipped) skipped.") foreach ($failure in $summary.Failures) { Write-ALbuildLog -Level Error $failure } # A failure message names a codeunit; it does not say where in it. The AL callstack does, but its line # numbers are counted from each procedure's declaration, so on their own they point nowhere. Resolving # them here means every surface - the DevOps task, the CLI, the MCP server - gets the file and line # without doing this itself. Best effort by design: no source, no resolution, and the run is unaffected. $failureSources = @() if (-not $summary.Success -and $ProjectFolder -and (Test-Path -LiteralPath $ProjectFolder)) { try { $failureSources = @(Resolve-BcTestFailureSource -ResultPath $ResultPath -WorkspaceRoot $ProjectFolder) foreach ($fs in $failureSources) { $where = @($fs.Frames | Where-Object { $_.Resolved } | ForEach-Object { "$($_.File):$($_.Line)" }) if ($where.Count -gt 0) { Write-ALbuildLog -Level Error (" $($fs.Name) -> " + ($where -join ' <- ')) } } } catch { Write-ALbuildLog -Level Verbose "Could not resolve failure sources: $($_.Exception.Message)" } } $result = [PSCustomObject]@{ Passed = $summary.Passed Failed = $summary.Failed Skipped = $summary.Skipped Total = $summary.Total Failures = $summary.Failures FailureSources = $failureSources Success = $summary.Success ResultPath = $ResultPath CoveragePath = if ($coverageFiles.Count -gt 0) { $CodeCoveragePath } else { '' } CoverageFiles = @($coverageFiles) ClientErrors = @($clientRunErrors) Filtered = [bool]($TestCodeunit -or $TestProcedure) Expected = $null } # How many tests SHOULD have run. A run can pass while a whole codeunit never ran - the result file # only knows what ran. Compared after the results are copied, so nothing collected is lost. $expected = 0 $expectedFrom = '' if ($MinimumTests -gt 0) { $expected = $MinimumTests; $expectedFrom = "the minimum of $MinimumTests" } if ($ExpectDeclaredTests) { if ($ProjectFolder -and (Test-Path -LiteralPath $ProjectFolder)) { $codeunitFilter = if ($TestCodeunit) { ConvertTo-BcTestCodeunitFilter -Filter $TestCodeunit } else { '' } $declared = Get-BcDeclaredTestCount -WorkspaceRoot $ProjectFolder -TestCodeunit $codeunitFilter -TestProcedure $TestProcedure Write-ALbuildLog ("Declared tests: {0} in {1} test codeunit(s){2}." -f $declared.Total, @($declared.Codeunits).Count, $(if ($result.Filtered) { ' matching the filter' } else { '' })) if ($declared.Total -gt $expected) { $expected = $declared.Total; $expectedFrom = "the $($declared.Total) [Test] procedure(s) the test apps declare" } } else { Write-ALbuildLog -Level Warning '-ExpectDeclaredTests needs -ProjectFolder (the test app source); the declared test count was not checked.' } } if ($expected -gt 0) { $result.Expected = $expected if ([int]$summary.Total -lt $expected) { throw ("Only {0} test(s) ran, but {1} were expected ({2}). A test codeunit did not run, or the suite lost tests." -f $summary.Total, $expected, $expectedFrom) } Write-ALbuildLog -Level Success ("Test count check passed: {0} test(s) ran, {1} expected." -f $summary.Total, $expected) } # A server error dialog during the run is NOT a test failure: the client stopped doing what it was # told, so the results describe a suite that was never fully run. Reported after the results have # been read and copied, so nothing collected is lost, and independently of -FailOnTestFailure - # there is no reading of this state under which the run can be called successful. if (@($clientRunErrors).Count -gt 0) { throw ("The test client hit $(@($clientRunErrors).Count) server error(s) during the run, so the suite did not run as requested" + " ($($summary.Total) test(s) executed): $(@($clientRunErrors) -join ' | ')") } if ($FailOnTestFailure -and -not $summary.Success) { throw "$($summary.Failed) test(s) failed." } return $result } |