Modules/businessdev.ALbuild.Apps/Public/Test-BcTestAssertion.ps1
|
function Test-BcTestAssertion { <# .SYNOPSIS Gates a build on every AL [Test] procedure containing at least one assertion (anti-mock gate). .DESCRIPTION A green test suite proves nothing if the tests assert nothing: a [Test] that only calls the code under test passes whatever the code returns, and it still counts as "covered". This scans the AL [Test] procedures in a workspace and reports every one whose body contains no recognisable assertion, so a pull request cannot introduce tests that cannot fail. Recognised as an assertion: * the assertion library - Assert.*, LibraryAssert.*, Assertion.* * assertion members - .AreEqual/.AreNotEqual/.IsTrue/.IsFalse/.IsEmpty/.RecordCount/ .ExpectedError/.KnownFailure/.Equal/.NotEqual/.Fail * the AL error keywords - asserterror, TestField(), FieldError() * a guard Error() - the hand-rolled assert pattern `if <unexpected> then Error(...)` * verification helpers - a call whose name contains Verify/Validate/Ensure/Assert/Check, so a test that delegates its checks to a helper procedure is not reported. Use -IgnoreVerificationHelpers for the stricter reading. A test that asserts nothing in its own body is not reported until the CALLS have been followed: every procedure it calls that exists in the scanned source is examined for an assertion, then their callees, up to -MaxHelperDepth levels. This is what a matrix or factory suite looks like - one [Test] per case whose body is `MatrixLibrary.RunCase('<case id>')`, with the arrangement and the checks stated once in the library instead of once per case - and it needs no marker in the source to pass. Unlike the name heuristic above, the walk is honest: the resolved body has to contain a real assertion (the Verify*/Check* name pattern is NOT accepted at this level), so a `VerifyFoo()` that checks nothing is still reported. Two boundaries keep the walk from turning the gate into a rubber stamp: * it stays inside AL projects (app.json folders) that declare [Test] procedures, so a call into production code is never followed - assertions belong to the test app; * a bare `Error(` does not count in a followed body. It is the hand-rolled assert only in a test body; anywhere else it is ordinary error handling, and production-like code is full of it. A product that really does assert through a guard `Error()` in its helpers can put that back with -AdditionalAssertionPattern '\berror\s*\('. The last resort, for an assertion that is genuinely not in the scanned source - it lives in a test library shipped as a dependency app - is a directive comment. It survives the AL compiler, is counted and listed as an exemption rather than silently accepted, and reads: // albuild:no-assert-required <why> Written above the codeunit's first test (or anywhere before it) it covers the whole file; written with the test's attributes or inside its body it covers that test. An empty test body is reported as well (reason 'Empty'), and a directive does not exempt it: an empty [Test] cannot delegate anything. The scan is deliberately generous: it must not cry wolf on a real test, because a gate that produces false positives gets switched off. Related: Get-BcTestQuality scores the whole suite (assertion density, arrange/act/assert smells) for reporting; this cmdlet answers the single yes/no question a build gate needs and shares the same AL scan, so the two can never disagree about which procedures are tests. .PARAMETER WorkspaceRoot One or more AL source roots to scan for [Test] procedures. Several roots are scanned as ONE suite (a test found under two overlapping roots is counted once), so a multi-project repository yields a single verdict instead of one per project. .PARAMETER Path A single .al file to check instead of a whole workspace. .PARAMETER AssertionPattern Overrides the built-in assertion regex (case-insensitive, matched per statement line) - for a product whose tests assert through an in-house library the default does not know. This REPLACES the built-ins; to keep them and add one more pattern use -AdditionalAssertionPattern. .PARAMETER AdditionalAssertionPattern An extra assertion regex, OR-ed into the built-ins (and into the pattern used to resolve helpers). The additive form of -AssertionPattern, so recognising an in-house assertion cannot accidentally switch the built-in ones off. .PARAMETER MaxHelperDepth How many call levels to follow from a test body that does not assert itself, looking for an assertion in a procedure of the scanned source. Default 3; 0 switches the walk off (the pre-2.45 behaviour, where a delegating test was reported). .PARAMETER IgnoreVerificationHelpers Do not accept a call to a Verify*/Validate*/Ensure*/Assert*/Check* helper as an assertion. Stricter: the test must then assert in its own body, or an assertion must be reachable from it through the call walk. Recommended together with the walk - the name is no longer taken on trust, but a test that really delegates its checks still passes. .PARAMETER ThrowOnFailure Throw a terminating error when at least one test has no assertion (for use as a pipeline gate). .PARAMETER Quiet Suppress the gate's own PASSED/FAILED line and the offender list. For callers that report the verdict themselves -- the Azure DevOps task renders it as a pipeline issue, as a warning or an error depending on failOnIssue -- so the same result is not logged twice. .EXAMPLE Test-BcTestAssertion -WorkspaceRoot . -ThrowOnFailure Fail the build if any [Test] procedure in the repository asserts nothing. .EXAMPLE Test-BcTestAssertion -WorkspaceRoot . -IgnoreVerificationHelpers -ThrowOnFailure The strict reading: a helper's NAME proves nothing, but an assertion reachable through it does. .EXAMPLE (Test-BcTestAssertion -WorkspaceRoot .).offenders | Format-Table codeunit, testName, line List the tests that need an assertion. .OUTPUTS PSCustomObject verdict: passed, testCount, testCodeunitCount, offenderCount, offenders[], helperResolvedCount, helperResolved[], exemptCount, exempt[]. #> [CmdletBinding(DefaultParameterSetName = 'Workspace')] [OutputType([PSCustomObject])] param( [Parameter(Mandatory, ParameterSetName = 'Workspace')] [ValidateNotNullOrEmpty()] [string[]] $WorkspaceRoot, [Parameter(Mandatory, ParameterSetName = 'File')] [ValidateNotNullOrEmpty()] [string] $Path, [string] $AssertionPattern, [string] $AdditionalAssertionPattern, [ValidateRange(0, 10)] [int] $MaxHelperDepth = 3, [switch] $IgnoreVerificationHelpers, [switch] $ThrowOnFailure, [switch] $Quiet ) $helperRx = '\b\w*(verify|validate|ensure|assert|check)\w*\s*\(' # Unambiguous assertions: they say the outcome is being checked and nothing else. These are the only # ones accepted in a body reached through the call walk. $unambiguous = @( '\b(assert|libraryassert|assertion)\s*\.' '\.(areequal|arenotequal|istrue|isfalse|isempty|isnotempty|recordcount|expectederror|knownfailure|equal|notequal|fail)\s*\(' '\basserterror\b' '\.(testfield|fielderror)\s*\(' ) # A bare Error() is the hand-rolled assert `if <unexpected> then Error(...)` in a TEST body, but in any # other procedure it is ordinary error handling, so it stays out of the walk. A product that does assert # through a guard Error() in its helpers can put it back with -AdditionalAssertionPattern '\berror\s*\('. $builtIn = @($unambiguous) + @('\berror\s*\(') # Two patterns, because the two questions differ. In the test's own body a call to a Verify*/Check* # helper is accepted (unless -IgnoreVerificationHelpers): generous, so the gate does not cry wolf. # In a body reached through the call walk it is NOT - there the callee is already the helper, and # accepting its name again would let a helper that checks nothing satisfy the gate. $strict = @($unambiguous) $direct = @($builtIn) if (-not $IgnoreVerificationHelpers) { $direct += $helperRx } if ($AdditionalAssertionPattern) { $strict += $AdditionalAssertionPattern; $direct += $AdditionalAssertionPattern } $pattern = if ($AssertionPattern) { $AssertionPattern } else { '(' + ($direct -join '|') + ')' } $strictPattern = if ($AssertionPattern) { $AssertionPattern } else { '(' + ($strict -join '|') + ')' } # IgnoreCase on the regex rather than an inline (?i) in the composed pattern: AL is case-insensitive, # and a caller's own -AssertionPattern is documented as case-insensitive too - it must not silently # depend on whether the caller happened to prefix it with (?i). $ignoreCase = [System.Text.RegularExpressions.RegexOptions]::IgnoreCase $assertRx = [regex]::new($pattern, $ignoreCase) $strictRx = [regex]::new($strictPattern, $ignoreCase) # @(...) around the whole expression: a single test would otherwise be unrolled to a scalar, and # .Count on a scalar throws under StrictMode on Windows PowerShell 5.1. $tests = @(if ($PSCmdlet.ParameterSetName -eq 'File') { Get-BcAlTestProcedure -Path $Path } else { $found = [System.Collections.Generic.List[object]]::new() foreach ($root in $WorkspaceRoot) { foreach ($proc in (Get-BcAlTestProcedure -WorkspaceRoot $root)) { $found.Add($proc) } } # Overlapping roots (a project inside another) would otherwise count the same test twice. $found | Sort-Object FilePath, Line -Unique }) # Pass 1: the body itself. Everything that survives it is a candidate for the call walk, so the # procedure index - the expensive part - is built only for a suite that actually needs it. $candidates = [System.Collections.Generic.List[object]]::new() foreach ($proc in @($tests)) { $asserts = @(@($proc.Statements) | Where-Object { $assertRx.IsMatch($_) }).Count if ($asserts -eq 0) { $candidates.Add($proc) } } $index = $null if ($candidates.Count -gt 0 -and $MaxHelperDepth -gt 0) { $index = @{} $all = @(if ($PSCmdlet.ParameterSetName -eq 'File') { Get-BcAlProcedure -Path $Path } else { $procs = [System.Collections.Generic.List[object]]::new() foreach ($root in $WorkspaceRoot) { foreach ($proc in (Get-BcAlProcedure -WorkspaceRoot $root)) { $procs.Add($proc) } } $procs | Sort-Object FilePath, Line -Unique }) # The walk stays on the test side of the repository. An assertion belongs to the app that declares # the tests; production code is full of `Error(` and of procedures whose name contains Check or # Validate, so following a call into it would let the gate pass anything that touches the product. $projectCache = @{} $projectOf = @{} $testProjects = @{} foreach ($proc in @($all)) { $key = Get-BcAlProjectRoot -FilePath $proc.FilePath -Cache $projectCache $projectOf[$proc.FilePath] = $key if ($proc.IsTest) { $testProjects[$key] = $true } } foreach ($proc in @($all)) { if (-not $testProjects.Contains($projectOf[$proc.FilePath])) { continue } # Name collisions across objects are kept side by side: the walk asks whether an assertion is # reachable, so every procedure that could be meant is examined. if (-not $index.Contains($proc.Name)) { $index[$proc.Name] = [System.Collections.Generic.List[object]]::new() } $index[$proc.Name].Add(@($proc.Statements)) } } $offenders = [System.Collections.Generic.List[object]]::new() $resolved = [System.Collections.Generic.List[object]]::new() $exempt = [System.Collections.Generic.List[object]]::new() foreach ($proc in @($candidates)) { $statements = @($proc.Statements) if ($statements.Count -gt 0 -and $null -ne $index) { $reach = Resolve-BcAlAssertionReach -Statements $statements -Index $index -AssertionRegex $strictRx -MaxDepth $MaxHelperDepth if ($reach) { $resolved.Add([PSCustomObject]@{ codeunit = $proc.Codeunit testName = $proc.TestName filePath = $proc.FilePath line = $proc.Line via = $reach.via depth = $reach.depth }) continue } } # An empty [Test] is never exempt: it delegates nothing, so there is nothing to point the # directive at. $directive = @(@($proc.Directives) | Where-Object { $_ -match '(?i)^no-assert-required\b' }) if ($statements.Count -gt 0 -and $directive.Count -gt 0) { $exempt.Add([PSCustomObject]@{ codeunit = $proc.Codeunit testName = $proc.TestName filePath = $proc.FilePath line = $proc.Line note = ($directive[0] -replace '(?i)^no-assert-required\s*:?\s*', '') }) continue } $offenders.Add([PSCustomObject]@{ codeunit = $proc.Codeunit testName = $proc.TestName filePath = $proc.FilePath line = $proc.Line statementCount = $statements.Count reason = if ($statements.Count -eq 0) { 'Empty' } else { 'NoAssertions' } }) } $verdict = [PSCustomObject]@{ passed = ($offenders.Count -eq 0) testCount = $tests.Count testCodeunitCount = @($tests | Select-Object -ExpandProperty Codeunit -Unique).Count offenderCount = $offenders.Count offenders = @($offenders) helperResolvedCount = $resolved.Count helperResolved = @($resolved) exemptCount = $exempt.Count exempt = @($exempt) assertionPattern = $pattern helperAssertPattern = $strictPattern maxHelperDepth = $MaxHelperDepth helperCallsAccepted = (-not $IgnoreVerificationHelpers) } if (-not $Quiet) { # The two soft outcomes are always stated, also on a pass: a suite whose tests all assert through # a factory should say so in the log, and an exemption must never be invisible. if ($resolved.Count -gt 0) { $deepest = @($resolved | Measure-Object depth -Maximum).Maximum Write-ALbuildLog ("{0} test(s) assert through a called procedure (deepest chain: {1} level(s))." -f $resolved.Count, $deepest) } foreach ($e in $exempt) { Write-ALbuildLog -Level Warning (" {0}::{1} ({2}:{3}) - exempted by // albuild:no-assert-required {4}" -f $e.codeunit, $e.testName, (Split-Path -Leaf $e.filePath), $e.line, $e.note) } } if ($verdict.passed) { if (-not $Quiet) { Write-ALbuildLog -Level Success ("Assertion gate PASSED: all {0} [Test] procedure(s) in {1} codeunit(s) assert." -f $verdict.testCount, $verdict.testCodeunitCount) } } else { $msg = "{0} of {1} [Test] procedure(s) contain no assertion - they cannot fail." -f $offenders.Count, $verdict.testCount if (-not $Quiet) { # Offenders as plain lines: one ##vso error per test would bury the summary in the pipeline UI. foreach ($o in $offenders) { Write-ALbuildLog (" {0}::{1} ({2}:{3}) - {4}" -f $o.codeunit, $o.testName, (Split-Path -Leaf $o.filePath), $o.line, $o.reason) } Write-ALbuildLog -Level Warning "Assertion gate FAILED: $msg" } if ($ThrowOnFailure) { throw "Assertion gate failed: $msg" } } return $verdict } |