Fixing SonarCloud code coverage condition count with multiple .NET Core test projects

SonarSource made a great improvement, it will now show conditional coverage of your tests. Unfortunately, when using the pipeline as described in my previous blogpost, SonarCloud reports way too many conditions. For instance, a simple if(condition) would result in 10 possible conditions which clearly is incorrect but easy to fix. We submitted a bug report for this.

The problem appears to be that for every test project an OpenCover file is created which shows the coverage for that single test projects. Every OpenCover file contains all statements and their conditions. SonarCloud properly merges line coverage, but it appears to sum the amount of conditions. So where the most simple if statement should have 2 conditions, SonarCloud actually reported 2x5=10 conditions (we have 5 test projects) with only 2 conditions being covered.

This resulted in a massive drop in reported code coverage, as SonarCloud immediately started using conditional coverage as part of the calculations for “code coverage”. This didn’t make us look great anymore and more importantly our pull requests were failing. In the screenshot below the drop in coverage is clearly visible, along with the sudden appearance of conditional coverage. Time to fix this! Just want the Yml? Scroll a bit down :)

What does this pipeline do?

Many of the same things that are already described in my previous blogpost. A summary:

1. Pool/agent selection
2. Variables are defined. The test output directory is important for the guide as it will contain all test & coverage files we need.
3. Prepares for SonarCloud analysis. We’re setting it up to collect TRX files which contain testresults and also to collect a single XML file containing all code coverage. Last, we are ignoring coverage on some files.
4. Installs .NET Core 2.x runtime, as the SonarCloud plugin depends on it.
5. Installs .NET Core 3.1.101 SDK to build our software.
6. Restores NuGet packages from a private feed.
7. Builds the code with the provided configuration, while skipping the restore step for each project saving a few seconds.
8. Runs all tests, skipping restore and build steps for each project saving some more seconds. TRX files (--logger trx) and Cobertura files (--collect:"XPlat Code Coverage") are written to the test output directory. The fact that it writes Cobertura files isn’t explicit, it’s just the default output of Coverlet.
9. Runs ReportGenerator which collects all Cobertura files (one for each test project) and merges them. The output is written to 3 different formats: An HTML report to show in the build output, a Cobertura file to publish to Azure DevOps and a SonarQube specific format used by SonarCloud. Make sure to ignore the same files as specified at step 4.
10. SonarCloud analyzes the codebase.
11. SonarCloud publishes the result to Azure DevOps to show if the Quality Gate passed for your build.
12. WhiteSource bolt runs to scan your dependencies for vulnerabilities.
13. The web project is published and zipped to the artifact staging directory.
14. The artifact staging directory contents are published as build artifacts.

Improvements on pipeline from previous blogpost

• ReportGenerator used to create a single truth that is used by both SonarCloud and Azure DevOps. I would love to see both SonarCloud and Azure DevOps being able to deal with multiple test/coverage files, but currently this appears to be the best solution.
• No more Coverlet.runsettings file needed to set Coverlet output to OpenCover.
• No longer generating HTML dashboards with ReportGenerator (standard PublishCodeCoverageResults works fine now). So no more need of HtmlInline_AzurePipelines parameter or disable.coverage.autogenerate variable.
• New pipeline uses SonarCloud instead of SonarQube. Be aware, this requires a different extension in Azure DevOps:

Pipeline

pool:
  name: Default
  demands:
  - Agent.OS -equals Linux
  
variables:
  buildConfiguration: Release
  project: '$(Build.SourcesDirectory)/Solution.sln'
  testOutputDirectory: '$(Agent.TempDirectory)/testresults'

steps:
- task: SonarCloudPrepare@1
  inputs:
    SonarCloud: 'SonarCloud'
    organization: 'organization'
    scannerMode: 'MSBuild'
    projectKey: 'projectKey'
    projectName: 'projectName'
    extraProperties: |
        sonar.cs.vstest.reportsPaths=$(TestOutputDirectory)/*.trx
        sonar.coverageReportPaths=$(TestOutputDirectory)/mergedcoveragereport/SonarQube.xml
        sonar.coverage.exclusions=**/Migrations/*.cs,**/*Tests*/**/*

- task: UseDotNet@2
  displayName: 'Install .NET Core 2.x runtime as it is needed for SonarCloud plugin'
  inputs:
    packageType: 'runtime'
    version: '2.x'
    
- task: UseDotNet@2
  displayName: 'Use .NET Core sdk'
  inputs:
    packageType: sdk
    version: 3.1.101
    installationPath: $(Agent.ToolsDirectory)/dotnet

- task: DotNetCoreCLI@2
  displayName: 'dotnet restore private feed'
  inputs:
    command: restore
    projects: '$(project)'
    vstsFeed: '00000000-0000-0000-0000-000000000000'

- task: DotNetCoreCLI@2
  displayName: 'dotnet build'
  inputs:
    projects: '$(project)'
    arguments: '--no-restore --configuration $(BuildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: 'dotnet test'
  inputs:
    command: test
    publishTestResults: false
    projects: '$(project)'
    arguments: '--no-restore --no-build --configuration $(BuildConfiguration) --logger trx --collect:"XPlat Code Coverage" --results-directory $(TestOutputDirectory)'
    
- task: reportgenerator@4
  inputs:
    reports: '$(TestOutputDirectory)/*/coverage.cobertura.xml'
    targetdir: '$(TestOutputDirectory)/mergedcoveragereport'
    reporttypes: 'Cobertura;SonarQube'
    assemblyfilters: '-*Tests*'
    filefilters: '-*/Migrations/*.cs'

- task: PublishCodeCoverageResults@1
  inputs:
    codeCoverageTool: 'Cobertura'
    summaryFileLocation: '$(TestOutputDirectory)/mergedcoveragereport/Cobertura.xml'

- task: SonarCloudAnalyze@1

- task: SonarCloudPublish@1
  inputs:
    pollingTimeoutSec: '300'

- task: WhiteSource Bolt@20
  inputs:
    cwd: '$(Build.SourcesDirectory)'

- task: DotNetCoreCLI@2
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)' 
    artifactName: 'artifactName'
  condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))

The result

For this project we’re back at a proper coverage: