Publishing Code Coverage in both Azure DevOps and SonarQube

I spent more time than I care to admit trying to get the proper configuration for reporting code coverage to both the Azure DevOps pipeline and SonarQube. The solution was, well, fairly simple, but it is worth me writing down.

Testing, Testing…

After fumbling around with some of the linting and publishing to SonarQube’s Community Edition, I succeeded in creating build pipelines which, when building from the main branch, will run SonarQube analysis and publish to the project.

I modified my build template as follows:

- ${{ if eq(parameters.execute_sonar, true) }}:
    # Prepare Analysis Configuration task
    - task: SonarQubePrepare@5
      inputs:
        SonarQube: ${{ parameters.sonar_endpoint_name }}
        scannerMode: 'MSBuild'
        projectKey: ${{ parameters.sonar_project_key }}

  - task: DotNetCoreCLI@2
    displayName: 'DotNet restore packages (dotnet restore)'
    inputs:
      command: 'restore'
      feedsToUse: config
      nugetConfigPath: "$(Build.SourcesDirectory)/nuget.config"
      projects: "**/*.csproj"
      externalFeedCredentials: 'feedName'

  - task: DotNetCoreCLI@2
    displayName: Build (dotnet build)
    inputs:
      command: build
      projects: ${{ parameters.publishProject }}
      arguments: '--no-restore --configuration ${{ parameters.BUILD_CONFIGURATION }} /p:InformationalVersion=$(fullSemVer) /p:AssemblyVersion=$(AssemblySemVer) /p:AssemblyFileVersion=$(AssemblySemFileVer)'
      

## Test steps are here, details below

  - ${{ if eq(parameters.execute_sonar, true) }}:
    - powershell: |
        $params = "$env:SONARQUBE_SCANNER_PARAMS" -replace '"sonar.branch.name":"[\w,/,-]*"\,?'
        Write-Host "##vso[task.setvariable variable=SONARQUBE_SCANNER_PARAMS]$params"

    # Run Code Analysis task
    - task: SonarQubeAnalyze@5

    # Publish Quality Gate Result task
    - task: SonarQubePublish@5
      inputs:
        pollingTimeoutSec: '300'

In the code above, the execute_sonar parameter allows me to execute the Sonarqube steps only in the main branch, which allows me to keep the community edition happy but retain the rest of my pipeline definition on feature branches.

This configuration worked and my project’s analysis showed in Sonarqube.

Testing, Testing, 1, 2, 3…

I went about adding some trivial unit tests in order to verify that I could get code coverage publishing. I have had experience with Coverlet in the past, and it allows for generation of various coverage report formats, so I went about adding it to the project using the simple collector.

Within my build pipeline, I added the following (if you are referencing the snippet above, this is in place of the comment):

- ${{ if eq(parameters.execute_tests, true) }}:
    - task: DotNetCoreCLI@2
      displayName: Test (dotnet test)
      inputs:
        command: test
        projects: '**/*tests/*.csproj'
        arguments: '--no-restore --configuration ${{ parameters.BUILD_CONFIGURATION }} --collect:"XPlat Code Coverage"'

Out of the box, this command worked well: tests were executed, and both Tests and Coverage information were published to Azure DevOps. Apparently, the DotNetCoreCLI@2 task defaults publishTestResults to true.

While the tests ran, the coverage was not published to Sonarqube. I had hoped that the Sonarqube Extension for Azure DevOps would pick this up, but, at last, this was not the case.

Coverlet, DevOps, and Sonarqube… oh my.

As it turns out, you have to tell Sonarqube explicitly where to find the coverage report. And, while the Sonarqube documentation is pretty good at describing how to report coverage to Sonarqube from the CLI, the Azure DevOps integration documentation does not specify how to accomplish this, at least not outright. Also, while Azure DevOps recognizes cobertura coverage reports, Sonarqube prefers opencover reports.

I had a few tasks ahead of me:

Generate my coverage reports in two formats
Get a specific location for the coverage reports in order to pass that information to Sonarqube.
Tell the Sonarqube analyzer where to find the opencover reports

Generating Multiple Coverage Reports

As mentioned, I am using Coverlet to collect code coverage. Coverlet allows for the usage of RunSettings files, which helps to standardize various settings within the test project. It also allowed me to generate two coverage reports in different formats. I created a coverlet.runsettings file in my test project’s directory and added this content:

<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="XPlat code coverage">
        <Configuration>
          <Format>opencover,cobertura</Format>          
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

To make it easy on my build and test pipeline, I added the following project to my test csproj file:

<RunSettingsFilePath>$(MSBuildProjectDirectory)\coverlet.runsettings</RunSettingsFilePath>

There are a number of settings for coverlet that can be set in the runsettings file, the full list can be found in their VSTest Integration documentation.

Getting Test Result Files

As mentioned above, the DotNetCoreCLI@2 action defaults publishTestResults to true. This setting adds some arguments to the test command to output trx logging and setting a results directory. However, this means I am not able to specify a results directory on my own.

Even specifying the directory myself does not fully solve the problem: Running the tests with coverage and trx logging generates a trx file using username/computer name/timestamp AND two sets of coverage reports: one set is stored under the username/computer name/timestamp folder, and the other under a random Guid.

To ensure I only pulled one set of tests, and that Azure DevOps didn’t complain, I updated my test execution to look like this:

  - ${{ if eq(parameters.execute_tests, true) }}:
    - task: DotNetCoreCLI@2
      displayName: Test (dotnet test)
      inputs:
        command: test
        projects: '**/*tests/*.csproj'
        publishTestResults: false
        arguments: '--no-restore --configuration ${{ parameters.BUILD_CONFIGURATION }} --collect:"XPlat Code Coverage" --logger trx --results-directory "$(Agent.TempDirectory)"'
    
    - pwsh:
        Push-Location $(Agent.TempDirectory);
        mkdir "ResultFiles";
        $resultFiles = Get-ChildItem -Directory -Filter resultfiles;

        $trxFile = Get-ChildItem *.trx;
        $trxFileName = [System.IO.Path]::GetFileNameWithoutExtension($trxFile);
              
        Push-Location $trxFilename;
        $coverageFiles = Get-ChildItem -Recurse -filter coverage.*.xml;
        foreach ($coverageFile in $coverageFiles) 
        {
          Copy-Item $coverageFile $resultFiles.FullName;
        }
        Pop-Location;
      displayName: Copy Test Files

    - task: PublishTestResults@2
      inputs:
        testResultsFormat: 'VSTest' # 'JUnit' | 'NUnit' | 'VSTest' | 'XUnit' | 'CTest'. Alias: testRunner. Required. Test result format. Default: JUnit.
        testResultsFiles: '$(Agent.TempDirectory)/*.trx'

    - task: PublishCodeCoverageResults@1
      condition: true # always try publish coverage results, even if unit tests fail
      inputs:
        codeCoverageTool: 'cobertura' # Options: cobertura, jaCoCo
        summaryFileLocation: '$(Agent.TempDirectory)/ResultFiles/**/coverage.cobertura.xml'

I ran the dotnet test command with custom arguments to log to trx and set my own results directory. The Powershell script uses the name of the TRX file to find the coverage files, and copies them to a ResultsFiles folder. Then I added tasks to publish test results and code coverage results to the Azure DevOps pipeline.

Pushing coverage results to Sonarqube

I admittedly spent a lot of time chasing down what, in reality, was a very simple change:

  - ${{ if eq(parameters.execute_sonar, true) }}:
    # Prepare Analysis Configuration task
    - task: SonarQubePrepare@5
      inputs:
        SonarQube: ${{ parameters.sonar_endpoint_name }}
        scannerMode: 'MSBuild'
        projectKey: ${{ parameters.sonar_project_key }}
        extraProperties: |
          sonar.cs.opencover.reportsPaths="$(Agent.TempDirectory)/ResultFiles/coverage.opencover.xml"

### Build and test here

  - ${{ if eq(parameters.execute_sonar, true) }}:
    - powershell: |
        $params = "$env:SONARQUBE_SCANNER_PARAMS" -replace '"sonar.branch.name":"[\w,/,-]*"\,?'
        Write-Host "##vso[task.setvariable variable=SONARQUBE_SCANNER_PARAMS]$params"
    # Run Code Analysis task
    - task: SonarQubeAnalyze@5
    # Publish Quality Gate Result task
    - task: SonarQubePublish@5
      inputs:
        pollingTimeoutSec: '300'

That’s literally it. I experimented with a lot of different settings, but, in the end, simply setting sonar.cs.opencover.reportsPaths in the extraProperties input of the SonarQubePrepare task.

SonarQube Success!

In the small project that I tested, I was able to get analysis and code coverage published to my SonarQube instance. Unfortunately, this means that I know have technical debt to fix and unit tests to write in order to improve my code coverage, but, overall, this was a very successful venture.