Visualizing Operational Tests with Jenkins and Pester

The Problem

I love Pester, and I really want to get on the operational testing band wagon. But one of the perpetual issues involved with testing is how do I visualize my test metrics, and can I take action on failed tests automatically?

Enter Jenkins and Pester. You can express a lot of concepts in the form of Pester tests and Jenkins is more than happy to take Pester’s output and not only visualize it for you, but take down stream action based on failures and show you those results too.

The Tools


There are lots of great tutorials out there for installing Jenkins, but it’s a big subject that I’m not going to cover here. We’re going to use Jenkins for this, but I’ll assume you have an instance running already.


Installing pester is very easy, but I like to use the Install-Module CmdLet from PSGet and that needs to be installed too if it isn’t already.


That was easy. Don’t get complacent though, take a look at the directory the module was installed into. The problem here is that if you try to load the module from Jenkins right now, it won’t be able to find the module. To make it discover able for Jenkins and for other users we need to copy that module folder to the system wide module folders.

# Get the Pester Module Path
$pesterPath = Get-Module -ListAvailable Pester | Split-Path

# Create variable for our system wide module paths
$modulePaths = "C:\windows\system32\windowspowershell\v1.0\Modules","C:\windows\SysWOW64\WindowsPowerShell\v1.0\Modules"

# Copy Pester to the module paths. We want it available in both 32 and 64 bit PowerShell
$modulePaths.ForEach({Copy-Item -Path $pesterPath -Destination $_ -Recurse})

Notice that I copied the module to the SysWOW64 modules folder. This is because in your own testing you will mostly use 64bit PowerShell, but at the same time, most of you will have downloaded and installed 32bit Jenkins. Using 32Bit Jenkins means you must copy any modules you want to use into the SysWOW64 folder because it can’t see the 64 bit folder. We are going to run a check shortly to make sure the module will be available within Jenkins before we start to run into confusing errors.


We are going to need three plugins to make this workflow happen. Go to Manage Jenkins > Manage Plugins and select the Available tab to select the plugins we need to install: Nunit plugin, Parameterized Trigger plugin, Copy Artifact plugin, and the PowerShell plugin, as depicted in the screenshots below.

Creating the First Project

Our first project is going to be called Test Permissions Job.


The only thing we are going to ask this job to do at first is confirm that the Pester module is going to be available to Jenkins as we expect. It’s worth discussing for just a second though how I intend to accomplish this. Below is a screenshot of the script I’m going to run in the project.



That’s not much right? Here’s the thing. Writing actual PowerShell code in that little text box would suck really really bad. I don’t want to do it. So instead all I do is use the automatic environment variables that Jenkins gives me to find the path to scripts that I will edit in PowerShell ISE instead.

Using this strategy buys me a lot of nice things. The config is nice, clean and easy to read. The scripts are easy to edit in the normal PowerShell ISE. Finally, if I want to make a change to the way the job behaves, I can just edit the script, hit save in the ISE, and execute the job again, without ever having to actually change the job configuration at all! It really makes your testing iterations go much much faster. I think you’ll like it.

If you’re working with a brand new project, the workspace won’t exist yet so you won’t be able to save anything there, but just execute the project with no configuration so it doesn’t do anything, and Jenkins will create the workspace folder for you. Using the the workspace folder is usually a good choice btw, instead of somewhere else on the drive so that if you set up slaves, the jobs will continue to run and be able to access the scripts they need. The workspace folder on the master is usually found at C:\Program Files (x86)\Jenkins\workspace\<job name>

Ok, moving on. The contents of the script are as follows:

try {
    Get-Module -ListAvailable
catch {
    throw $_
    exit -1

And we are going to save it to the workspace folder:


Get-Module is a very simple CmdLet, but don’t forget to almost always put your code in try/catch blocks when scripting for Jenkins. The reason is that to execute PowerShell, Jenkins is calling Powershell.exe from cmd.exe. If you, the script author are not diligent about not only catching errors, but returning non zero return codes, then Jenkins doesn’t have any way of knowing that something went wrong. This will result in job steps that encounter errors, but do not halt project execution, and do not cause projects to be marked as failures. So what we do is catch the error, immediately throw it back out so that the error makes it to the console for logging, and then throw a non zero return code to ensure the project gets marked as failed.

Hopefully though, if we’ve done our job right, the console output from this first job run will show us a list of all available modules, including Pester. If you don’t see Pester in the list try adding a line to the script to output the $PSModulePath variable and ensure the Pester module is in one of those directories.

The First Tests

First we’ll save a script to the workspace folder just like the one above. I am going to call it Get-CustomLocalGroupMembership .ps1 with the contents below:

function Get-CustomLocalGroupMembership

        foreach($computer in $computername)
            $props = @{computername="$computer";members=@()}
            $ADSIcomputer = [adsi]("WinNT://$computer,computer")

            try{$members = $ADSIcomputer.psbase.Children.Find('Administrators','Group').psbase.invoke("Members")}
            catch{Write-verbose "cannot find memberships for $computer"}

            foreach($member in $members)
                try{$props.members += $member.GetType().InvokeMember("Name",'GetProperty',$null,$member,$null)}
                catch{$props.members += $null}

            Write-OutPut (New-Object -TypeName PSObject -Property $props)

Next, the tests file that will leverage that function. I am going to call it UserPermissionsTests.tests.ps1. It will have the contents below and I’ll save it to the same directory.

. $env:workspace\Get-CustomLocalGroupMembership.ps1

$requiredUsers = 'LocalUser1','LocalUser2','LocalUser3'

Describe "Server Alive Tests" {
    $processes = Get-Process

    it "Should be running things" {
        $processes.count | Should BeGreaterThan 1

Describe "Users and groups tests" {

    Context "Group membership context" {
        $members = Get-CustomLocalGroupMembership -computername $env:COMPUTERNAME

        it "Should have returned members" {
            $members | Should Not BeNullOrEmpty

        foreach($user in $requiredUsers){
            it "Should Contain Required User: $user" {
                $members.members -eq $user | Should Be $user

So let’s note a couple things. When I look at that script in PowerShell ISE, that $env:workspace variable isn’t going to mean anything. To test that script effectively in PowerShell ISE you may need to assign an $env:workspace variable in your session manually before testing to ensure it will execute as you expect without having to do make modifications you might forget to remove and break your project.

Next, the list of local users. I actually created them for the purposes of this demo, so feel free to do so yourself to follow along.

Now let’s look at the project’s only build step:


That was easy. Invoke-Pester will automatically do a recursive search in the current working directory ($env:workspace) for any files with <name>.tests.ps1 as the name format and execute them. This project step is very clear and easy to read.

If you noticed that after I just got done saying that we should almost always put our code in try/catch blocks, I didn’t do it here, you aren’t wrong. The trick is that Invoke-Pester is going to take care of this for me with the -enableExit parameter. If any errors occur during execution Invoke-Pester is kind enough to bubble up the error for me and return a non zero exit code. Even if there are no unexpected exceptions; if I simply have failed tests, it will return all of the errors and return an exit code equal to the number of failed tests.

So let’s run the project and see what we get. Our output should look like below:


This is really fantastic. We can see not only that tests failed, but exactly what user we expected to present and wasn’t found. Take some time to click around in the job and look at all the nice results you get, and realize it only gets better as tests start to pass later.

Now let’s see if we can get the output file and read it to figure out how we can make some use of it.


Well, that’s ugly but it will work to help us figure out what to do next. We can use this kind of testing to figure out exactly how to query the xml file to get the data we need, and what we see there is very close. I don’t know NUnitXML format well enought to tell you what query will get you the data you need. I just know XML well enough to keep querying until I have what I need. Later you’ll see the query I came up with to make the project to fix the permissions work. So let’s move on and set up the fix.

Before you attempt to implement the configuration below, create a new Free Style Project called “Fix Permissions Job Step” if you want to follow along, and then add the Post-Build actions shown below to the Permissions Testing project.


Next, in the Fix Permissions Job project we will tell it to copy the xml result file from the permissions test project. You will also see the build step that invokes a pester test. The code that follows will be the content of the AddUsersToAdmins.tests.ps1 file that the build step is invoking along with the code for a helper function it needs.


. $env:JENKINS_HOME\userContent\PowershellScripts\Add-DomainUserToLocalGroup.ps1

[xml]$NUnit = Get-Content $env:WORKSPACE\PermissionsTestsOutput.xml

$users = $NUnit.SelectNodes('//test-case[@result = "Failure"]').failure.message | ForEach-Object{($_ -split '{([a-zA-Z\d\s]+)')[1]}

Describe "Adding Users to Admins" {

    foreach($user in $users){

        it "Should add user to admins: $user" {
            {Add-DomainUserToLocalGroup -domain $env:COMPUTERNAME -user $user -computer $env:COMPUTERNAME -group Administrators} | Should Not Throw


Function Add-DomainUserToLocalGroup
    Write-Host "Adding $user to group: $Group"

    $de =  [ADSI]("WinNT://$computer/$Group,group" )

    Write-host "$user successfully added to $group`n"

Make sure that the test users are NOT a part of the admins group and then run the test job. What you should see is a tests job that runs and fails three tests, then executes the fix job which will add the users and mark those tests as passed.

Once you do that, go ahead and kick off the permissions test project again and you will see the tests not only pass this time, but Jenkins knows the tests failed last time and doesn’t mark them as just passed, but as fixed.

Lets say that those user permissions being missing is a serious problem. In the past you might have set an alert on them being missing so someone could fix it. Now, there is only a need to send out an alert if the attempt at FIXING it fails, which Jenkins will know about as soon as it attempts to fix it for you.


This kind of operational validation can be extended to testing things like ensuring a web site is up. Don’t just test to ensure that the w3wp service is running and that the SQL service is running. You can actually run Invoke-WebRequest and test to ensure that you get a return code of 200 and that the elements you expect to find are present in the web page, using Pester, and if they aren’t, you can run further tests in follow up projects to make automated attempts at solving some of the common issues you know might cause an outage.

You won’t get a midnight alert because Jenkins fixed it for you, but you can see in your build statuses the next morning that something went wrong and take a look at what it was based on the tests that failed.

Thanks for reading and of course if you have any questions, hit me up on Twitter!