How to limit a number of PowerShell Jobs running simultaneously

Recently, I received a task which required me to run a particular command on a several thousands of servers. Since execution of this command takes some time, it is just logical to run them in a parallel mode. And PowerShell Background Jobs are the first thing comes in mind.
But resources of my PC are limited — it cannot run more than a 100 jobs simultaneously. Unfortunately, PowerShell doesn’t have a built-in functionality for limiting background jobs yet.

Though, I’m not the first one who stuck with the same problem: official “Hey, Scripting Guy!” blog has introduced us a queue based on .NET Framework objects. But I couldn’t manage this solution to work and needed something simpler. After all, all we need are:

  • a loop
  • a counter, for how much jobs are active and running
  • a variable, allowing next job in queue to start

Eventually, I came up with a piece of code like this:

Whereas my version is similar to a solution proposed at StackOverflow (which I only found AFTER completing my own version, of course), the SO version suffers from a bug where some items in queue may be skipped.

While PS Jobs are so easy to play with, Boe Prox claims that runspaces work much and much faster. Go, check it out, how you can use them to accelerate your job queue in his blog.

Some other queueing technics in PowerShell:

7 thoughts on “How to limit a number of PowerShell Jobs running simultaneously”

  1. Well, finally somebody who knows how to keep it simple. Nice and clean and works like a charm

  2. Your version suffers from a while loop excessively checking/polling for the running status count to change
    A better solution would be to use the built-in Wait-Job -Any command-let to check for threads finishing.

    The problem with the stack overflow version is that it was doing an if else and it only waited in the else then moved on to the next iteration item without starting a job for it.

    My suggestion is to check for the conditions to wait on first.
    If the condition is true wait
    then start the job – always happens.

    $maxConcurrentJobs = 100

    foreach ($object in $objects) {
    $running = @(Get-Job -State Running)
    $freeMemory = (Get-WmiObject win32_OperatingSystem).freephysicalmemory * 1KB
    if ($freeMemory -lt 1.5GB -or $running.Count -ge $maxConcurrentJobs) {
    $null = $running | Wait-Job -Any
    }
    $scriptBlock = {
    #Insert the code of your workload here
    Write-Output “Do your work here…”
    }
    Start-Job -ScriptBlock $ScriptBlock
    Start-Sleep -Seconds 3 # not needed but gives some breathing room.
    }

    I added the memory check to avoid throwing System.OutOfMemoryException but that value is dependent on your system

  3. @Kirill and @Mackenson
    thanks both so much. It’s so simple but effective and exactly what I needed.
    I had been asked to supply a script with a list of objects, could be 10, could be 200, whatever the DB spits out, but only do 10 at a time, and NOT break up the script into multiple runs of 10.
    I used the “Wait-Job -Any” code and I just added in front of it … write-host ” Waiting … ”
    Great stuff.

  4. All great examples, thank you. Simplified it a bit by creating a reusable function for this:

    “`powershell
    Function Wait-MaxRunningJobsHC {

    [CmdletBinding()]
    Param (
    [Parameter(Mandatory)]
    [System.Management.Automation.Job[]]$Name,
    [Parameter(Mandatory)]
    [Int]$MaxThreads,
    [Int]$FreeMemory = 1GB
    )

    Begin {
    Function Get-FreeMemoryHC {
    (Get-WmiObject win32_OperatingSystem).FreePhysicalMemory * 1KB
    }
    Function Get-RunningJobsHC {
    @($Name).Where( { $_.State -eq ‘Running’ })
    }
    }

    Process {
    while (
    ((Get-FreeMemoryHC) -lt $FreeMemory) -or
    ((Get-RunningJobsHC).Count -ge $MaxThreads)
    ) {
    $null = Wait-Job -Job $Name -Any
    }
    }
    }
    “`

    Hope this might helpful for others

Leave a Reply

Your email address will not be published. Required fields are marked *