The Purge! Purging data from Exchange Online

By | May 27, 2022

So many many years ago we used to have a lovely tool called search-mailbox which allowed us to search for some mail based on some parameters then permanently purge the results from the mailbox.

This capability has been removed and replaced with the pairing of Compliance searches and Compliance search actions.

The big issue for some is that a Compliance search action can only remove 10 items from a mailbox per run. I get the idea, I grasp the concept of why but I maintain there are times you need to be able to reach out and shred a bunch of mail from a bunch of mailboxes. (Removing confidential information prior to migrating to a new tenant during acquisitions/mergers/divestiture for example)

Running into this issue headlong I have seen multiple places claiming your can simply loop the compliance search action with various people offering things like rerunning the searches. None of this worked for me (tested 25/05/2022)

  • Simply looping the Compliance search action a bunch of times or ANY variant of rerunning the original search then another compliance search action failed to do anything other that remove the first 10 matches then never anything else. This included clearing the Purges folder out with Managed Folder Assistant prior to rerunning the search. (For anyone that doesn’t know – if you run a purge with harddelete the mails will be moved the purges folder in the mailbox which is emptied when managed folder assistance runs against it)
  • Creating a new search before the Purges folder was emptied by the Managed Folder Assistant was at best “inconsistent” as the search results included the purges folder
  • Creating a search after the Purges folder had been cleared by managed folder assistant worked fine but I am utterly convinced the start-managedfolderassistant cmdlet was deprecated 6 years ago and does nothing at all so you could need to wait days between runs which is “not great”
  • So the one that worked was to find the purges folder for each mailbox in scope, add them as an exception to the search and then create a loop which ran that search, ran the purge on the results, deleted the search, then ran it again until it returned zero results.

It’s not fast, but here it is:

NOTES – the target mailboxes must be added to a mail enabled security or distribution group but I’m gonna guess you do that anyway. This is fully tested where single item recovery is NOT on and there is no Compliance level retention policy configured. In both of those cases you are going to see mail going into Recoverable Items not Purges which means the search results will include stuff you already deleted. I might revisit this should I ever need to but for now it wont work.

# connect-ippssession
# connect-exchangeonline
#-----------------------------------------------------------------------------------------------------------------
#Update these
$keywords = "Omega Molecule"
$targetgroup = "Dodgy Characters"

#Leave these alone
$targetgroupSMTP = (Get-DistributionGroup -Identity $targetgroup).PrimarySMTPAddress
$searchname = "KeywordCleanse - "+$keywords
$purgename = $searchname+"_Purge"
$contentquery = $keywords
$items = 1
$j=0
#------------------------------------------------------------------------------------------------------------------
#Get the purge folders
Write-Host
Write-Host "Extracting purge folders" -ForegroundColor Yellow
Function Get-FolderID([array]$Mailbox, [array]$FolderPath)
{
    [System.Collections.ArrayList]$FolderIDValues = @()
    Foreach ($Identity in $Mailbox)
    {
        try { [array]$Folders = Get-MailboxFolderStatistics -Identity $Identity -ea stop }
        catch { "Mailbox $($Identity) not found. Skipping."; Continue }
        foreach ($Folder in $FolderPath)
        {
          $MailboxIdentity = $Identity.ToString()
          $OriginalFolderId = ($Folders | ? { $_.FolderPath -imatch "$Folder`$" }).FolderId
          If ($OriginalFolderId)
          {
            $FolderName = $Folder.ToString()

            # Convert to correct folder encoding
            $Encoding = [System.Text.Encoding]::GetEncoding("us-ascii");
            $FolderIdBytes = [Convert]::FromBase64String($OriginalFolderID);
            $Nibbler = $encoding.GetBytes("0123456789ABCDEF");
            $IndexIdBytes = New-Object byte[] 48;
            $IndexIdIdx = 0;
            $FolderIdBytes | select -skip 23 -First 24 | %{ $IndexIdBytes[$IndexIdIdx++] = $Nibbler[$_ -shr 4]; $IndexIdBytes[$IndexIdIdx++] = $Nibbler[$_ -band 0xF] }
            $FolderId = $($encoding.GetString($indexIdBytes))
          }
          Else { $FolderId = "Folder path/id not found in mailbox"; $FolderName = $Folder }
      
          # Create the entry object to add to the collection of folders
          $entry = New-Object PSObject
          $entry | Add-Member -Name "Mailbox" -MemberType NoteProperty -Value $MailboxIdentity
          $entry | Add-Member -Name "FolderName" -MemberType NoteProperty -Value $FolderName
          $entry | Add-Member -Name "FolderPath" -MemberType NoteProperty -Value ($Folders | ? { $_.FolderPath -imatch "$Folder`$"}).FolderPath
          $entry | Add-Member -Name "FolderId" -MemberType NoteProperty -Value $FolderId
          $FolderIDValues += $entry
        }
      }
      Return $FolderIDValues
    }

$mbxs = (Get-DistributionGroupMember -Identity $targetgroup).primarysmtpaddress
Foreach ($mbx in $mbxs) {
    # $mbx
    $folderid=Get-FolderID -Mailbox $mbx -FolderPath Purges
    $contentquery += " (c:c) NOT folderid:"+$folderid.FolderId
}


Write-Host "Generating search query" -ForegroundColor Yellow
#-----------------------------------------------------------------------------------------------------------------------
#SearchLoop
Do {
    $j++
    Write-Host "Starting Search "$j -ForegroundColor Green
    $tonull = New-ComplianceSearch -Name $searchname -ExchangeLocation $targetgroupSMTP -ContentMatchQuery $contentquery
    $tonull = Start-ComplianceSearch -Identity $searchname

    Do {
        $progress = get-compliancesearch -Identity $searchname
       # $progress.Status
        Start-Sleep -Seconds 10
        Write-Host "." -ForegroundColor Cyan -NoNewLine

    } until ($progress.Status -eq "Completed")

    Write-Host "Search Complete" $progress.Items "Items Found" -ForegroundColor Red
    $items = $progress.Items

    #-----------------------------------------------------------------------------------------------------------------------
    #Purge
    If ($items -ne 0) {
        Write-Host "Starting Purge - Purging 10 items per mailbox" -ForegroundColor Green
        $tonull = New-ComplianceSearchAction -SearchName $searchname -Purge -PurgeType HardDelete -Confirm:$false
        #$results= Get-ComplianceSearchAction -Identity $searchname
        #$results
        Do {
            Start-Sleep -Seconds 10
            Write-Host "." -ForegroundColor Cyan -NoNewLine
             $results= Get-ComplianceSearchAction -Identity $purgename
             #$results
        } Until ($results.Status -eq "Completed")
        Write-Host "Purge Complete" -foregroundcolor Red
        Write-Host "Results"
        $results.Results
    }
    #-----------------------------------------------------------------------------------------------------------------------
    #Update Results

    Write-Host "Removing Search" -ForegroundColor Green
    Remove-ComplianceSearch $SearchName -Confirm:$false

} Until ($items -eq 0)

Please note the function call to get the folderid. I stole the whole thing from here so go there and click some stuff so the author knows he’s appreciated:)

Loading

Leave a Reply

Your email address will not be published. Required fields are marked *