Message deferred by categorizer agent

 

 

I woke up to a new years surprise in January 2022,  as did many e-mail , infrastructure and network admins out there.

The Issue:

 

 

E-mail is down and messages are piling up in the message queue. You did the usual things already like check exchange server resources, disk space, cpu ,memory , check back pressure mode , restart Exhange transport services and even restart the servers themselves.

 

The Cause

 

 

Due of a latent date problem in a signature file utilized by the malware scanning engine inside Exchange Server, messages are blocked in transport queues on Exchange Server 2016 and Exchange Server 2019. When the problem arises you will notice the following error messages:

Error messages

In the message queue you see error message “message deferred by categorizer”

you'll observe problems in the Exchange Server's Application event log, notably events 5300 and 1106 (FIPFS), as seen below:

Event ID 5300

The FIP-FS "Microsoft" Scan Engine failed to load. PID: 38648, Error Code: 0x80004005. Error Description: Can't convert "2201010009" to long.

Event ID 1106

The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error

Additional Errors:

Event ID 5801
Source: MSExchange Antimalware

The anti-malware agent encountered an error while scanning. MessageId: XXXXXXX391.1641112993320.JavaMail.SERVER$@DOMAIN.COM Message sent: 2022-01-02 08:43:13 AM From: EMAILADDRESS@DOMAIN.COM Size: 6756 Bytes Error: Microsoft.Filtering.ScanAbortedException: Exception of type 'Microsoft.Filtering.ScanAbortedException' was thrown.

at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException(WSM_ReturnCode code, String message)

at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar)

at Microsoft.Exchange.Transport.Agent.Malware.MalwareAgent.OnScanCompleted(IAsyncResult ar)

 

The Workaround

 

 

1. Find the exchange scripts folder and run the disable antimalware script. Its not instant so give it a minute or two.

cd "C:\Program Files\Microsoft\Exchange Server\V15\Scripts"

.\Disable-AntimalwareScanning.ps1

2. Restart the Microsoft Exchange Transport Services

Get-Service MSExchangeTransport |Restart-Service

I hope this has been helpful to you and saved you some New Years time

Let me know in the comments down below.

Additional Information can be sourced from:

https://techcommunity.microsoft.com/t5/exchange-team-blog/email-stuck-in-transport-queues/ba-p/3049447

https://www.itwriting.com/blog/11910-exchange-emails-stuck-in-queue-because-message-deferred-by-categorizer-agent-happy-new-year-admins.html

Update:

 

 

Source:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked       : 01/01/2022 08:58:22 PM -08:00
LastUpdated        : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001
UpdateStatus          : UpdateAttemptSuccessful

Manual Solution

Source:

In lieu of using the script, customers can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange server in your organization:

Remove existing engine and metadata
1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine
1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info
1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.