(Y2K22 Bug) Microsoft Exchange Buggy Update cause Historical Bug

Update: 

1. Microsoft acknowledges the issue in Exchange 2016 &2019 that leads to Email Stuck in Transport Queues.
2. Microsoft with the official fix and guide on how to apply.

 

An update for Microsoft Exchange 2019 makes “Microsoft Filtering Management Service” being stopped and caused the error reports with issues in mail flow. This can be a serious issue that may ruin the holidays of many exchange servers admins.

The root cause of the bug was due to using a signed int32 for the date and the new date value by Microsoft, of 2,201,010,001 is overvalued the maximum value of a signed long integer i.e 2147483647. The bug gets triggered when the “MS Filtering Engine Update” process received the “220101001” update version just some hours before the New Year, and the process started crashing as time was 0000 (O'clock). In the application event logs, the errors from the FIPFS service say: Cannot convert “220101001” too long.

Microsoft Acknowledgment

Microsoft has announced that they are working on a fix to their FIP-FS antivirus engine which is used in Microsoft Exchange 2016 and 2019. If this problem isn't fixed soon it could cause emails to be truncated before they arrive. This issue was first reported right as the New Year arrived.

Microsoft noted that the problem relates to a date check failure in the change of the near from 2021 to 2022. 

This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.

We are actively working on resolving this issue and expect to release details on how to resolve this issue later today. In the meantime, if your organization performs malware scanning of messages outside of your on-premises Exchange servers (for example, by routing mail through Exchange Online, or by using a third-party message hygiene solution), you can bypass or disable malware scanning on your Exchange servers and clear your transport queues. 

Microsoft assures the users that they are working on a fix of the issue and an update is in final test validation. This update requires customer action, but it will provide the quickest time to resolve the issue for users.

In the meantime, Microsoft recommends Two documents be available on this topic (how to disable or bypass malware scanning). One is about the Antimalware protection in Exchange Server, while the other covers procedures for antimalware protection in Exchange Server.

Affected Application

According to the Reddit thread, these bad updates affected Microsoft Exchange 2016 & 2019 versions. Furthermore, a Reddit user @ceno666 also confirms that the issue also seems to occur with the 220101002 update version as well. 

A user name JulianSiebert had hinted about the issue on the announcement post of the Exchange team, where the team already mentioned the postponed of December 2021 Exchange Server Cumulative Updates.

FIP-FS is probably the anti-malware virus scanner that has been on board since Exchange Server 2013. This is supposed to scan the on-premises Exchange Server installation for malicious content. However, this anti-malware scan engine seems to cause problems more often.

Microsoft uses the first two numbers of the updated version to denote the year of the update. So when the year was 2021, the first two numbers were “21” in “2147483647”, and everything was fine. Now that it’s 2022, the updated version, converted to a “long” would be 2,201,01,001 — which is above the maximum value of the “long” data type. If users check for another update, it won't be going to help as it returned “MS Filtering Engine Update process has not detected any new scan engine updates”.

Note this is only for On-Prem routed mail it seems and of course, 365 is working correctly

Solution and Guide to Apply Fix

Microsoft has created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. 

Implementation of the solution requires customer actions. These actions can be automated with the scan engine reset script or they can be performed manually.

Using the Automated Solution

Run the script on each Exchange server in your organization. You can run this script on multiple servers in parallel. After the script has been completed, you will see the following output:

[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Reset-ScanEngineVersion.ps1
EXCH1 Stopping services...
EXCH1 Removing Microsoft engine folder...
EXCH1 Emptying metadata folder...
EXCH1 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EXCH1 Starting engine update...
Running as EXCH1-DOM\Administrator.
--------
Connecting to EXCH1.CONTOSO.com.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>Get-EngineUpdateInformation

Engine                : Microsoft
LastChecked           : 01/01/2022 08:58:22 PM -08:00
LastUpdated           : 01/01/2022 08:58:31 PM -08:00
EngineVersion         : 1.1.18800.4
SignatureVersion      : 1.355.1227.0
SignatureDateTime     : 01/01/2022 03:29:06 AM -08:00
UpdateVersion         : 2112330001
UpdateStatus          : UpdateAttemptSuccessful

If you use the automated script, you can run it on multiple servers in parallel. We recommend starting with the servers in your environment that have the largest queues. You can use Get-Queue to view queue size and determine which queues are the largest. 

Using the Manual Solution

Server Admins can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange server in your organization:

Remove existing engine and metadata

1. Stop the Microsoft Filtering Management service.  When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to the latest engine

1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.

Verify engine update info

1. In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
2. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001.

After updating the engine, Microsoft also recommends that admins verify that mail flow is working and that FIPFS error events are not present in the Application event log.

Some users report that they are still having issues even after applying the fix and updating the system. So to fix this we recommend all server admins restart the whole server after applying the fix, to get the whole thing working again.

If you previously disabled or bypassed antimalware scanning as a mitigation for this issue, we recommend that you re-enable it after performing the above steps.

Fix for Exchange mail flow breaks

On the 10th of the December, Microsoft Exchange team already announced that team will not be releasing any Cumulative Updates (CUs) for Exchange Server in December and will share more information about our next CU release at a later time. So to date, there is no news for the update.

Security enthusiastic people calling this issue as Y2K22 and triggering this Time value bug on many 2016 & 2019 exchange servers, makes many server admins unhappy. Exchange Server admins can temporarily fix the issue by disabling anti-malware scanning, to restore mail flow. For this purpose, there is the script Disable-AntiMalwareScanning.ps1. Then malware scans are no longer executed

CD "C:\Program Files\Microsoft\Exchange Server\V15\Scripts\"
.\Disable-AntiMalwareScanning.ps1
Restart-Service MSExchangeTransport

Another way is by disabling the filtering on the exchange server -

Set-MalwareFilteringServer exch-19 -BypassFiltering $true

Hope the Microsoft Exchange team soon released a fix of the bug.

Note: Disabling OR bypassing anti-malware filtering puts many servers at risk of phishing or other malware threats.

History of the Same bug (Y2K)

In the year 1999, a bug arose due to a problem in the coding of computerized systems that was projected to create havoc in computers and computer networks around the world at the beginning of the year 2000. This bug dubbed as Y2K (in metric measurements, k stands for 1,000), also called the Year 2000 bug or Millennium Bug,

Until the 1990s' many computer programs (especially those written in the early days of computers) were designed to abbreviate four-digit years as two digits to save memory space. These computers could recognize “98” as “1998” but would be unable to recognize “00” as “2000,” perhaps interpreting it to mean 1900. 

Many feared that when the clocks struck midnight on January 1, 2000, many affected computers would be using an incorrect date and thus fail to operate properly unless the computers’ software was repaired or replaced before that date. Other computer programs that projected budgets or debts into the future could begin malfunctioning in 1999 when they made projections into 2000. 

In addition, some computer software did not take into account that the year 2000 was a leap year. And even before the dawn of 2000, it was feared that some computers might fail on September 9, 1999 (9/9/99), because early programmers often used a series of 9s to indicate the end of a program.

In the year 1998, Microsoft issued a software fix to address the Year 2000 problems in Windows 98 that the company described as more of an inconvenience than a threat. Without the fix, certain parts of the operating system might inaccurately display dates come the millennium, Microsoft said.