High Write Latency on Azure Local (Stack) HCI

High volume write latency for no obvious reason on Azure Local HCI is likely a misconfiguration of the storage volume. In one of our 1st Azure Local configurations we had enabled Dedup and compression on the same volume and over time that had a very strange effect.

After the 1st few weeks of no issues we started getting the following emails

When we 1st reviewed this error there was no latency write spikes in the logs and the system continued to perform well however over time we then began to get random reboots due to this issue and then as time went on we started to see latency events that could only be resolved by a reboot of the hosts.

The write latency started to appear where at its best we got up to 6.6353 hour of latency on a volume. Impressive!!

We worked with Microsoft for quite some time on this and upon detailed analysis the issue was a result of both Dedup and Compression being enabled on the same volume which is unsupported at the moment due to a bug.

Microsoft documentation is updated to reflect this so please follow closely when configuring volumes to avoid these issues.

https://learn.microsoft.com/en-us/azure-stack/hci/manage/refs-deduplication-and-compression?tabs=powershell#when-compression-is-enabled-using-the--fullrun-parameter-on-jobs-after-the-first-optimization-run-might-result-in-a-deadlock-in-the-system