Voicemail Issues Today

We had a minor voicemail issue today that affected a small number of messages left. Here's what happened:

Our voicemail messages are stored on Network Storage Devices so multiple voicemail servers can access them easily.

In this case, there apparently was a very small amount of latency between the server in question and the NAS today. It was just a few milliseconds so barely noticeable.

The voicemail system checks every message immediately after its recorded to make sure it's not silent and meets a minimum length (in case of a hang up). This tiny amount of latency today was basically having it check it before the message was fully saved. So then it marked it as silent and removed it.

So really the silence detection was happening right before the message was fully saved.

Since the latency wasn't consistently there, this is apparently why it was happening on some messages today, but not all.

We've made a few changes to the timing of the detection which should prevent this from happening again.