Note to self

 Another few hundred emails generated by the service health script as it monitors O365 service health using the API. Whats going on? It would seem theres a rogue server handing out data for the Office 365 Service Communications API. As well as doubling up (536 messages instead of 248) there seems to be some messages missing from the 'larger' dump, in particular: IT234239 - Microsoft Intune - False positive IT234538 - Microsoft Intune - False positive MO234433 - Microsoft 365 suite - False positive TM233796 - Microsoft Teams - Service restored TM234913 - Microsoft Teams - False positive FL234254 - Microsoft Power Automate - False positive It would seem that some testing may have gone wrong and theres some test or backup data creeping in. As well as the above incidents not being in the 'core' 248 which seem 'normal', the status also changes to have white spaces removed ie "Service Degredation" and "ServiceDegredation"

 An interesting start to the week. On the 15th of January 2021, Quovadis/Digicert revoked an intermediate certificate without any apparent notification. As 'luck' would have it some patching happened shortly after and the server rebooted. The services would not come up. After some checking of new software deployments (security/management), validating patching, restoring VMs from backup it was discovered that changing the system clock back to the 14th Jan let the services start. With such a specific cut-off date/time and the service event logs (not always pointing in the right direction!) it pointed a finger at certificate revokcation. We knew the public certs weren't due to expire for 6 months and the dates on the intermediate and root cert were all fine. Some google-fu found that we were not alone AusCERT statement “QuoVadis Global SSL ICA G3” issue impacting multiple customers QuoVadis Intermediate Revoke Update | Jisc community Even though we double-checked the certifica...