Resolved -
Following an extended period of monitoring, both our provider's data and our own shows that connections and data usage remain stable.
We believe this incident is now resolved. Please contact our support team with a list of impacted ICCIDs to arrange a refund.
We are conducting a postmortem and will share our findings once we have a full understanding of the cause and impact. We are very sorry for the disruption this has caused to your business.
Jun 10, 09:30 UTC
Update -
Following successful implementation of a fix from our upstream provider around 4pm, we have not been made aware of ongoing issues and we are seeing data connections establish within normal parameters.
If your customers are still experiencing problems, please contact us. Our 24/7 teams continue to monitor stability and remain on standby as per our SLA. We will begin processing refunds tomorrow once the incident is classed as completely resolved.
We apologise for the impact this may have had on your customer teams and your customers.
Jun 9, 21:41 UTC
Update -
We are observing a stable service with all new orders, however, we are still receiving intermittent reports related to eSIMs ordered before the outage which started at 10:00 UTC on 09/06. This has been raised to our upstream provider for escalation and investigation.
If you are facing an outstanding issue with an eSIM, please issue a replacement eSIM and provide any impacted ICCIDs to our support team, who will be able to issue a refund to your account.
Jun 9, 18:16 UTC
Update -
Our upstream provider has successfully rerouted traffic to the alternate route and we are waiting for the cloud provider to confirm that the emergency maintenance window was successful.
We are starting to see traffic levels increase again but will continue to monitor until traffic returns to normal.
Jun 9, 15:19 UTC
Update -
We have received confirmation from our upstream mobile operator that the fix was not able to restore service to full capacity. The mobile operator is now rerouting traffic to an alternate route on their network which should improve stability, ahead of a full fix.
We have our mobile operator, their cloud provider and our OCS vendor all working on this and will continue to monitor traffic as we progress.
The issue arose during emergency maintenance related to Cloud Direct Connect infrastructure between the Mobile Operator and our OCS vendor. An emergency maintenance window has been put in place by the cloud provider that started at 11 and is now expected to last until 15:00 UTC. We will provide a further update when this is complete. Once service is resumed, we will be conducting a complete RCA and will share this.
Jun 9, 13:42 UTC
Update -
Following a period of monitoring, we have not seen traffic return to normal so we have escalated this with our upstream provider.
They are implementing another fix and we will be able to provide a further update at 13:30 UTC.
Jun 9, 12:49 UTC
Monitoring -
Upstream have put a fix in place to restore service. Traffic is increasing towards normal operational levels.
We will continue monitoring this incredibly closely and will continue to work with our provider to understand the root cause.
Further updates to this issue will be provided once we know more.
Jun 9, 11:43 UTC
Update -
The temporary 503 errors were due to our provider rebooting some systems to improve service.
The API is now fully operational.
Connectivity degradation is still being investigated.
Jun 9, 11:28 UTC
Update -
We are continuing to experience issues with our upstream providers for Profile 1 and 5. This may result in our API returning 503 "Service Unavailable" errors.
We are still investigating this and will provide updates as soon as possible.
Jun 9, 10:51 UTC
Investigating -
Our Profile 1 & 5 provider has informed us that they are currently investigating issues with degraded connectivity across their platform.
They are investigating this with urgency and will provide an update as soon as possible.
Jun 9, 10:24 UTC