[7/12/2020] TK373/TK374 SIM Network Degradation/Outage

News release and discussion.
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

[7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by ATSupport »

We were just notified by our SIM card carrier that there's a network degradation since about 2:10PM EDT.
This affected the TK373/TK374 users in certain areas only. The SIM provider network engineers are working on the issue and this thread will be updated for latest status.
We sincerely apologize for the inconvenience and thank you for your patience.

[Update 7/14/2020 7:06M EDT]:
We see users from most US regions are now online. Some trackers might take more time as it wakes up when the vehicle starts moving. Some trackers are actively uploading "cached data" from yesterday at the moment, in which case you'll see a "floppy disk" icon in the status window on the tracking page, with the location timestamped earlier. To view the locations collected during the outage, please go to the history page then click the "calendar" icon on the bottom and choose the time range. Cached data may take some time to get fully uploaded, so if you don't see them yet please check again later.

If your trackers are still offline, please contact us directly and we can reset the data plan and reboot them remotely: https://secure.accutracking.com/contact.php

Quote from the SIM network provider 7/12/2020:
Service Degradation 12th July 2020

This is to notify you that the data network is experiencing a service degradation. The network engineers are investigating the reason for outage, updates will be shared as soon as they become available.

We apologize for the inconvenience caused.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: to be delivered
[Update from SIM provider 5:12PM EDT]:
Our engineers have identified the issue, and they are working to resolve it, as soon as we have and ETA we will provide the information.
[Update from SIM provider 6:40PM EDT]:
We are continuing to experience a network outage with our data service.
We are still working to resolve the issue and we will provide further updates as information becomes available.
We apologize for the inconvenience caused.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: to be delivered
[Update from SIM provider 8:42PM EDT]:
We are continuing to experience a network outage with our data service.
We are still working to find a swift resolution, this is being treated as critical.
We apologize for the inconvenience caused.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: to be delivered
[Update from SIM provider 10:09PM EDT]:
We are continuing to experience a major network outage on our data service.
Our engineers are seeing a gradual improvement on their side. We are manually rerouting traffic as we speak with the aim of fully restoring services shortly.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: to be delivered
[Update from SIM provider 12:12AM EDT]:
We are seeing some improvement after the data service degradation and are currently monitoring closely whilst liaising with the network engineers in charge of bringing up the network to normal traffic levels.

Due to the scale of this outage and number of devices coming back online, this service restoration is currently gradual.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: TBD
[Update from SIM provider 1:44AM EDT]:
Through close liaison with the network engineers, we haven't been seeing much improvement since around 30% of traffic was restored around two hours ago. We have however just received confirmation that 2G and 3G traffic should be starting to return to normal - Due the huge numbers of devices connecting, we expect that the improvement that you will see will be gradual at first.

Please note that we haven't received confirmation regarding 4G LTE just yet. This is still with the engineering team.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: TBD
[Update from SIM provider 3:24AM EDT]:
We now expect to see more and more 2G and 3G devices able to start a data session – Traffic should gradually return to normal. In the last 30 minutes, we are seeing a massive improvement.

4G LTE status is still critical. The links to the LTE provider were overloaded once the devices tried to connect all at once so the engineers are working directly with the LTE provider to resolve this as soon as possible. They expect to see a major improvement in the next hour for LTE.
[Update from SIM provider 5:06AM EDT]:
Whilst 2G and 3G is restored, the network is working on increasing link bandwidth in order to allow for more devices to connect per second
due to the huge numbers of devices connecting.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: TBD
[Update from SIM provider 7:09AM EDT]:
We are now able to see an increase in the number of devices that are reconnected, however depending on the nature of each device and congestion in the region not all devices will reconnect at the same time.

Start Time: Sunday , July 12th 2020 13:10 CST
End Time: TBD
We now see trackers gradually coming back online. Your tracker will also start uploading the history data collected during the outage. From the history page you may go back to the outage time and view history data once the tracker finishes uploading the history data.

[Update from SIM provider 8:11AM EDT]:
The number of devices that are reconnected continue to increase slowly, we have now reached the 50% of the total recovery. Please note that this depends on the nature of each device and also it´s location as some regions continue to experience a higher congestion in their networks than others, therefore not all devices will reconnect at the same time as mentioned in our previous notification.

Our engineers are still continuing to monitor the situation very closely and will update with further developments as they become available.

We apologize for the situation and will continue to share with you all the new pieces of information we receive to keep you informed.
[Update from SIM provider 10:35AM EDT]:
The engineers have increased bandwidth on their diameter links (4G) and are able to see their end a positive impact on service restoration, however not all devices and locations are restoring the service at the same time due to the congestion that have been caused on the network I am afraid

We are pushing the network engineers for new updates so that we can get this situation resolved as soon as possible, we share updates as soon as they are made available to us.

Our apologies on behalf of the team for the situation this is causing you.
[Update from SIM provider management 10:46AM EDT]:
I would like to reach out to you all directly in order to give you an update, this is truly an unprecedented event, one that we have not seen the like of in our history as an MVNO.

The fault is upstream of all of our systems but I can assure you that we are doing all we can to ensure that services return to normal levels as soon as possible and we will keep you as up to date as is possible.

The nature of this event/severity of it means that the recovery we are seeing a slower than we would expect due to sheer number of devices trying to reconnect at one time.

We fully understand the impact that this type of incident has on your business, as it does on ours. We pride ourselves on trying to provide the most resilient and reliable solution possible for IoT given its critical nature, and I would, therefore, like to reassure you again that this is being treated as an absolute priority and we are doing everything in our power to return the service to an acceptable level.
...
...
[Update from SIM provider 3:07PM EDT]:
Factual situation & timeline of events:
- At around 7pm UTC on Sunday, July 12th, we experienced a network incident that impacted all fixed and mobile services.
our engineers worked through the night to fix the issue and an investigation is underway as to the root cause taking logs from across the various network elements.
- Investigation showed that IP core routers lost their capacity to communicate between each other.
- As the first routers were updated, services started to come back online during the course of the early hours.
- All links are now up, all services carried over those links are in the process of recovery.

Executive summary of the incident:
- This failure was likely caused by a synchronization clock in the network starting to transmit the wrong date and time to all the core network switching equipment. This caused linked equipment to react by disconnecting itself from our network and to restore service required a physical visit to network switches, where they were manually reset. The need for this manual reset by engineers physically on site, is the major contributing factor to the length of time it took to restore service.
- We are investigating why the network clock started to transmit the wrong time in the first place and to fully analyze all of the system logs and time-stamp data produced from the incident.
- We are continuing to see a steady increase of traffic across global operators. However some networks were recovering more slowly due to volume of signalling traffic. The network team is working to ensure the routes are fully operational as soon as possible.

Is this a one off & what actions are taken to avoid a repeat?
- The actions taken to restore service mean that a second failure through the same cause in the future is highly unlikely.
- However, clearly this has been a major service outage and it will take network engineers some time to work through the root cause analysis with their own suppliers.
In the meantime, we are closely monitoring the network and our engineers remain on-site to ensure there will be no repeat.
[Update from SIM provider 7:06PM EDT]:
Links are still in the process of recovery. We are seeing a gradual return online for many sims, however a full resolution is still to be determined. We are now focusing our efforts on restoring full service with urgency. We also continue to work with the network engineers to ensure the routes are fully operational to the networks as soon as possible.
[Update from SIM provider management 11:21PM EDT]:
Network traffic is still in the process of recovery. We still are seeing a gradual return online for many sims globally, however a full resolution is still to be determined. We still working with operators to restore traffic. We continue to focus our efforts on restoring full service for all networks with complete urgency. During the past half an hour, we are seeing some improvement globally.

As we continue to attempt to fully restore services globally I wanted to write to you all and offer my sincerest apologies for any inconvenience that this has caused and continues to cause your businesses.

I understand that without action, words can be pretty meaningless, but I can assure you that my team and I are working tirelessly to ensure that wherever possible service is returning to expected levels as quickly as possible.

As mentioned previously, this is an unprecedented event of a magnitude that we have never, fortunately, experienced in our company's history, and therefore there will be a lot for us to unpack with our network partner in order to be able to provide you with a satisfactory report, and more importantly, long term solutions to ensure that this kind of incident can't occur again, or if it does, that there are ways in which it can be resolved with little to no impact on your service.

We are seeing significant improvements around the world and in many cases services have been fully restored as our network partner continues to work with the remaining networks previously listed, That said particularly in the USA we are still experiencing a significant amount of disruption and this has now become our priority.
[Update from SIM provider 1:14AM EDT]:
We are speaking regularly with the network engineers but do not have a major update at this time. We are still monitoring traffic levels closely and reporting back findings to the network. There has been improvement on some networks in the past few hours which were previously showing very little traffic.

We shall continue to provide regular updates.
[Update from SIM provider 4:04AM EDT]:
We are seeing varying levels of traffic in different regions - Some areas have returned to normal after the service degradation and some areas are remaining the same with no improvement.

From our logs and traffic graphs, we have a clear picture of the current situation and are using this to communicate with the network engineers.

We are currently meeting with the HQ network management. We are hoping that these meetings and engineering calls will result in more meaningful updates and, most importantly, visible improvement.
[Update from SIM provider 7:06AM EDT]:
Our mobile core network provider has been working to continue recovery of the services affected from their network outage. 4G LTE services appear to be stabilizing. For 2G and 3G services due to the number of devices which have been attempting to reconnect to the network has been causing congestion in the network, thus causing instability and a degradation of service.The network engineering team and signalling provider have applied additional rate limiting to help alleviate the congestion and help to bring these services back online in a controlled manner.

As of 00:20 GMT We are seeing a positive improvement in traffic from this work as devices reconnect. We will continue to monitor and work with the network team to manage this situation.
We see users from majority of regions in the US are now online. Some trackers might take more time as it wakes up when the vehicle starts moving. Some trackers are actively uploading "cached data" from yesterday at the moment, in which case you'll see a "floppy disk" icon in the status window on the tracking page, with the location timestamped earlier.

If your trackers are still offline, please contact us directly and we can reset the data plan and reboot them remotely: https://secure.accutracking.com/contact.php
AccuTracking Support
support(at)accutracking.com
Stln3

Re: [7/12/2020] TK373/TK374 SIM Network Degradation

Post by Stln3 »

Well I hope they fix it soon because I didn’t pay the monthly fee for nothing!!!!!
Sumasshasmysn

Re: [7/12/2020] TK373/TK374 SIM Network Degradation

Post by Sumasshasmysn »

Have they fixed this yet, or close??????
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

Re: [7/12/2020] TK373/TK374 SIM Network Degradation

Post by ATSupport »

Sumasshasmysn wrote: Sun Jul 12, 2020 8:55 pm Have they fixed this yet, or close??????
They are still working on it...the latest update from them was the 8:42PM EDT one pasted in the first post.
Really appreciate your patience and apologies again.
AccuTracking Support
support(at)accutracking.com
Stln3

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by Stln3 »

When they gonna fix it I’m gonna
f242ss

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by f242ss »

This is taking way to long to fix!!
When my units come back online will I be able to see all the movements during the outage??
How much can they store??
Guest

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by Guest »

i have a daughter leaving from nc to georgia this morning hope its fix soone. :evil:
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by ATSupport »

f242ss wrote: Mon Jul 13, 2020 5:55 am This is taking way to long to fix!!
When my units come back online will I be able to see all the movements during the outage??
How much can they store??
Yes you will - TK37x trackers have the capability of "cache and send later" when there's no cell network connection. When network recovers later, the data will be retried and re-sent. It can store 250~300 waypoints.

Thank you for your patience and we sincerely apologize for the scale of the situation.
AccuTracking Support
support(at)accutracking.com
f242ss

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by f242ss »

So what is the latest update? When will the system be back up and working?
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by ATSupport »

f242ss wrote: Mon Jul 13, 2020 8:13 am So what is the latest update? When will the system be back up and working?
We just got the latest update from the SIM provider 5 minutes ago and updated the first post - 50% connections restored.
Thank you so much for your patience.
AccuTracking Support
support(at)accutracking.com
f242ss

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by f242ss »

Well I guess my units were not included in that 50% any idea when Florida will be restored..
zpilot67

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by zpilot67 »

It appears that my entire 30 mile radius is down for 61565 and 61540. Not a single update today. I have had update and coverage issues since I installed this unit about a week ago. I hope all this work helps the coverage or I'll have no choice but to look for a different provider which I'd rather not do again.
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by ATSupport »

f242ss wrote: Mon Jul 13, 2020 8:38 am Well I guess my units were not included in that 50% any idea when Florida will be restored..
Can you please provide your account info at https://secure.accutracking.com/contact.php so we can look into your units.
AccuTracking Support
support(at)accutracking.com
ATSupport
AccuTracking Staff
Posts: 1906
Joined: Tue Jan 27, 2004 4:36 pm
Contact:

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by ATSupport »

zpilot67 wrote: Mon Jul 13, 2020 8:50 am It appears that my entire 30 mile radius is down for 61565 and 61540. Not a single update today. I have had update and coverage issues since I installed this unit about a week ago. I hope all this work helps the coverage or I'll have no choice but to look for a different provider which I'd rather not do again.
Apologies and completely understand your frustration - can you please provide your account info via https://secure.accutracking.com/contact.php for investigation. This time the outage is related to SIM network only and not the devices, but if the issues was from a week ago, it could be device related. We have life-time warranty on the hardware and will be able to check/replace it asap.
AccuTracking Support
support(at)accutracking.com
Guest

Re: [7/12/2020] TK373/TK374 SIM Network Degradation/Outage

Post by Guest »

I sent you my account information please let me know what you find out.
Post Reply