[12/01/2020] TK37x SIM Network Degradation

Post a reply

Smilies
:D :) :( :o :shock: :? 8) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :wink: :!: :?: :idea: :arrow: :| :mrgreen:

BBCode is ON
[img] is ON
[url] is ON
Smilies are ON

Topic review
   

Expand view Topic review: [12/01/2020] TK37x SIM Network Degradation

Re: [12/01/2020] TK37x SIM Network Degradation

by ATSupport » Wed Dec 02, 2020 12:56 am

Thank you so much for your support and understanding! The latest update was just added to the first post. We are seeing the recovery in progress now. 4G network is recovering more quickly and 3G is improving as well.

The COVID-19 situation made the network problem solving process a bit more complicated and slower, as data center tightens the access protocol and requires COVID test clearance for engineers entering the data center. Last year network incidents were addressed quickly and never caused prolonged outage/degradation but 2020 added more challenges and uncertainty. Hopefully things get back to normal in 2021 and thank you again for being with us during this stressful time.
Shallamon wrote: Wed Dec 02, 2020 12:41 am Thanks AccuTracking for being proactive and putting out this advisory. It eases the stress and allows your users to relax knowing that you have things under control.

BTW, you have got a terrific product, it is far superior to anything I expected.

Regards,
-=| Shallamon |=-

Re: [12/01/2020] TK37x SIM Network Degradation

by Shallamon » Wed Dec 02, 2020 12:41 am

Thanks AccuTracking for being proactive and putting out this advisory. It eases the stress and allows your users to relax knowing that you have things under control.

BTW, you have got a terrific product, it is far superior to anything I expected.

Regards,
-=| Shallamon |=-

Re: [12/01/2020] TK37x SIM Network Degradation

by ATSupport » Wed Dec 02, 2020 12:31 am

Appreciate your patience - the first post was just updated with the status we received from the upstream team. They are still working on the issue. Network traffic is still not fully passing through. We are still waiting for the next update and expect a fix by tomorrow morning. Truly sorry for the disruption caused.
kvoelk20@gmail.com wrote: Tue Dec 01, 2020 11:56 pm ok so what is the latest I've been down since 11 am mountain.

Re: [12/01/2020] TK37x SIM Network Degradation

by kvoelk20@gmail.com » Tue Dec 01, 2020 11:56 pm

ok so what is the latest I've been down since 11 am mountain.

Re: [12/01/2020] TK37x SIM Network Degradation

by ATSupport » Tue Dec 01, 2020 7:40 pm

Thank you for understanding - just got another update from the provider - looks like it's related to firewall and should be restored soon.
Site tech wrote: Tue Dec 01, 2020 6:57 pm Thx for the update. hopefully is up soon

Re: [12/01/2020] TK37x SIM Network Degradation

by Site tech » Tue Dec 01, 2020 6:57 pm

Thx for the update. hopefully is up soon

Re: [12/01/2020] TK37x SIM Network Degradation

by ATSupport » Tue Dec 01, 2020 4:00 pm

Thank you for your patience - from the past experience, it usually takes a few hours before it's back.
o264833 wrote: Tue Dec 01, 2020 3:19 pm for the SIM issue is there any ETA on a fix?

Re: [12/01/2020] TK37x SIM Network Degradation

by o264833 » Tue Dec 01, 2020 3:19 pm

for the SIM issue is there any ETA on a fix?

[12/01/2020] TK37x SIM Network Degradation

by ATSupport » Tue Dec 01, 2020 2:27 pm

We were notified by our SIM card carrier that there's a network degradation since about 1:30PM EDT.
This affected the TK373/TK374 users in certain areas.
The tracker will store the data collected locally on the device when network is not available
and uploads data when network connection is recovered.

The SIM provider network engineers are working on the issue and this thread will be updated for latest status.
We sincerely apologize for the inconvenience caused.

[Update 06:45pm EST]
We have just received an update that the service in affected areas is expected to be restored soon.
Quote from the provider:
This is to notify you that network engineers are deploying a fix to resolve the service degradation on sims. We will continue to monitor our traffic levels and will send another update shortly.
[Update 07:38pm EST]
Quote from the provider:
Engineers have isolated the problem to the signaling firewall, which experienced loss of connectivity. Connectivity to the signaling firewall has been restored (20:00 UTC) and efforts are continuing to fully restore service. Further update to follow shortly.
[Update 10:47pm EST]
Quote from the provider:
The network has been bypassing a number of its signaling links and monitoring any improvements. At the same time, the engineers are engaged with data center to assist with managing the traffic.
[Update 12:26am EST]
Quote from the provider:
The engineers are analyzing different traffic types and global titles in a multi-pronged effort.
[Update 12:45am EST]
We are seeing significant improvement, 4G LTE has improved more quickly, 3G seems to be improving as well, we are hopeful that a full resolution comes shortly.

[Update 1:01am EST]
Good news from the provider:
We have been assured by the network that corrective actions are currently underway.
Through our logs, we are seeing a gradual traffic improvement at the same time.

The signaling team is increasing bandwidth manually to each of the STPs (Signal Transfer Points). The aim of this is to stabilize the links.
[Update 2:10am EST]
The network is recovering. Quote from the provider:
The signalling congestion is continuing to be alleviated and at the same time, there is marked increase in global traffic and new connections.
[Update 4:56am EST]
Network recovered. Quote from the provider:
There has been a marked period of stability since our last update but we continue to monitor. If you are still experiencing issues please do reach out to us.We will continue to monitor before the resolution is confirmed.
Network shows normal traffic now. You tracker should be back online shortly after the vehicle starts moving. History data collected during the outage period will be re-uploaded by the tracker. Thank you very much for your patience and please contact our support if there's any issue.

Top