GPS troubles: No GPS acquired, stopped working, or not detecting movement

Hi,

I have had a device being deployed for several months where the GPS has suddenly stopped working (heartbeat 1 hour). I don’t have access to the device since it is currently anchored in almost 4 meters waves offshore, but everything else appears to work as it should - transmitting data ferociously. I suspected a disconnected antenna, but that does not seem very plausible given that the modem is working.

Recently, we tried to deploy a couple of buoys in South Africa and the GPS is struggling there as well, not acquiring a fix - resulting in the clock being off (Notecard GPS message stops updating - #4 by gauteh), which is weird since we would normally use the modem time anyway ?

We also noticed that the GPS did not detect movement when deployed in the water, which is weird, since there is always some movement that should trigger the GPS. This was solved with the sub-hour heartbeat (GPS heartbeat of less than 1 hour).

We’re using version 4.4.1 (LTS) of the firmware.

Any ideas on what could be going on?

Regards, Gaute

Hi Gaute,

Sorry for the delay in responding, I needed to escalate this to one of the engineers to get the best help.

Firstly, you say that you didn’t think a disconnected antenna was plausible, as the modem was still working - it is worth noting that the GPS antenna is separate from the Cellular Modem Antenna, so it is a possibility.

Regarding the time concerns, our engineer said the following:

The Notecard gets time from cellular, GPS and from Notehub each time it syncs. The clock is then propagated by the CPU’s RTC using a precision crystal - so the clock should not drift more than a few seconds per day. Significant clock errors can occur if the voltage supply to the Notecard browns out - falls below 2.5V, but provided there is a constant voltage supply > 2.5V the clock should not have errors, If there are errors we would need to see when the clock errors occurred and examine the GPS, sync and voltage history of the Notecard to troubleshoot.

The best thing we can suggest though, which will help diagnose both the lack of movement detection, and any issues in getting GPS fixes, is to enable some debugging on the notecard. To do this set the _log environment variable to gps for the notecard(s) you want to debug, and they will then send diagnostics in the _log.qo note which will get synced to notehub. I think you have used these logs in the past, from looking at your previous forum posts.

Do this, and we should be able to help you further.

Thanks, Kimball

Thanks, that’s a good tip. I’ll enable the logs and report back. I was also thinking about triggering a notecard reboot by doing a firmware upgrade, but maybe there’s some env variable I can use to do that too?

No more diagnostics in the log. The accelerometer is giving a lot of errors, but the _track.qo events do not contain gps position. Additionally, my firmware is now reporting an invalid request when trying to get the position (no further information available), but it does a card.location.

an hour ago: accelerometer IO errors: 43/123445 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 0
an hour ago: accelerometer IO errors: 6/123402 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
an hour ago: accelerometer IO errors: 6/123396 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
an hour ago: accelerometer IO errors: 6/123390 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
an hour ago: accelerometer IO errors: 6/123384 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
an hour ago: accelerometer IO errors: 162/123378 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
2 hours ago: accelerometer IO errors: 33/123216 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 0
2 hours ago: accelerometer IO errors: 4/123183 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
2 hours ago: accelerometer IO errors: 6/123179 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
2 hours ago: accelerometer IO errors: 6/123173 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
2 hours ago: accelerometer IO errors: 6/123167 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07 07
2 hours ago: Fatal error in main loop: location: Err(InvalidRequest), note/drain_queue: Ok(188416), note/check_and_sync: Ok(()). Tries left: 14

I had two devices where the GPS stopped updating. One got its last position 7 months ago. After restating (Understanding Environment Variables - Blues Developers) remotely it started working again on both. The device is moving as it is in the ocean.

I still suspect that there is a bug somewhere in the notecard gps penalty / ratio that causes this to happen.