Memory Issues? (Notes Stop Adding but Session Notefiles Continue)

Troubleshooting
1

Hi all,

I am experiencing an issue in which my device continues to upload session notefiles hourly but stops adding notefiles that include my sensor data after a certain period of time (has been anywhere between a few days to a few weeks - it is very inconsistent between devices despite using same exact hardware setup, host firmware and notecard firmware versions). Performing a power restart of the device causes notes to start being added again.

I suspected this was a memory issue at first because initially I failed to include “NULL checks” (if (req != NULL)) in my code before adding notes. After incorporating this, I did not encounter the issue again, but meanwhile was unable to reproduce the error with a test code that does not have these “NULL checks”, so my confidence in the suspected fix dropped.

After running into the issue again, I found something that may provide a bigger clue as to what is going on.

After I noticed this morning that my notefiles with sensor data stopped uploading yesterday morning, I tried to see what updating the notecard firmware (from 6.1.1 to 6.2.2) would do. I encountered this error in the device health messages:

Mon 02:19:48 PM: boot (dfu: encountered {memory} dfu:1233 [16332] [4.0] G:Gp:W~ M:Mp:W900 U:W0 R:Rp:W~ S:Spdd T I:H~ W X Y)

I retried the update and it seemed to be successful:

Mon 02:32:40 PM: boot (relaunch after DFU [16520])
Mon 02:31:05 PM: DFU notecard firmware version updated to: {'org':'Blues Wireless','product':'Notecard','target':'u5','version':'notecard-u5-6.2.2','ver_major':6,'ver_minor':2,'ver_patch':2,'ver_build':16520,'built':'Apr 10 2024 16:03:28'}
Mon 02:28:12 PM: DFU notecard firmware download started from from notecard-u5-6.2.2.16520$20240410160742.bin

The boot did not cause my notefiles with sensor data to be added again.
To me, this suggests my device is having memory issues.

Here is some more info about my setup:

  • I am using NOTE-WBGLW

  • Notecard communication is set to serial, hub config is:

  J *req = notecard.newRequest("hub.set");
  JAddStringToObject(req, "product", productUID);
  JAddStringToObject(req, "mode", "continuous");
  JAddNumberToObject(req, "duration", 60);
  JAddNumberToObject(req, "inbound", 60);
  JAddBoolToObject(req, "sync", true);
  notecard.sendRequestWithRetry(req, 5);  // 5 seconds
  • Firmware was updated from 6.1.1 to 6.2.2 (see above)
  • My host firmware relies on the use of 6 environment variables, including wifi SSID & password
  • I am adding 2 notes (one quite large with arrays with nDOTS = 50; template created in setup function) every 60 seconds like this:
    {
      J *req = NoteNewRequest("note.add");
      if (req != NULL) {
        JAddStringToObject(req, "file", "velocity.qo");

        J *arr = JCreateArray();
        J *arr2 = JCreateArray();
        J *arr3 = JCreateArray();
        J *arr4 = JCreateArray();
        J *arr5 = JCreateArray();
        J *arr6 = JCreateArray();
        J *arr7 = JCreateArray();

        for (int i = 1; i <= nDOTS; i++) {
          JAddItemToArray(arr, JCreateNumber(vRms[i][0]));
          JAddItemToArray(arr2, JCreateNumber(vRms[i][1]));
          JAddItemToArray(arr3, JCreateNumber(vRms[i][2]));
          JAddItemToArray(arr4, JCreateNumber(vPeak[i][0]));
          JAddItemToArray(arr5, JCreateNumber(vPeak[i][1]));
          JAddItemToArray(arr6, JCreateNumber(vPeak[i][2]));
          JAddItemToArray(arr7, JCreateNumber(vTime[i]));
        }
        J *body = JAddObjectToObject(req, "body");
        if (body != NULL) {

          JAddItemToObject(body, "v_RMS_x", arr);
          JAddItemToObject(body, "v_RMS_y", arr2);
          JAddItemToObject(body, "v_RMS_z", arr3);
          JAddItemToObject(body, "v_PEAK_x", arr4);
          JAddItemToObject(body, "v_PEAK_y", arr5);
          JAddItemToObject(body, "v_PEAK_z", arr6);
          JAddItemToObject(body, "velocity_time", arr7);
        }
        JAddBoolToObject(req, "sync", true);
        notecard.sendRequest(req);
      }
    }

    delay(100);

    {
      J *req = NoteNewRequest("note.add");
      if (req != NULL) {
        JAddStringToObject(req, "file", "sensor.qo");
        J *body = JAddObjectToObject(req, "body");
        if (body != NULL) {

          JAddNumberToObject(body, "ambienttempC", temp.x);
          JAddNumberToObject(body, "objecttempC", temp.y);
          JAddNumberToObject(body, "ambienttempF", temp.x * 9 / 5 + 32);
          JAddNumberToObject(body, "objecttempF", temp.y * 9 / 5 + 32);
          JAddNumberToObject(body, "current1", current.x);
          JAddNumberToObject(body, "current2", current.y);
          JAddNumberToObject(body, "current3", current.z);
          JAddNumberToObject(body, "SAMPLE_RATE", sensorInterval);
          JAddNumberToObject(body, "RunTime_period", RunTime_sum);
        }
        JAddBoolToObject(req, "sync", true);
        notecard.sendRequest(req);
      }
    }

Can anyone help me troubleshoot identify the root cause of this issue? I am happy to provide any additional info that may help. Is there anything I am missing?

Hoping to install these devices with customer trials next month but cannot move forward until we are confident we won’t have issues like this.

Thank you!!!

2

Hi @tsuzenski,

We’ve been looking over this for a bit now to try and identify the source issue. One issue (though not likely a culprit) is the use of both note-c and note-arduino syntax in your requests. The difference is subtle and very easy to mix up, but:

note-c

J *req = NoteNewRequest("note.add");

note-arduino

J *req = notecard.newRequest("hub.set");

I’d recommend sticking with the note-arduino syntax consistently.

Now the likely culprit is the plain fact that the Notecard ran out of memory. Clearly you’re putting a lot of Notes in a Notefile and the structure of the Notes themselves are non-trivial (with arrays and sub-objects) and thus take up a lot of Notecard memory.

If not doing so already, you should start by creating Notefile templates. However, this size/complexity of your template would really push the bounds of what the Notecard can handle (but it’s worth testing).

There are other options too:

  1. Try breaking up the Notes into smaller requests (ideally Notefile sizes are around 1KB or less).
  2. You can try using web transactions which are tuned for larger objects and have a nice side effect of not persisting data to local flash.
  3. You can use the card.binary APIs to binary-encode the data and deliver the payloads more efficiently.

2 and 3 aren’t nearly as convenient as your note.add workflow, but you’re pushing a lot of data (relatively speaking).

Hope this helps!
Rob

1 Like
3

Thank you so much, Rob - this is helpful! Yeah, I am pushing a lot of data. :grimacing: I will look into what you said and try out your suggestions!

I do have a template being created in my setup function for the velocity.qo file but not for my sensor.qo file.