Sending Data from CSV file to Notehub

Hello, I am working on a project that uses BME280, a Micro SD card, Swan, and Notecarrier F. I am taking sensor readings every minute for a day and saving the data as a CSV file on the SD card(also including the date, time, latitude and longitude from the GPS). The idea is to sync the data to the Notehub every midnight daily. I want to keep track of the parameters (temp, pressure, humidity) over time. I can read the sensor data and save it to the sd card. Not sure how to go about uploading it to notehub. I have the code below but its not working as expected.

#include <Wire.h>
#include <Notecard.h>
#include <TimeLib.h> 
#include <SPI.h>
#include <SD.h>

#define SD_CS 10
#define MEASUREMENTS_PER_NOTE 16
#define productUID "xxx"  
Notecard notecard;
File myFile;
bool dataAvailable = true;

void setup() {
    Serial.begin(9600);
    Wire.begin();
    notecard.setDebugOutputStream(Serial);
    notecard.begin();

    if (!SD.begin(SD_CS)) {
        Serial.println(F("SD Card Error!"));
        while(1); // Consider modifying this to avoid infinite loop
    } else {
        Serial.println(F("SD Card detected."));
    }

     J *req = notecard.newRequest("hub.set");
    JAddStringToObject(req, "product", productUID);
    JAddStringToObject(req, "mode", "periodic");
    JAddNumberToObject(req, "outbound", 10);//every 30 mins
    notecard.sendRequest(req);

    myFile = SD.open("DATA/DATA000.csv");
    if (!myFile) {
        Serial.println("Error opening DATA/DATA000.csv");
        return; // Exit if file cannot be opened
    }

    registerTemplate();
}

void loop() {
    if (dataAvailable && myFile.available()) {
        sendMeasurementsNote();
    } else {
        // No more data to read; stop further processing
        dataAvailable = false;
        Serial.println("All data has been sent.");
    }
    delay(2000);
}

void registerTemplate() {
    J *req = notecard.newRequest("note.template");
    JAddStringToObject(req, "file", "sensors.qo");

    J *body = JCreateObject();
    J *dataArray = JAddArrayToObject(body, "measurements");
    
    JAddItemToArray(dataArray, JCreateString("string")); // Ensure correct type definitions are used
    JAddItemToObject(req, "body", body);

    if (!notecard.sendRequest(req)) {
        Serial.println("Failed to send note.template request.");
    }
}

void sendMeasurementsNote() {
    J *req = notecard.newRequest("note.add");
    JAddStringToObject(req, "file", "sensors.qo");

    J *body = JCreateObject();
    J *dataArray = JAddArrayToObject(body, "measurements");

    String measurement = myFile.readStringUntil('\n');
    JAddItemToArray(dataArray, JCreateString(measurement.c_str()));

    JAddItemToObject(req, "body", body);

    if (!notecard.sendRequest(req)) {
        Serial.println("Failed to send note.add request.");
    } else {
        Serial.println("Measurements sent successfully.");
    }
}

Hi @chadha_14,

I would approach this in a different way. You can certainly continue to write data to the CSV file and store it on an SD card, but there is no need to create a process that reads that data and sends it to Notehub. Instead, you can continue to use the note.add API to save data to flash storage on the Notecard (that request doesn’t automatically sync data with the cloud, just saves it to flash). Using the hub.set API you can tell the Notecard to sync any pending data with Notehub every hour, every day, whatever you want to do. (You’re already using a Note Template, so that’s good!)

Basically the changes you’d need to make are:

  1. Change your hub.set to a larger outbound value (right now you’re syncing every 10 mins).
  2. Instead of one large measurements array that includes all your values, add separate values to the body for every individual sensor you are using. Will make it easier to parse IMO.
  3. Make sure any changes to #2 are reflected in your note.template call as well.

Hope that helps!
Rob

1 Like

Thanks, @RobLauer. This does make sense. I’ll test it out. Unfortunately, my GPS is no longer able to get a fix even after 5 hours. I tried leaving it outside my office for some time, but that didn’t work. This is my latest attempt at cracking this. Any idea what could be the issue?
I keep getting this error

[ERROR] serial-over-i2c: no response to read request {io}
[WARN] serial-over-i2c: reattempting to read Notecard response
[ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
[WARN] serial-over-i2c: reattempting to read Notecard response
[ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
[WARN] serial-over-i2c: reattempting to read Notecard response
[ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
[WARN] serial-over-i2c: reattempting to read Notecard response
[ERROR] i2c: unknown error on TwoWire::endTransmission() {io}
[ERROR] error occured during receive
[WARN] retrying I/O error detected by host

and the one below here

[INFO] {"stop":true,"status":"GPS search (304 sec, 0/0 dB SNR, 0/0 sats, HDOP 0.00) {gps-active}","mode":"continuous"}
[INFO] {"req":"card.location","crc":"00C0:F0AE3444"}
[INFO] {"stop":true,"status":"GPS search (306 sec, 0/0 dB SNR, 0/0 sats, HDOP 0.00) {gps-active}","mode":"continuous"}
[INFO] {"req":"card.location","crc":"00C1:F0AE3444"}
[INFO] {"stop":true,"status":"GPS search (309 sec, 0/0 dB SNR, 0/0 sats, HDOP 0.00) {gps-active}","mode":"continuous"}
[INFO] {"req":"card.location","crc":"00C2:F0AE3444"}
[INFO] {"stop":true,"status":"GPS search (311 sec, 0/0 dB SNR, 0/0 sats, HDOP 0.00) {gps-active}","mode":"continuous"}
[INFO] {"req":"card.location","crc":"00C3:F0AE3444"}
#include <Notecard.h>
#include <Wire.h>
#include <TimeLib.h>


#define usbSerial Serial
#define productUID "xxx"  
Notecard notecard;

double lat;
double lon;
unsigned long gps_time_s;
unsigned long current_time_s;
const int timeZoneOffset = -4;
String actual_time;

void setup() {
    delay(5000);
    usbSerial.begin(115200);
    notecard.begin();
    notecard.setDebugOutputStream(usbSerial);

    J *reqs = NoteNewRequest("card.restore");
    JAddBoolToObject(reqs, "delete", true);
    JAddBoolToObject(reqs, "connected", true);
    NoteRequest(reqs);


    J *req = notecard.newRequest("hub.set");
    JAddStringToObject(req, "product", productUID);
    JAddStringToObject(req, "mode", "periodic");
    JAddNumberToObject(req, "outbound", 1); // Outbound sync every minute
    notecard.sendRequest(req);

    J* req1 = notecard.newRequest("card.location.mode");
    JAddStringToObject(req1, "mode", "periodic");
    JAddNumberToObject(req1, "seconds", 60); // Update location every minute
    notecard.sendRequest(req1);
}

void loop() {
    fetchGPSLocation();
    actual_time = formatDateAndTime();
    // Print the location and current time
    usbSerial.print("Time = ");
    usbSerial.print(actual_time);
    usbSerial.print(" Latitude = ");
    usbSerial.print(lat);
    usbSerial.print(" Longitude = ");
    usbSerial.println(lon);
    delay(60000); // Delay 1 minute before fetching location again
}

void fetchGPSLocation() {
    J *req = notecard.newRequest("card.location.mode");
    JAddStringToObject(req, "mode", "continuous");
    notecard.sendRequest(req);

    size_t timeout_s = 120;  // Timeout in seconds for GPS fix
    size_t start_ms = millis();

    while (millis() - start_ms < timeout_s * 1000) {
       while (lat == 0 || lon == 0) {
        J *rsp = notecard.requestAndResponse(notecard.newRequest("card.location"));
        if (rsp && !notecard.responseError(rsp)) {
            unsigned long current_gps_time_s = JGetInt(rsp, "time");
            if (current_gps_time_s != gps_time_s) {  // Check for new location data
                lat = JGetNumber(rsp, "lat");
                lon = JGetNumber(rsp, "lon");
                gps_time_s = current_gps_time_s;
                NoteDeleteResponse(rsp);

                delay(5000); 

                
                // Fetch the current time after a successful GPS fix
                J *req2 = notecard.newRequest("card.time");
                J *time_rsp = notecard.requestAndResponse(req2);
                if (time_rsp && !notecard.responseError(time_rsp)) {
                    current_time_s = JGetInt(time_rsp, "time");
                    NoteDeleteResponse(time_rsp);

                } else {
                    usbSerial.println("Failed to fetch current time.");
                    if (time_rsp) NoteDeleteResponse(time_rsp);
                }
                
                break;
            }
            NoteDeleteResponse(rsp);
        }
        delay(2000);  // Delay between retries
    }

    if (millis() - start_ms >= timeout_s * 1000) {
        usbSerial.println("Timed out looking for a location.");
    }

    // Restore to periodic mode after attempting to get location
    req = notecard.newRequest("card.location.mode");
    JAddStringToObject(req, "mode", "periodic");
    notecard.sendRequest(req);
}
}
String formatDateAndTime() {
  char dateBuffer[30];
  sprintf(dateBuffer, "%04d/%02d/%02d %02d:%02d:%02d", year(), month(), day(), hour() + timeZoneOffset, minute(), second());
  return String(dateBuffer);
}

Hi @chadha_14,

There are a couple of potential issues here:

  1. Looks like you’re having I2C issues, which means you may want to double check your wiring. You should also make sure you’re on the latest version of the Notecard firmware.
  2. You’re in hub.set/periodic mode, but your outbound is 1 minute. That’s effectively putting the Notecard in continuous mode.
  3. You’re trying to get a GPS fix every 1 minute, also effectively setting card.location.mode to continuous. This conflict is likely what is causing these issues. The Notecard is switching back and forth trying to get a cell connection and GPS connection, likely without being able to do either.

If you need concurrent cell/GPS connectivity, check out this blog on using an external GPS module.

I would set your hub.set/outbound argument to something high (like 30). IMO you can also just set your card.location.mode to continuous as well (assuming you really want to get regular location updates saved to the Notecard). You can then get rid of the fetchGPSLocation method entirely, as the Notecard handles all of this for you automatically.

Rob

1 Like