I have the attached a log of events from the notehub. On the Tuesday (start of this log) , I sent a note to the note card (I use the interupt method to retrieve notes from the notecard via raspberry pico). The note was picked up and processed ok around 9am that morning. Today, I sent a note around 12:45 and there was no response from the notecard. I checked the events log and see several
{“why”:“continuous connection mode (TLS)”}
I have attached a pdf of the log of events since I last sent and successfully retrieved note
According to the notehub, the last time the card connected to the hub was only minutes before my posting a note today, so it is clear that the notecard is speaking to the notehub. The problem is at some point the interrupt listening for notes is no longer working. The config on the notecard end is a pico connected to a notecard.
Any suggestions as to what might be going on?
An update… about 50 minutes since sending both notes to the notehub. The expected twillio sms arrived. This is very odd as I normally receive a response within minutes?
Some additional details
For the note creation, the upload time and capture time are the same for each note
captured upload
note 1 12:46:22 same
note 2 12:49:49 same
The response from the note card, the capture and upload are significantly different
captured Upload
Note 1 response 12:46:25 1:36:44
Note 2 response 12:49:52 1:36:45
The good news as that a response was received for notes, so the interupt was working fine. The unexplained is why such a long time between seeing the note and processing a response. The syncing I have the note card set up for is continuous, so I am not clear what is happening, hopefully someone has some idea?
Hi @rksteeves,
Can you please post the hub.set command and an example note.add you use on this device?
Thanks,
Rob
Hi Rob,
I believe the hub.set is something like
{ “req”:“hub.set”, “mode”:“continuous”, “sync”:“true”}
The card is in a remote location a couple of hours from me so I cant get to the actual card and for some reason my notes on this are not clear. Although just so you know this has been working fine for some time.
The add note is the following python code:
data = ‘{“body”: {“TempCheck”: “On”}}’
headers = CaseInsensitiveDict()
headers[“X-SESSION-TOKEN”] = mytoken # token genenrated elsewhere
url = “https://api.notefile.net/v1/projects/app:xxxxx/devices/dev:xxxx/notes/notes.qi”
response = requests.post(url, headers=headers, data=data)
Again this has been working for some time. And, some time after the 2 responses came through, so both messages generated and interrupt and were serviced, but the actual upload was delayed. At the same time I checked the notehub and my random checking of the notehub status of the card seems to show less than 10 mins since last contact, normally minutes.
I hope this info helps
Can you clarify the overall workflow of this solution just so I can understand what’s happening? It sounds like:
- You’re sending an inbound Note (your Python code above) to Notehub that sets
TempCheck to "On".
- Since
hub.set is in continuous mode with sync:true, the Notecard on your device should be immediately (within minutes) connecting with Notehub to download data.qi.
- Are you then processing something on the Notecard device to send a new Note back to Notehub which hits a Twilio route?
It may be helpful for you to add me as a viewer to your Notehub project temporarily: rlauer@blues.com
Thanks,
Rob
Happy to add you as a viewer, can you give me directions?
The workflow is as follows:
The notecard is connected to a pico and the pico is waiting on a note interrupt. On interrupt the pico will process the note and execute one of the following
- send temperature via sms text
- turn on/off relay for furnace on/off & send a confirmation through sms
- create hourly temps notes that are routed to ubidots
From my location, I create via pyhon the required note: temp check sms, relay on/off, track temp
In general result of the process is as you have identified is almost immediate. When I create A note, I generally receive a response within less than a minute
Yes, the responses back from the notecard are for each of the actions:
- for temp check create a note which has routing to twillio
- for relay on/off, create a note which routes to twillio and create a note each hour that is sent to ubidots
The above has all worked pretty well, with the exception of the outage that occurred a while ago.
I hope this helps. Let me know what I need to do to add you to the project.
In your Notehub project, go to Settings → Members and click the “Invite Members” button in upper right corner.
When creating the outbound Notes on the Pico, do you also specify sync:true on your note.add requests so they are sent immediately? Or do you rely on the outbound interval set on hub.set?
I have added rlauer@blues.io
it looks like the pico adds sync: true on all outbound notes
Use rlauer@blues.com please, thanks!
I need 20 characters to post so this should do it
done!
Thanks! Looking at your data, I see what you mean. Normally it takes only seconds to receive a Note from Notehub and turn around and upload a new Note (and have the Twilio route triggered).
Based on what I can see in Notehub, your device created the outbound_twilio.qo Note successfully, but then couldn’t reach Notehub for 50 minutes. Obviously this is difficult to debug, because we don’t know precisely what was happening on the device!
My best idea is based on a recent _session.qo Note…as your cellular connection is very very weak:
"rssi": -85,
"sinr": -1,
"rsrp": -114,
"rsrq": -14,
"rat": "lte",
"bars": 1,
This has the potential of leading to difficulties in connecting to the nearest tower, which can put the Notecard into a temporary penalty box and disable connections for awhile. There is no remote solution for this, but I’d always recommend using one of our wideband Notecards (e.g. WBNA) and include a diversity antenna. More suggestions available in our doc on diagnosing cellular connectivity issues.
This doesn’t help you today, but I’d recommend keeping an eye on those _session.qo Notefiles as they are useful for inspecting your signal strength.
Ok, I will look into this a bit. I do believe I have the more sensitive notecard. As I had had some issues when I first started this project.
On a side note, this is rather amusing. The reason that I am using this technology is the fact that I dont have descent internet, or cell signals.
Just so you know, for the most part I have not had an issue. Over the last month since this has been running in a pilot, I have from time to time checked the connection info through note hub and normally the last update was in the last 3-5 minutes in terms of the hub connection. as you can see, there were some other temp checks a week or so ago that went fine.
One thing you did not comment on specifically is the countless messages:
{“why”:“continuous connection mode (TLS)”}
what exactly is this for?
ok, so I did not notice the penalty box messages until now
{
“text”: “penalty: entering penalty box: connect delayed (61 min remaining): can’t open session to notehub: connect: Connection refused (111) {service}{extended-service-failure}”
}
I really dont remember seeing those, even when I had an issue yesterday? This is a problem. I will look into the links you provided on your note above.
For using the diversity option, is this just using a second antenna? The only antenna I have used is the one that came with the original notecard kit I got for the raspberry pi. is there a better antenna, or antenna’s for diversity? Any suggestions for the best antenna set up for diversity.
Finally, I just tried another temp request and so far no return sms after 10mins. I then ran a check on the note hub and the last seen was within minute after my request was sent. Does the penalty box not affect the last seen syncing with note hub?
This was looking like a very good solution, so this news is a bit disappointing.
Check out this doc on system notefiles which includes the _session.qo Notefile.
Also, I was reminded that you can remotely enable logging on a Notecard by adding the _log environment variable with a value of all for this device specifically. This will create _log.qo Notefiles that you can inspect on Notehub:
For the diversity antenna, what you want to do is cover as many LTE bands as possible and your other antenna can both do that and be set at and angle very different from your current antenna. It may or may not help depending on local cell coverage, but worth trying.
Rob
It looks like your connectivity was good up until around Feb 13th FYI:
Yes, up until Feb 13, the testing was at home in Toronto. The signals from then on, are from where the notecard would live. From my testing up north, I have consistently had 1 bar, but communication always seemed to be pretty solid.
I sent another test Temp note at 12:56 , the return SMS came to my phone at 13:49. Based on the info on notehub with captured/uploaded, can I assume that the notecard actually received the notes.qi requesting temp in a timely way because of the capture time for the twillio.qo is close to the notes.qi, but was delayed in creating the note twillio.qo based on the upload time of the twillio.qo? Does this mean that the notecard sees the incoming note, but cannot connect for the outgoing note? I am a bit confused here as I said before, the hub last seen of the notecard is always within minutes.
Any idea where I go from here?
Are you positive you are using sync:true with your note.add request? You’re right that it wouldn’t make sense that the Notecard could immediately receive inbound Notes, but not be able to send outbound Notes immediately afterwards.
This is the python code running on the pico connected to the notecard for sending the sms
req = {"req": "note.add"}
req["file"] = "twilio.qo"
req["sync"] = True
req["body"] = { "body":subject_matter, "from": "+1647xxxxxxx", "to": "+1416xxxxxxx" }
rsp = card.Transaction(req)
FYI, I just did a test this morning , just now and the results came back immediately. By the way, I noticed several events from the test simulator (Beverly MA), do you know the source of these events?
@rksteeves,
Good to hear. Still brainstorming here, but without being able to do a full reproduction with your code it’s tough to diagnose.
The dev:soft:account... device is a Notecard Simulator. Presumably you are using the simulator (as it’s associated with your ProductUID)? If you started using it, but stopped, let me know as we may have a rogue process that is still connected.
Rob
