Routing data to google cloud - stop working

1

Hi,

I have different routes to google cloud working for a few months now. Since last night, they ALL stopped working. It seems like the json parsed to the google cloud functions are now empty.

I have not changed anything on both notehub and google cloud functions.

Have checked with webhooks, data is parsed correctly there. Getting this json and testing the google cloud function works fine.

Has anyone else experienced that lately?

Any help would be appreciated.

Info:

Functions are all in python

The problem is occurring with all the routes to google cloud function.

The problem started at the same time for all routes

Using the json parse to the webhooks to test a Triggering Event on google cloud works fine.

I still have credits

Best regards,

Theo

2

My routes stopped working today as well. Not sure if it’s related, but I’ve tracked it down to the removal of the Content-Type header.

My code is expecting

Content-Type: application/json

But logging the request headers I get this

2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http process request header line
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Host: xxx.xxx.com"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "User-Agent: Go-http-client/1.1"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Content-Length: 889"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Elastic-Apm-Traceparent: 00-9cccdf81784a4006ff810f333201d608-61d7423c1c0ed086-01"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Traceparent: 00-9cccdf81784a4006ff810f333201d608-61d7423c1c0ed086-01"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Tracestate: es=s:1"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header: "Accept-Encoding: gzip"
2023/06/27 13:12:04 [debug] 2960534#2960534: *1 http header done

I can probably work around it, but the Content-Type should really be set

3

Mine have also stopped roughly 2100 UTC last night.
Sending to Initial State not Google cloud.

The route error in Notehub is
error transforming request: json: cannot unmarshal array into Go value of type map[string]interface {}

Although oddly no errors or failed routes were reported until I tried to manually route events.

4

For me, it happens when I try to transform it with jsonata , even with a trivial json body

e.g.

    "body": {
        "foo": "bar"
    },

It happens when I Transform data to “Flattened body Keys/Values” but “Body Only” works.

Hoping for an urgent fix for this!

5

I’d just written some to code to relax the content-type check, then the correct headers are now being sent.

  'user-agent': 'notes',
  'content-type': 'application/json',

These have been added again and my code is now working

6

I don’t think that works for me. I was already explicitly sending content-type.

I see this in webhook.site

content-type application/json
accept-version ~0
content-length 13
user-agent notes
host webhook.site

(Although the event doesn’t appear to reach webhook.site at all if I apply a jsonata transformation first)

7

Mine is working back again without any changes. But it would be interesting to hear from Blues Wireless about the issue.

8

It was not just google cloud, our custom End Point also experienced the same problem with ~20 hours downtime.
There is also a post in https://status.notehub.io/

Everything is working again for us as of 15 hours ago.

9

Hello,

We deployed some code on June 26, 4:30pm EDT which contained a bug in our routing code which was omitting two default headers in some cases. As you’ve correctly identified, the two headers were:

Content-Type: application/json
User-Agent: notes

This was causing some routes to fail, specifically the Google Cloud routes, which depended on the Content-Type header. Other route types may have been affected too, depending on whether the the receiver requires this header.

We deployed a fix on June 27, 10:30am EDT, which we believe should have resolved this issue fully. If you are seeing any other issues, please reach out and we will investigate!

1 Like
10

All of the problems described in this thread are consistent with the bug that was fixed yesterday. We strive to do better than this.

To prevent this ever happening again, we’ve added a regression test to validate that the default headers are not omitted.

For awareness, when problems like this occur, we will post details on https://status.notehub.io.

11

My issue has not been fixed yet.
I still see

error transforming request: json: cannot unmarshal array into Go value of type map[string]interface {}

when sending anything with a jsonata transformation.

Please let me know if you need any more info from me.
I’ve tried to get a minimal case by sending

"body": {
    "foo": "bar"
},

to webhook.site

I see the same behaviour as yesterday, it works if I send “Body Only” but not if I send Flattened Body Keys/Values, and not if I use a jsonata transformation.

12

@calcut

I’d be happy to debug this issue with you. On the affected project, would you mind going to Project → Members on the left nav, then click “Invite Support”? This would greatly help in the debugging process. I only need access for 1 day

13

Done, is it just me experiencing this?

14

@calcut

It appears like the “Flattened Body Keys/Values” transformation you are using in your route is broken in our backend, so it’s failing before it’s even attempting to route the event. This kind of transformation isn’t used very often and it wasn’t covered by our tests. In the short term, if you switch to a different kind of transformation you can sidestep this issue.

On our end, I will be fixing this bug shortly but it may take a few days to be deployed.

I will keep you updated on the status of this issue

15

That was supposed to just a simplified example. I’m using a relatively complex JSONata expression typically. It worked fine until whatever was changed this week. I believe it broke at the same time as the Google cloud stuff

16

I’m using a relatively complex JSONata expression typically

Did your JSONata transformation produce an JSON array by any chance?

17

Yes, but it also fails if my JSONata expression is simply “when”

18

This particular bug will be exposed any time the transformation produces something that’s not a JSON object, does that seem to track with what you’re seeing?

19

Don’t have time to experiment just now. But how about reverting whatever change was made that broke this? It’s obviously not production ready. I’m using this for work and it looks pretty bad bad that my dashboard has effectively gone offline

20

@calcut

I have a fix and we will deploy it ASAP. I apologize for the inconvenience this has caused and I appreciate your patience!