Web.post chunks get forwarded immediately in contradiction to documentation

Troubleshooting
5

@RobLauer - in working on this, I found a few other areas where the documentation can be improved. Specifically:

  • The documentation is ambiguous if “total” and “offset” are bytes positions in the underlying raw payload or the base64 encoded payload. Through trial and error I determined it was the former.
  • The documentation says the “status” md5sum should be base64 encoded but in practice I had to use hex encoding.
  • I was getting a response with a “payload” member that contained a base64 encoded representation of the upstream payload. This member is not documented at all.

I also ran across a few challenges while debugging that might be useful features to add to Notehub. I couldn’t find anywhere within Notehub where I could see the request received by the Web Proxy, nor the response received from the upstream web service.

In my cases, there were error conditions where the upstream replied with content that exceed 8192 bytes, so all I got on the Notecard was a message said the response was too large. It would have been very helpful to login to Notehub to see a log of the request and response between the WebProxy and the web service (similar to the very helpful Events log) . Perhaps that exists and I just I missed it.

@scjerry - in Python. Here is a helper function I made and have tested to work correctly with the latest firmware.

def web_post(card, route, payload, name=None, chunk_size=8192):
    offset = 0
    fragmented = ( len(payload) > chunk_size )
    while offset < len(payload):
        req = {"req": "web.post"}
        req["route"] = route
        if name:
            req["name"] = name
        if fragmented:
            fragment = payload[offset:offset+chunk_size]
            req["total"] = len(payload)
            req["payload"] = base64.b64encode( fragment ).decode("ascii")
            req["status"] = hashlib.md5( fragment ).hexdigest()
            req["offset"] = offset
            req["verify"] = True # not sure if this is required or not
            logging.debug("sending web.post fragment of length %s at offset %s", len(fragment), offset)
        else:
            logging.debug("sending web.post of length %s", len(payload))
            req["payload"] = base64.b64encode( payload ).decode("ascii")

        offset += chunk_size
        rsp = card.Transaction(req)
        logging.debug("web.post response %s", rsp)
        # if data remains to be transmitted we expect a 100 request
        if offset < len(payload) and rsp.get("result") != 100:
            raise RuntimeError("error in fragmented web.post")

    # Get the last response payload, if any
    response_payload = None
    if rsp.get("payload"):
            response_payload = base64.b64decode(rsp['payload'])
    return rsp, response_payload
1 Like