IQ.Lvbs/IQ.Pilot
8
7
Fork 2
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Boot-log uploads to hosted ingest fail with empty responses (JSONDecodeError) — iquploaderd retries with exponential backoff indefinitely #6

New Issue
Closed
opened 2026-08-23 15:06:44 -05:00 by n0n4m3 · 1 comment
n0n4m3 commented 2026-08-23 15:06:44 -05:00
Copy Link

Summary

The device's iquploaderd daemon uploads boot logs (boot/*.zst) to the hosted Konn3kt ingest (api-konn3kt-ingests.konn3kt.com, JWT/HS512-authenticated connectincoming path). Every attempt fails with an empty response body, surfacing as JSONDecodeError in the uploader. The uploader then retries with exponential backoff forever; boot/crash logs are never delivered. This predates and is unaffected by the backend signing-key fix (konn3kt-node issue #2).

Environment

  • Device: konik a1m (comma 3X-class), dongle 1ffe88edf31ca489, hostname iq-d880a6d1
  • Software: IQ.Pilot 1.0c (commit 9f9c9a70cce4655844c33b8758c33fce09bfcf4f, prebuilt release @ ab07000)
  • Daemons: hephaestusd (mints upload URLs via _comma_connect_get_upload_urls), iquploaderd (executes queue)
  • Queue param: AthenadUploadQueue; boot entries JWT-signed, route segments HMAC-signed to the node

Symptom

swaglog (process 36450, iquploaderd):

upload_start   boot/0000001e--42d80e95a2.zst   sz=68896
upload_failed  boot/0000001e--42d80e95a2.zst   JSONDecodeError('Expecting value: line 1 column 1 (char 0)')
iquploaderd backoff 102.4
  • The response body is empty/non-JSON when parsed; no HTTP status is logged (the code path appears to json.loads() the raw response without surfacing status/body).
  • Backoff escalates 3.2 → 6.4 → 12.8 → 25.6 → 51.2 → 102.4 → … indefinitely.
  • Observed failing continuously from 2026-08-23 14:12 UTC through 20:00 UTC, before and after the backend key-derivation fix.

Evidence

  • Boot URLs target https://api-konn3kt-ingests.konn3kt.com/connectincoming/{dongle_id}/... with ?sig=<JWT>.
  • JWT: {"typ":"JWT","alg":"HS512"}, claims {"identity":"1ffe88edf31ca489","nbf":…,"iat":…,"exp":…} (24 h lifetime).
  • Route-segment uploads to the self-hosted node (HMAC) fail differently (403 signature mismatch) and were fixed by the backend change; the ingest failure is independent.

Questions

  1. Is the hosted ingest expected to accept boot-log uploads for this device/account (entitlement/flag)? If not, the uploader should skip boot uploads cleanly rather than retry forever.
  2. Is connectincoming + the boot path shape (…/boot/{file}.zst) correct for boot logs on the ingest host?
  3. What does the ingest return on rejection? An empty body is indistinguishable from a network truncation — please return a JSON error with status.
  4. Is the JWT issued by the backend accepted by the ingest (same identity derivation as issue #2, or separate)?

Suggested fixes

  • Log the HTTP status code and response body (first N bytes) on upload failure so failures are diagnosable.
  • Handle non-JSON / error responses gracefully (drop the item or cap retries) instead of infinite exponential backoff.
  • Confirm/rectify ingest acceptance for JWT-authenticated boot-log uploads from this device.

I'm happy to provide full swaglog excerpts or run a manual repro (the JWT URL can be captured from AthenadUploadQueue between retries).

## Summary The device's `iquploaderd` daemon uploads boot logs (`boot/*.zst`) to the hosted Konn3kt ingest (`api-konn3kt-ingests.konn3kt.com`, JWT/HS512-authenticated `connectincoming` path). Every attempt fails with an **empty response body**, surfacing as `JSONDecodeError` in the uploader. The uploader then retries with exponential backoff forever; boot/crash logs are never delivered. This predates and is unaffected by the backend signing-key fix (konn3kt-node issue #2). ## Environment - Device: konik a1m (comma 3X-class), dongle `1ffe88edf31ca489`, hostname `iq-d880a6d1` - Software: IQ.Pilot 1.0c (commit `9f9c9a70cce4655844c33b8758c33fce09bfcf4f`, prebuilt release @ ab07000) - Daemons: `hephaestusd` (mints upload URLs via `_comma_connect_get_upload_urls`), `iquploaderd` (executes queue) - Queue param: `AthenadUploadQueue`; boot entries JWT-signed, route segments HMAC-signed to the node ## Symptom swaglog (process 36450, iquploaderd): ``` upload_start boot/0000001e--42d80e95a2.zst sz=68896 upload_failed boot/0000001e--42d80e95a2.zst JSONDecodeError('Expecting value: line 1 column 1 (char 0)') iquploaderd backoff 102.4 ``` - The response body is empty/non-JSON when parsed; no HTTP status is logged (the code path appears to `json.loads()` the raw response without surfacing status/body). - Backoff escalates 3.2 → 6.4 → 12.8 → 25.6 → 51.2 → 102.4 → … indefinitely. - Observed failing continuously from 2026-08-23 14:12 UTC through 20:00 UTC, before and after the backend key-derivation fix. ## Evidence - Boot URLs target `https://api-konn3kt-ingests.konn3kt.com/connectincoming/{dongle_id}/...` with `?sig=<JWT>`. - JWT: `{"typ":"JWT","alg":"HS512"}`, claims `{"identity":"1ffe88edf31ca489","nbf":…,"iat":…,"exp":…}` (24 h lifetime). - Route-segment uploads to the self-hosted node (HMAC) fail differently (403 signature mismatch) and were fixed by the backend change; the ingest failure is independent. ## Questions 1. Is the hosted ingest expected to accept boot-log uploads for this device/account (entitlement/flag)? If not, the uploader should skip boot uploads cleanly rather than retry forever. 2. Is `connectincoming` + the boot path shape (`…/boot/{file}.zst`) correct for boot logs on the ingest host? 3. What does the ingest return on rejection? An empty body is indistinguishable from a network truncation — please return a JSON error with status. 4. Is the JWT issued by the backend accepted by the ingest (same identity derivation as issue #2, or separate)? ## Suggested fixes - Log the HTTP status code and response body (first N bytes) on upload failure so failures are diagnosable. - Handle non-JSON / error responses gracefully (drop the item or cap retries) instead of infinite exponential backoff. - Confirm/rectify ingest acceptance for JWT-authenticated boot-log uploads from this device. I'm happy to provide full swaglog excerpts or run a manual repro (the JWT URL can be captured from `AthenadUploadQueue` between retries).
teal commented 2026-08-23 15:35:49 -05:00
Owner
Copy Link

Fixed!

Fixed!
teal closed this issue 2026-08-23 15:35:49 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
teal
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: IQ.Lvbs/IQ.Pilot#6
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?