Problems with sync (getting headers)

arhtur-d42 · January 16, 2025, 1:10pm

I’m encountering an issue where sync works fine with empty tables but fails when trying to sync rows containing data. The error message suggests a WiFi/Gateway configuration problem:

SyncProgressTracker: FINAL SYNC Notification -default TEXT:WiFi Gateway or Configuration Error

There’s a specific error indicates a missing header which from the source code seems like it might result in this error message (which is related to the “not open data kit server”):

ProcessRowDataPushLocalChanges: Exception in synchronizeTable - pushing data up to server on table: SCAR 
exception: Response is missing required header: X-OpenDataKit-Version

org.opendatakit.services.sync.service.exceptions.NotOpenDataKitServerException: Response is missing required header: X-OpenDataKit-Version

I’ve done some investigation:

I can verify the headers are present using curl requests (using the full url)
The manifest etags have been verified
Initially suspected a problem with the aggregate URI due to truncated logs, but believe this is just normal log truncation:

HttpRestProtocolWrapper: buildBasicRequest: agg_uri is https://eqm.bandim.sdu.dk/odktables/default/tables/SCAR/ref/uuid:b204a908-7c03-429b-8809-be88f5...

Has anyone encountered similar issues or can suggest what might be causing the missing X-OpenDataKit-Version header specifically when syncing tables containing data?

Any help would be appreciated!

arhtur-d42 · January 16, 2025, 1:11pm

Screenshot of the error (This is from a waydroid emulator, but the same problem with the same logs occur on a physical device)

Emil · February 13, 2025, 12:12pm

Hello.

Hmm… I believe this could indicate some error with the endpoint configuration (sometimes this error pops up if the server returns a 5xx error)…
Is it behind a load balancer? (I see some extra headers related to HSTS, some strict mime-type/no-sniffing stuff, and a browser cookie being added, that are not typically part of the sync-endpoint default setup).

I would suggest the following debug process:
Perhaps you could use logcat to see the logs from the android device when you try to connect to see if that reveals some more detail?

adb logcat | grep -i "odk"

And if that doesn’t reveal anything obvious maybe, try to SSH to your sync-endpoint and inspect the logs for nginx and sync-endpoint respectively…

docker service logs syncldap_nginx

Look for something like e.g. *5 connect() failed (111: Connection refused) while connecting to upstream, client ... that would indicate some error with the upstream server (e.g. sync-endpoint).
I’ve seen this specific error due to a version-specific nginx misconfiguration, so check if you have something like nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/conf.d/default.conf:14 in the logs. If so, check what version of nginx you are running (it appeared that you were using 1.21.3?), and make sure that it is not set to ‘latest’ in the docker-compose.yml file.
Then maybe shell into the sync container and check the logs…

docker ps #to get the ids of running containers
docker exec -it [id_of_the_sync_container] /bin/bash
## once inside the container
cd logs
tail -f catalina[the date of the current logfile] #see if anything stands out
tail localhost_access_log.[the date] #see if you have any HTTP status errors (e.g. 500, 502 etc.)

Sometimes this can also be solved by simply restarting the docker swarm… .e.g

docker stack rm syncldap
## wait until docker ps returns nothing
docker stack deploy -c docker-compose.yml -c docker-compose-https.yml syncldap

I would recommend using VS Code with the extension ‘REST Client’ and try firing off this request while looking at the logs on the server:

GET https://(your endpoint)/odktables/default/tables/ HTTP/1.1
Authorization: BASIC your_username:your_password

Good luck, and let me know if any of that reveals anything

//emil

Emil · February 13, 2025, 12:44pm

Actually… after some deep-diving into this, I notice that using a newer image version of nginx can provoke this error. However, in my setup version “1.21.3” did not cause this - can you check if you image is specified with a specific version?

To get it working with the latest version of nginx (nginx/1.27.4 as of now) it appears some special handing of headers is required… Specifically, the poor tomcat server in sync-endpoint crashed if it sees a HEAD request, which it appears newer versions of nginx is quite explicit about (whereas previous versions maybe hid this issue). The solution to that is to just use a normal GET header…
You can achieve that by editing some files in your sync-endpoint-default-setup folder

config/nginx/sync-endpoint-https.conf (I’m assuming you use the https version). Add these lines in the top of the file:

map $request_method $proxy_method {
    HEAD GET;
    default $request_method;
}

config/nginx/sync-endpoint-locations.conf: Here we need to make 2 changes;
- remove the ‘Host’ header that is forwarded to the web-ui java spring boot application, by commenting out the line proxy_set_header Host $host:$server_port;
- in the section for location ^~ /odktables/ { you need make it use our custom proxy method, by inserting this line proxy_method $proxy_method; in the top of that section (you should also add it to the favicon.ico section). Your file should look something like this afterwards:

location = /favicon.ico {
        proxy_method $proxy_method;
        proxy_pass http://sync:8080/favicon.ico;
}

location = / {
        return 301 /web-ui/;
}

location ^~ /odktables/ {
        proxy_method $proxy_method;
        proxy_pass http://sync:8080/odktables/;

        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header Host $host:$server_port;

        proxy_redirect default;
}

location ^~ /web-ui/ {
        proxy_pass http://web-ui:8080/web-ui/;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        #proxy_set_header Host $host:$server_port;

        proxy_redirect default;
}

docker-compose.yml: Here you can specify the nginx image image:nginx:1.27.4 instead of lastest

Then in your sync-endpoint directory, you must change the file

sync-endpoint-docker-swarm/resources/server.xml: Add ‘secretRequired=“false”’ to the AJP connector tag to avoid a crash in the sync endpoint

     <!-- Define an AJP 1.3 Connector on port 8009 -->
     <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" secretRequired="false"/>

Here’s a one-liner for that purpose

 sed -i 's|<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />|<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" secretRequired="false" />|' server.xml

Then you need to build the sync-endpoint image again by cd’in into the folder and build with

docker build -t odk/sync-endpoint .

I hope that brings you closer to a solution

//emil

arhtur-d42 · February 14, 2025, 1:18pm

Thank you so much for looking at at it - now that I could say for sure that it was going through, I could get the network admin to take it more seriously.
I’m using nginx 1.21.3 so that wasn’t the issue, but i finally found the real culprit!

It turns out that the User-Agent part of the header is malformed or something and got caught in my organizations firewall but then somehow created an error in the logging system so it was almost completely invisible to the network administrators.

I’ll try next week to get closer to the root of the issue (seems like something about how android encrypts/decrypts headers or a problem in that area)