Sync-endpoint-docker-swarm build failure

ornich · August 3, 2022, 4:39pm

Hi, i am new in odk-x. I try to install sync-endpoint but i get this error.

kind help please! i need to resolve it. Thank you

elmps2018 · August 7, 2022, 8:25pm

Can you please tell us a bit more about your setup, and what steps you took before this error? You may find the documentation on setup here Setup ODK-X Sync Endpoint with Cloud Services — ODK-X Docs helpful if you haven’t reviewed it already, or setup videos on our YouTube channel ODK-X Team - YouTube

ornich · August 8, 2022, 9:07am

@elmps2018 hi thank you for the reply, i Setup ODK-X Sync Endpoint Manually OS: ubuntu 22.04. I use this documentation Setup ODK-X Sync Endpoint Manually — ODK-X Docs

elmps2018 · August 8, 2022, 1:51pm

Hi @ornich! Thanks for these details. Could you tell us which steps you had completed and which one you were trying when you got this error?

W_Brunette · August 8, 2022, 6:32pm

I am not sure we have tested the recent upgrades to ubuntu yet with the older Tomcat. You might want to try ubuntu 18.

We need to do a maintence release and move all the versions up. It is on the to do list.

ornich · August 12, 2022, 9:11am

@elmps2018 i get this error when i setup sync-endpoint Manually following the steps:
1)cd sync-endpoint
2)mvn clean install

John_Mduda · September 4, 2022, 2:49pm

Hello,
Early this year, I installed sync-endpoint for testing on Ubuntu 18.04, 20.04, and even 21.04, and the sync-endpoint build went well. However, I recently got a new server and intended to deploy odk-x for production, however sync-endpoint build fails even with Ubuntu 18.04. I’ve attached the installation logs for your inspection.

Emil · September 8, 2022, 3:22pm

Hello,
There appears to be an error in the Dockerfile (it is missing the “unzip” command).
I will create a pull request to fix the problem , in the mean time you can fix it by installing it this way:

1: Edit the dockerfile by typing:
$ nano ~/sync-endpoint/sync-endpoint-docker-swarm/Dockerfile
2: Add this line 8 (before the other RUN command):
RUN apt-get update && apt-get install unzip
3: Save and exit, then rerun the mvn clean install command

Good luck and Let me know if it doesn’t work out

//emil

Update: I’ve created a pull request to try to fix this

John_Mduda · September 12, 2022, 11:48am

Hello,

That solves the problem but also “sync-endpoint/sync-endpoint-docker-test/Dockerfile” file miss the same line, I have to update it. However the error now occur at postgres-test. I attach the files for your review.

sync-endpoint-install.txt (1.7 MB)
failsafe-reports.zip (98.3 KB)

John

fzivo · September 14, 2022, 2:08pm

Thank you.
There is also a need to edit the sync-endpoint-docker-test Dockerfile in the same manner.

What failed for me now is the postgres-test

JohnSnow · October 14, 2022, 5:44pm

Hello,
I am experiencing the same issues. The longitudinal qualities and flexibility with custom prompts/table templates of ODK-X are great, but I am having issues with the sync-endpoint. I can only manually sync any data using adb pull commands.

As mentioned by fzivo, the sync-endpoint-docker-test Dockerfile needs to be updated in the same manner. Only the sync-endpoint-docker-swarm Dockerfile has been updated in the pull request.

Initially, I got one error when moving to building the postgres-test where the port was already in use. However, that was resolved by restarting the docker service.

Now, postgres starts but there seems to be an error with several unit tests, where pages that do not exist are requested:

[ERROR] Tests run: 10, Failures: 0, Errors: 9, Skipped: 0, Time elapsed: 6.991 s <<< FAILURE! - in org.opendatakit.aggregate.odktables.HashTableManagerTestIT
[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSingleAppLevelFiles  Time elapsed: 0.453 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSingleAppLevelFiles(HashTableManagerTestIT.java:451)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSameSingleTableLevelFileToBothTables  Time elapsed: 0.655 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSameSingleTableLevelFileToBothTables(HashTableManagerTestIT.java:267)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddTwoTableLevelFilesRemoveOne  Time elapsed: 0.397 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddTwoTableLevelFilesRemoveOne(HashTableManagerTestIT.java:347)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testGetTablesWithNoFiles  Time elapsed: 0.52 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testGetTablesWithNoFiles(HashTableManagerTestIT.java:153)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddTwoTableLevelFiles  Time elapsed: 0.432 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddTwoTableLevelFiles(HashTableManagerTestIT.java:298)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddNRemoveSingleTableLevelFiles  Time elapsed: 0.364 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddNRemoveSingleTableLevelFiles(HashTableManagerTestIT.java:216)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSingleTableLevelFiles  Time elapsed: 0.352 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddSingleTableLevelFiles(HashTableManagerTestIT.java:184)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddDifferentSingleTableLevelFileToBothTablesNAddAppLevel  Time elapsed: 0.611 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddDifferentSingleTableLevelFileToBothTablesNAddAppLevel(HashTableManagerTestIT.java:496)

[ERROR] org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddDifferentSingleTableLevelFileToBothTables  Time elapsed: 0.63 s  <<< ERROR!
org.springframework.web.client.HttpClientErrorException: 404 <!doctype html><html lang="en"><head><title>HTTP Status 404 – Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 – Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/8.5.83</h3></body></html>
        at postgres.test@2.1.9/org.opendatakit.aggregate.odktables.HashTableManagerTestIT.testAddDifferentSingleTableLevelFileToBothTables(HashTableManagerTestIT.java:420)

Any way to resolve this, e.g. by disabling these tests? It seems to be the same error as John experienced. I am not sure whether these errors come from my server, or an external server from where webpages are being pulled.

Thank you.
PS: yes, I will introduce myself on the appropriate thread.

W_Brunette · October 14, 2022, 6:06pm

The automatic script for setting up sync-endpoint skips the tests.

Here is the python function for building the swarm image

def run_sync_endpoint_build():
    os.system("git clone -b master --single-branch --depth=1 https://github.com/odk-x/sync-endpoint ; \
               cd sync-endpoint ; \
               mvn -pl org.opendatakit:sync-endpoint-war,org.opendatakit:sync-endpoint-docker-swarm,org.opendatakit:sync-endpoint-common-dependencies clean install -DskipTests")

They key command is:
mvn -pl org.opendatakit:sync-endpoint-war,org.opendatakit:sync-endpoint-docker-swarm,org.opendatakit:sync-endpoint-common-dependencies clean install -DskipTests

JohnSnow · October 16, 2022, 5:10pm

Thanks Waylon, and I appreciate the quick response (I was actually able to try this the same night, but haven’t had the time to respond to this topic yet)!

I can confirm that skipping the tests using this command indeed resolved the error I encountered. Unfortunately, I still am not able to log in after creating the appropriate LDAP users (same error as others reported) but that is a separate issue -will keep looking for a solution there.

JohnSnow · October 22, 2022, 9:10pm

Hi John,

I just wanted to highlight this other post of mine about the same issue: Problems with login in to ODK-X Server web-ui. Incorrect username or password - #52 by JohnSnow

I couldn’t get sync-endpoint to work either, but ODK Aggregate seems to be a good alternative until the issues with sync-endpoint get resolved. I really wished it didn’t use Docker.

Redeem_Grimm · October 27, 2022, 10:03pm

@JohnSnow Aggregate is a good alternative for now. sync-endpoint issue will be resolve soonest.

JohnSnow · October 28, 2022, 8:56am

That is great to hear! Thank you for the response. Once resolved, I will migrate to sync-endpoint.

Tomy_Pop · January 28, 2023, 9:25pm

If you haven’t already, open a terminal and ssh into the machine where you run your manager node. …
Run docker service inspect --pretty to display the details about a service in an easily readable format