CHT Upgrade failing in docker helper 4.x

[sugat009]

Describe the bug

I tried upgrading a 4.x CHT docker helper instance of CHT version 4.6.0 to 4.9.0 and the upgrade failed. The browser shows the 502 Bad Gateway page whereas the containers have been stopped if you look into the terminal.

To Reproduce

Steps to reproduce the behavior:

1. Start a CHT docker helper 4.x instance
2. Start the upgrade
3. Reload page
4. Check docker containers

Environment

• Instance: CHT docker helper 4.x
• Browser: Firefox
• Client platform: Linux
• App: N/A
• Version: 4.6.0

Additional context

I'm not sure which logs to put here so I haven't put anything. Please request the necessary information as required.

[garethbowen]

Also found on the forum: https://forum.communityhealthtoolkit.org/t/error-while-updating-cht-from-4-5-to-4-9/3766

[mrjones-plip]

Upgrades should work! I have done this many times in the past and just now deployed a 4.1.1 version in docker helper and successfully upgraded it to 4.2.4. Docker Helper deploys a 1:1 dir-cht-upgrade-service for each local CHT deployment specifically to enable in place upgrades.

Here's a list of services running for my test upgrade just now:

➜  ~ docker ps  --format '{{ .Names }}'
test_4_1_1_to_4_2_0_upgrade-nginx-1
test_4_1_1_to_4_2_0_upgrade-sentinel-1
test_4_1_1_to_4_2_0_upgrade-api-1
test_4_1_1_to_4_2_0_upgrade-haproxy-1
test_4_1_1_to_4_2_0_upgrade-healthcheck-1
test_4_1_1_to_4_2_0_upgrade-couchdb-1
test_4_1_1_to_4_2_0_upgrade-dir-cht-upgrade-service-1

so, this ticket is great to capture that upgrades at some point stopped working in helper.

[dianabarsan]

I wonder if this is still the case. Any updates on this?

[sugat009]

I tried this now in cht-docker-helper from version CHT version 4.14.0 to 4.15.0 and it seems to have worked.

1. 4.14.0

2. 4.15.0

However, I faced this issue on version CHT 4.6.0. So, I tried creating a cht-docker-instance of version 4.6.0 to do that and I couldn't. The app is made of version 4.15.0.

Here are the logs:

❯ ./cht-docker-compose.sh                                                                                                                                                                       ─╯

Would you like to initialize a new project [y/N]?y
grep: warning: stray \ before :
grep: warning: stray \ before :

Do you want to run the latest CHT Core version (4.15.0) [Y/n]? n
grep: warning: stray \ before :
grep: warning: stray \ before :

Which version to you want to run? (ctrl + c to quit)
 1) 128-move-navbar-to-bottom						      93) 9327-spacing-aligments-feedback
 2) 130-top-bar								      94) 9327-tab-nav-aligment
 3) 159-e2e-tests-move-navbar-to-botton-mobile				      95) 9327-tab-nav-feedback
 4) 160-e2e-tests-top-bar						      96) 9327-update-nav
 5) 160-e2e-tests-top-bar-take-3					      97) 9327-upgrade
 6) 161-e2e-tests-display-tab-nav-on-left-side-desktop			      98) 9383-clean-up-scripts-dependencies
 7) 162-e2e-test-new-nav-sidebar-menu					      99) 9389-fix-person-type-param
 8) 163-re-enable-skipped-tests						     100) 9395-move-dependency
 9) 163-reenable-upgrade-tests						     101) 9397-prototype-for-documenting-ui-state-contacts-page-list-view-access
10) 163-tati-test							     102) 9398-clean-up-config-dependencies
11) 179-update-docs-for-apdex-testing					     103) 9406-disable-unsupported-language
12) 182-hamburger-icon							     104) 9416-inline-pupil
13) 4.0.0								     105) 9425-create-contact
14) 4.0.1								     106) 9428-db-monitoring-data
15) 4.1.0								     107) 9429-all-reports-in-contact-summary
16) 4.10.0								     108) 9431-always-aggregate
17) 4.1.0-FR-supervisor-chw-create-beta.1				     109) 9431-for-4.13.x
18) 4.1.0-FR-supervisor-chw-create-beta.4				     110) 9433-fix-users-contact-update
19) 4.1.1								     111) 9443-offline-user-e2e-test-coverage-for-tasks
20) 4.11.0								     112) 9446-haporxy-test
21) 4.11.x								     113) 9467-update-rapidpro
22) 4.11.x-500-reports-to-contact-summary				     114) 9471-ui-nav-image-test-contact-user-management
23) 4.11.x-9394-targets							     115) 9471-ui-nav-visual-test-contact-user-management
24) 4.1.2								     116) 9475-update-version
25) 4.12.0								     117) 9489-offline-search-telemetry
26) 4.12.0-FR-freetext-lite						     118) 9514-close-sidebar
27) 4.12.0-FR-freetext-lite-jkuester					     119) 9516-e2e-training-cards
28) 4.13.0								     120) 9519-update-version
29) 4.13.0-beta.3							     121) 9527-fix-privacy-policy-error
30) 4.13.0-beta.4							     122) 9543-flaky-e2e-test-incorrect-locale
31) 4.13.x								     123) 9544-offline-freetext
32) 4.14.0								     124) 9551-update-rules-engine-telemetry
33) 4.14.0-beta.1							     125) 9552-backport-4.13
34) 4.14.0-beta.2							     126) 9552-for-4.13.x
35) 4.14.x								     127) 9552-interval-turnover
36) 4.15.0								     128) 9552-missing-aggregate
37) 4.15.0-beta.1							     129) 9561-fix-cht-form-logs
38) 4.2.0								     130) 9571-update-version
39) 4.2.1								     131) 9577-clean-up-and-organize-page-object-files
40) 4.2.2								     132) 9585-sentinel-start-stop
41) 4.2.3								     133) 9586-implement-freetext-search-in-cht-datasource
42) 4.2.4								     134) 9594-remaining-e2e-test-for-tasks
43) 4.3.0								     135) 9604-fix-integer-validation
44) 4.3.1								     136) 9604-fix-integer-validation-4.14.x
45) 4.3.2								     137) 9612-dont-debouce-twice
46) 4.4.0								     138) 9612-force-tasks-loading
47) 4.4.1								     139) 9614-bump-helmchart
48) 4.4.1-FR-barcode-beta.1						     140) 9618-api-should-start-wiht-broken-forms
49) 4.4.1-FR-enketo-7-beta.1						     141) 9622-fix-flaky-telemetry-test
50) 4.4.1-FR-enketo-7-beta.2						     142) 9624-cleanup-api-build
51) 4.4.1-FR-enketo-7-with-async-compute-beta.1				     143) 9636-node-22
52) 4.4.1-FR-enketo-7-with-async-compute-beta.2				     144) 9642-purging-without-changes-2
53) 4.4.2								     145) add-funding-json-doc
54) 4.5.0								     146) all-view-updates
55) 4.5.1								     147) apdex-automation-tests
56) 4.5.2								     148) bump-up-version
57) 4.6.0								     149) bump-version-4-11
58) 4.6.0-alpha.1							     150) check-ci-status
59) 4.7.1								     151) cht-deploy-publish
60) 4.7.2								     152) cht-deploy-troubleshooting-and-helm-chart-update
61) 4.8.0								     153) datasource-bind
62) 4.8.1								     154) dependabot-npm-and-yarn-compression-1.7.5
63) 4.9.0								     155) dependabot-npm-and-yarn-elliptic-6.6.0
64) 7375-service-worker-auto-update					     156) docker-helper-curl-tls-move
65) 8119-training-card							     157) e2e-send-message-refactor
66) 8181-remove-old-design						     158) e2e-test-old-nav
67) 8216-propagate-req-uuid-to-haproxy					     159) e2e-test-old-nav-mobile
68) 8225-update-relevant-patch						     160) e2e-training-cards-mobile
69) 8437-useUnknownInCatchVariables					     161) extend-content-full-width
70) 8664-bug-fix							     162) feature-flag-hide-old-nav
71) 8664-fix								     163) fix-display-wide-screens
72) 8698-clean-up-and-organize-about-admin-analytics-folder		     164) fix-training-cards-title
73) 8700-clean-up-and-organize-folders					     165) garethbowen-patch-2
74) 8704-clean-up-and-organize-reports-folder				     166) long-content-training-cards
75) 8705-clean-up-and-organize-folders					     167) master
76) 8781-helper3x-new-compose						     168) move-nav-feedback
77) 8806-combining-validation-rules					     169) ne-translations-201-800
78) 8947-outbound-push-cron-config-bug					     170) new-nav-sidebar-menu
79) 9169-integration-testing						     171) remove-compose-version-string-9476
80) 9193-api-endpoints-for-getting-contacts-by-type			     172) remove-obsolete-package-lock
81) 9269-last-three-months-targets					     173) revert-9494
82) 9286-dont-timeout							     174) tati-test
83) 9291-align-icon							     175) tati-test-adding-ugly-if
84) 9293-research-freetext-views					     176) test-fetch-local-ip-certs
85) 9293-research-freetext-views-jkuester				     177) training-cards-once-a-day
86) 9293-research-freetext-views-jkuester-type				     178) training-page
87) 9293-test-report-query-params					     179) update-e2e-sync
88) 9303-upgrade-to-couchdb-34x						     180) update-release-templates
89) 9327-demo-update-nav						     181) updating-check-list
90) 9327-desktop-tabs-nav						     182) upgrade-angular-in-webapp
91) 9327-snackbar-and-modal						     183) upgrade-tests-after-new-nav-release
92) 9327-snackbar-positioning
#? 4.6.0

How do you want to name the project? upgrade_test_v2

Downloading compose files ...grep: warning: stray \ before :
grep: warning: stray \ before :
 done 

homedir: /home/sugat009/.medic/cht-docker/upgrade_test_v2-dir
[+] Running 2/2
 ✔ Network upgrade_test_v2-cht-net                      Created                                                                                                                               0.1s 
 ✔ Container upgrade_test_v2-dir-cht-upgrade-service-1  Started                                                                                                                               0.5s 
Starting project "upgrade_test_v2". First run takes a while. Will try for up to five minutes.......


 -------------------------------------------------------- 

  Success! "upgrade_test_v2" is set up:

    https://192-168-110-248.local-ip.medicmobile.org:10445/ (CHT)
    https://192-168-110-248.local-ip.medicmobile.org:10445/_utils/ (Fauxton)

    Login: medic
    Password: password

 -------------------------------------------------------- 

Start existing project
    ./cht-docker-compose.sh ENV-FILE.env

Stop and keep project:
    ./cht-docker-compose.sh ENV-FILE.env stop

Stop and destroy all project data:
    ./cht-docker-compose.sh ENV-FILE.env destroy

https://docs.communityhealthtoolkit.org/hosting/4.x/app-developer/


 Have a great day! 

❯ docker ps                                                                                                                                                                                     ─╯
CONTAINER ID   IMAGE                                                 COMMAND                   CREATED          STATUS          PORTS                                                                                    NAMES
b76753cfa001   public.ecr.aws/medic/cht-nginx:4.15.0                 "/docker-entrypoint.…"    13 seconds ago   Up 11 seconds   0.0.0.0:10082->80/tcp, [::]:10082->80/tcp, 0.0.0.0:10445->443/tcp, [::]:10445->443/tcp   upgrade_test_v2-nginx-1
ee91ab56755b   public.ecr.aws/medic/cht-sentinel:4.15.0              "/bin/bash /service/…"    13 seconds ago   Up 12 seconds                                                                                            upgrade_test_v2-sentinel-1
17f615792ec5   public.ecr.aws/medic/cht-api:4.15.0                   "/bin/bash /service/…"    13 seconds ago   Up 12 seconds   5988/tcp                                                                                 upgrade_test_v2-api-1
dfc93b236c3d   public.ecr.aws/medic/cht-haproxy:4.15.0               "/entrypoint.sh"          13 seconds ago   Up 13 seconds   5984/tcp                                                                                 upgrade_test_v2-haproxy-1
53e4e58d105b   public.ecr.aws/medic/cht-couchdb:4.15.0               "tini -- /docker-ent…"    13 seconds ago   Up 13 seconds   4369/tcp, 5984/tcp, 9100/tcp                                                             upgrade_test_v2-couchdb-1
c349670f821f   public.ecr.aws/medic/cht-haproxy-healthcheck:4.15.0   "/bin/sh -c \"/app/ch…"   13 seconds ago   Up 13 seconds                                                                                            upgrade_test_v2-healthcheck-1
4e5098bad7a4   public.ecr.aws/s5s3h4s7/cht-upgrade-service:latest    "node /app/src/index…"    14 seconds ago   Up 14 seconds                                                                                            upgrade_test_v2-dir-cht-upgrade-service-1

[dianabarsan]

Nice. Can we close this then?

[sugat009]

I'm not sure. On one hand, the upgrade is working but not of the exact version from which the issue was raised and the other is the reported version cannot be reached at the moment.
Should another issue be created to track this?

[dianabarsan]

im not sure what you mean by "reported version cannot be reached".

if this is related to a specific version, we should at least change the issue title and mark it as won't fix (ancient) and close it. there's no point in keeping issues that we won't fix open.

[dianabarsan]

Sorry, I re-read your whole message.
So now the problem is that you can't even start a 4.6.0 cht in docker-helper.
This is a different issue then, and we should log that. Would you mind creating it? Thanks!

[sugat009]

I am closing this as the upgrading is now working. Check comment.

[sugat009]

A new ticket has been created for the above mentioned issue. #9676

[mrjones-plip]

Hrm - I just installed version 4.6.0 and tried to upgrade to 4.15.0. It started, I saw all the containers reboot, and then the admin web GUI said the upgrade failed (Error triggering update). I noted the couchdb container was still running 4.6.0 version. I stopped all containers and started them all using the upgrade service. The adming web GUI now shows 4.15.0 but the couchdb container is still very much running 4.6.0:

docker ps --format "table {{.Image}}\t{{.Status}}\t{{.Names}}" --filter "name=4_6_0_for*"
IMAGE                                                 STATUS          NAMES
public.ecr.aws/medic/cht-nginx:4.15.0                 Up 14 minutes   4_6_0_for_realsies-nginx-1
public.ecr.aws/medic/cht-sentinel:4.15.0              Up 14 minutes   4_6_0_for_realsies-sentinel-1
public.ecr.aws/medic/cht-api:4.15.0                   Up 14 minutes   4_6_0_for_realsies-api-1
public.ecr.aws/medic/cht-haproxy-healthcheck:4.15.0   Up 14 minutes   4_6_0_for_realsies-healthcheck-1
public.ecr.aws/medic/cht-haproxy:4.15.0               Up 14 minutes   4_6_0_for_realsies-haproxy-1
public.ecr.aws/medic/cht-couchdb:4.6.0                Up 14 minutes   4_6_0_for_realsies-couchdb-1
public.ecr.aws/s5s3h4s7/cht-upgrade-service:latest    Up 14 minutes   4_6_0_for_realsies-dir-cht-upgrade-service-1

so...maybe worth re-opening this ticket?

[mrjones-plip]

Ok - not sure what was going on there with my past comment, but I tried a clean install of 4.6.0. production style and then upgraded to 4.15.0 and that upgrade failed too.

Guessing some docker detritus, I removed all containers (docker rm -f $(docker ps -qa)) and then removed all volumes (docker volume prune -a). Then both production style and docker helper upgrades from 4.6.0 to 4.15.0 worked as expected. PEBCAK 🤷 ?

This ticket is good to keep closed I think.