Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Enterprise CommCare Client Version Compliance Report #35468

Open
wants to merge 40 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
52520df
Refactor: rename function
jingcheng16 Nov 26, 2024
52b62e7
Util to tell if a version is out of date
jingcheng16 Dec 3, 2024
4c2c923
Add Commcare Version Compliance Tile
jingcheng16 Dec 3, 2024
2ee9d14
Add CommCare Version Compliance API
jingcheng16 Dec 3, 2024
5609175
Refactor: move get_report_task to the base class
jingcheng16 Dec 3, 2024
94f54c6
Show -- for the total if the account has no mobile worker yet
jingcheng16 Dec 3, 2024
9cc4bda
Move is_out_of_date above _parse_version
jingcheng16 Dec 4, 2024
7338fcc
change text and fix format
jingcheng16 Dec 4, 2024
36138f7
Remove unnecessary code
jingcheng16 Dec 4, 2024
0512997
Change private function to public since it's imported by other file
jingcheng16 Dec 4, 2024
9dc9a81
delay the translation until rendering
jingcheng16 Dec 4, 2024
344a76c
Rename metric to description
jingcheng16 Dec 4, 2024
1aec926
Percentage in enterprise console will have one digit
jingcheng16 Dec 4, 2024
180eb0c
Use elastic search
jingcheng16 Dec 6, 2024
b8cb113
Having a util function to handle the commcare version the mobile work…
jingcheng16 Dec 6, 2024
b494264
isort
jingcheng16 Dec 6, 2024
f38e1c3
Remove formatting of version in enterprise console so it's comparable
jingcheng16 Dec 7, 2024
fcf3c61
ES returning dates in iso format so add conversion to avoid type error
jingcheng16 Dec 7, 2024
d72d426
Merge branch 'master' into jc/commcare_client_version_compliance_report
mjriley Dec 9, 2024
b1a6545
flake8
jingcheng16 Dec 9, 2024
cd61d41
Change title
jingcheng16 Dec 9, 2024
57c489a
Merge branch 'master' into jc/commcare_client_version_compliance_report
jingcheng16 Dec 10, 2024
c03d909
Rename description to total_description
jingcheng16 Dec 10, 2024
5470ea5
Update label....
jingcheng16 Dec 10, 2024
4bc93c8
adjust tile layout
biyeun Dec 11, 2024
1c1abb6
Merge branch 'master' into jc/commcare_client_version_compliance_report
jingcheng16 Dec 11, 2024
0d85d21
Rename ignored variable
jingcheng16 Dec 12, 2024
34e358b
Refactor: format version only once
jingcheng16 Dec 12, 2024
88252a0
Memoize get_latest_version_at_time and use minute precision for date_…
jingcheng16 Dec 12, 2024
8b75af1
Fix UnboundLocalError
jingcheng16 Dec 12, 2024
c0fd790
Remove microseconds too
jingcheng16 Dec 12, 2024
05f6058
Fetch CommCareBuildConfig only once
jingcheng16 Dec 12, 2024
adb7d81
memoize build time to avoid repetitively fetching build for each version
jingcheng16 Dec 12, 2024
4a81844
update naming convention for elements refrenced in javascript
biyeun Dec 12, 2024
b4d59b4
This tile don't need domain_obj
jingcheng16 Dec 12, 2024
d3da1d7
Move total above rows_for_domain
jingcheng16 Dec 12, 2024
1cc3178
Use gevent pooling for parallel domain processing
jingcheng16 Dec 12, 2024
7599675
Refer domain name as domain and fix a format in template
jingcheng16 Dec 13, 2024
44d9db5
quickcache on globally scoped function
jingcheng16 Dec 13, 2024
bb87820
keep exception handling as close as possible to the operation that ca…
jingcheng16 Dec 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions corehq/apps/enterprise/enterprise.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,11 @@

from memoized import memoized

from corehq.apps.reports.standard.deployments import _format_commcare_version
mjriley marked this conversation as resolved.
Show resolved Hide resolved
from couchforms.analytics import get_last_form_submission_received
from dimagi.utils.dates import DateSpan

from corehq.apps.builds.utils import get_latest_version_at_time, is_out_of_date
from corehq.apps.enterprise.exceptions import EnterpriseReportError, TooMuchRequestedDataError
from corehq.apps.enterprise.iterators import raise_after_max_elements
from corehq.apps.accounting.models import BillingAccount
Expand All @@ -34,6 +36,7 @@ class EnterpriseReport:
MOBILE_USERS = 'mobile_users'
FORM_SUBMISSIONS = 'form_submissions'
ODATA_FEEDS = 'odata_feeds'
COMMCARE_VERSION_COMPLIANCE = 'commcare_version_compliance'

DATE_ROW_FORMAT = '%Y/%m/%d %H:%M:%S'

Expand Down Expand Up @@ -67,6 +70,8 @@ def create(cls, slug, account_id, couch_user, **kwargs):
report = EnterpriseFormReport(account, couch_user, **kwargs)
elif slug == cls.ODATA_FEEDS:
report = EnterpriseODataReport(account, couch_user, **kwargs)
elif slug == cls.COMMCARE_VERSION_COMPLIANCE:
report = EnterpriseCommCareVersionReport(account, couch_user, **kwargs)

if report:
report.slug = slug
Expand Down Expand Up @@ -383,3 +388,68 @@ def _get_individual_export_rows(self, exports, export_line_counts):
)

return rows


class EnterpriseCommCareVersionReport(EnterpriseReport):
title = _('CommCare Client Version Compliance')
metric = _('%% of mobile workers on the latest commcare client version ')
mjriley marked this conversation as resolved.
Show resolved Hide resolved
mjriley marked this conversation as resolved.
Show resolved Hide resolved

def __init__(self, account, couch_user, **kwargs):
super().__init__(account, couch_user, **kwargs)
mjriley marked this conversation as resolved.
Show resolved Hide resolved

@property
def headers(self):
return [
_('Mobile Worker'),
_('Project Space'),
_('Latest Version Available at Submission'),
_('Version in Use'),

mjriley marked this conversation as resolved.
Show resolved Hide resolved
]

def rows_for_domain(self, domain_obj):
rows = []

for user in get_all_user_rows(domain_obj.name,
include_web_users=False,
include_mobile_users=True,
include_inactive=False,
include_docs=True):
mjriley marked this conversation as resolved.
Show resolved Hide resolved
user = CouchUser.wrap_correctly(user['doc'])
last_submission = user.reporting_metadata.last_submission_for_user
mjriley marked this conversation as resolved.
Show resolved Hide resolved
last_used_device = user.last_device

version_in_use = _('Unknown')
date_of_use = None

# If the user hasn't submitted a form, we use the last used device
if last_submission and last_submission.commcare_version:
version_in_use = _format_commcare_version(last_submission.commcare_version)
date_of_use = last_submission.submission_date
elif last_used_device and last_used_device.commcare_version:
version_in_use = _format_commcare_version(last_used_device['commcare_version'])
date_of_use = last_used_device.last_used

latest_version_at_time_of_use = get_latest_version_at_time(date_of_use)

if is_out_of_date(version_in_use, latest_version_at_time_of_use):
rows.append([
user.username,
domain_obj.name,
latest_version_at_time_of_use,
version_in_use,
])

return rows

@property
def total(self):
total_mobile_workers = 0
total_up_to_date = 0
for domain_obj in self.domains():
domain_mobile_workers = get_mobile_user_count(domain_obj.name, include_inactive=False)
if domain_mobile_workers:
total_mobile_workers += domain_mobile_workers
total_up_to_date += domain_mobile_workers - len(self.rows_for_domain(domain_obj))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is definitely a performance concern here, as if this summary query takes 2 minutes, that is 2 minutes blocking a web worker. It's worth trying to see if Elasticsearch is any faster here. Otherwise, we may want to cache this statistic.

Copy link
Contributor Author

@jingcheng16 jingcheng16 Dec 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried on production again, it takes between 2 minutes and 3 minutes. Last time I was about to leave my apartment so I didn't wait until it finished, so I guess last time when I query couch directly maybe it longer than 3 minutes. There is improvement by using elastic search. Is the improvement acceptable? I think maybe yes? Because not every enterprise account would have that amount of mobile workers...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're talking minute I think we need to think about caching or turning this into a celery task. I can imagine that multiple requests to this page could cause a disruption in our responsiveness that would be hard to diagnose

percentage = (total_up_to_date / total_mobile_workers) if total_mobile_workers else 0
return f"{percentage * 100:.0f}%"
mjriley marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,11 @@ hqDefine("enterprise/js/project_dashboard", [
const requestParams = {
url: initialPageData.reverse("enterprise_dashboard_total", slug),
success: function (data) {
$display.html(Number(data.total).toLocaleString());
if (typeof data.total === 'number') {
$display.html(Number(data.total).toLocaleString());
} else {
$display.html(data.total);
}
mjriley marked this conversation as resolved.
Show resolved Hide resolved
},
error: function (request) {
if (request.responseJSON) {
Expand Down
1 change: 1 addition & 0 deletions corehq/apps/enterprise/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ def platform_overview(request, domain):
EnterpriseReport.MOBILE_USERS,
EnterpriseReport.FORM_SUBMISSIONS,
EnterpriseReport.ODATA_FEEDS,
EnterpriseReport.COMMCARE_VERSION_COMPLIANCE,
)],
'metric_type': 'Platform Overview',
})
Expand Down