Stepper Tracking: Simplify `calypso_new_user_site_creation` #97130

escapemanuele · 2024-12-05T17:56:21Z

Proposed Changes

Simplify the logic to calculate calypso_new_user_site_creation.
It will be triggered only if the user has just been created and the site is new.

Why are these changes being made?

Too much complexity in the calculation may lead to problems in retrieving the right data.

Testing Instructions

Go through /start and /setup/onboarding
Verify that calypso_new_user_site_creation is triggered when using a new user, and it is not when using an already existing one.

github-actions · 2024-12-05T17:56:36Z

Calypso Live (direct link)

	https://calypso.live?image=registry.a8c.com/calypso/app:build-126807

Jetpack Cloud live (direct link)

	https://calypso.live?image=registry.a8c.com/calypso/app:build-126807&env=jetpack

Automattic for Agencies live (direct link)

	https://calypso.live?image=registry.a8c.com/calypso/app:build-126807&env=a8c-for-agencies

matticbot · 2024-12-05T18:04:45Z

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~29 bytes added 📈 [gzipped])

name           parsed_size           gzip_size
entry-stepper        +17 B  (+0.0%)      +29 B  (+0.0%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~1462 bytes removed 📉 [gzipped])

name                           parsed_size           gzip_size
write-flow                          -657 B  (-0.1%)     -247 B  (-0.1%)
transferring-hosted-site-flow       -657 B  (-0.6%)     -235 B  (-0.7%)
plugin-bundle-flow                  -657 B  (-0.3%)     -236 B  (-0.4%)
link-in-bio-tld-flow                -657 B  (-0.0%)     -229 B  (-0.0%)
import-hosted-site-flow             -657 B  (-0.1%)     -227 B  (-0.1%)
copy-site-flow                      -657 B  (-0.1%)     -236 B  (-0.1%)
build-flow                          -657 B  (-0.1%)     -244 B  (-0.1%)
signup                              -546 B  (-0.2%)     -165 B  (-0.3%)
home                                -196 B  (-0.0%)      -32 B  (-0.0%)
update-design-flow                   -94 B  (-0.0%)      -23 B  (-0.0%)
onboarding-flow                      -94 B  (-0.5%)      -26 B  (-0.4%)
jetpack-connect                      -77 B  (-0.0%)      -15 B  (-0.0%)
accept-invite                        -77 B  (-0.0%)      -10 B  (-0.0%)
site-settings                        +17 B  (+0.0%)      +11 B  (+0.0%)
settings                             +17 B  (+0.0%)      +11 B  (+0.0%)
checkout                             +17 B  (+0.0%)      +13 B  (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~42 bytes added 📈 [gzipped])

name                                                                              parsed_size           gzip_size
async-load-calypso-lib-analytics-signup                                                 -77 B  (-0.6%)       -9 B  (-0.2%)
async-load-purchase-modal-wrapper                                                       +17 B  (+0.0%)      +11 B  (+0.0%)
async-load-my-sites-checkout-purchase-modal-is-eligible-for-one-click-checkou...        +17 B  (+0.0%)      +12 B  (+0.0%)
async-load-calypso-layout-masterbar-checkout                                            +17 B  (+0.0%)      +15 B  (+0.1%)
async-load-calypso-blocks-editor-checkout-modal                                         +17 B  (+0.0%)      +13 B  (+0.0%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

matticbot · 2024-12-05T18:07:28Z

This PR modifies the release build for the following Calypso Apps:

For info about this notification, see here: PCYsg-OT6-p2

blaze-dashboard
odyssey-stats

To test WordPress.com changes, run install-plugin.sh $pluginSlug stepper-signup-complete-tracking on your sandbox.

alshakero

Found unrelated issue. Testing now.

alshakero · 2024-12-05T19:59:33Z

client/landing/stepper/declarative-flow/internals/components/help-center/async.tsx

-		<AsyncLoad
-			require="@automattic/help-center"
-			placeholder={ null }
-			handleClose={ handleClose }
-			shouldUseHelpCenterExperience={
-				! isLoading && experimentAssignment?.variationName === 'treatment'
-			}
-		/>


These seem to have leaked from #97119

alshakero · 2024-12-05T20:00:08Z

client/layout/index.jsx

-		'calypso_helpcenter_new_support_flow'
-	);
+
+	const shouldUseHelpCenterExperience = user?.ID ? user.ID % 100 < 75 : false;


alshakero · 2024-12-05T20:00:15Z

client/layout/index.jsx

-			shouldUseHelpCenterExperience={
-				! isLoading && experimentAssignment?.variationName === 'treatment'
-			}
+			shouldUseHelpCenterExperience={ shouldUseHelpCenterExperience }


alshakero

Can confirm it works as advertised! Will do a final test once it's cleaned up and approve.

alshakero · 2024-12-05T20:47:33Z

client/lib/analytics/signup.js

-			is_new_user: isNewUser,
-			is_newish_user: isNewishUser,


Just double checking that this is intentional.

Well, if we go with this new version, is_new_user will always be true, and we are not using is_newish_user anymore.

Let's hear what @MicBosi thinks!

Yes indeed! I also expected to remove these two fields which were added recently and I don't think are actually being used by us in data, perhaps they were more useful for debugging - and now we don't need them because the if condition.

escapemanuele · 2024-12-06T09:58:22Z

Oh well, I may have done too many things at the same time, let me clean things up!

MicBosi · 2024-12-06T14:16:10Z

@escapemanuele I noticed that in Stepper (not in Start) if I have already created a new user and new site and then try to go again to /setup/onboarding/ and complete the signup (new site creation) then calypso_signup_complete reports is_new_user: null rather than false.

calypso_new_user_site_creation does not fire, as expected, so that part is good.

MicBosi · 2024-12-06T14:36:04Z

@escapemanuele Actually, it looks like stepper sets is_new_user to null also when a new user is being created but I close the tab and reopen a new one in the same window. In /start this scenario seem to be working fine.

In the Stepper test Calypso knows I'm already logged in and redirects me from /setup/onboarding/ to the domains step, however, it does not seem to remember that a new user is being created in this "flow", maybe this has to do with how getSignupIsNewUserAndClear works compared to whatever /start is using to remember that a new user is being created. Perhaps /start is using some heuristic to determine if a user is "new" like based on time and/or whether they have 0 sites?

In this case I also don't see the expected calypso_new_user_site_creation since is_new_user is not true but null. Maybe this is the bug we were looking for 🤞

alshakero

I tested and can confirm the event is fired for new users with and without closing the tab.

MicBosi · 2024-12-09T12:22:50Z

I also confirmed closing the tab (but not the window) and reopening it worked as expected:

calypso_new_user_site_creation is fired
calypso_signup_complete { is_new_user: true, is_new_site: true }

I also tested both /start and /setup in the scenario where:

We complete the user step
We completely close the incognito window (so we lose all the Calypso state)
Login again and continue the onboarding from the domain step, select free domain and free plan

Result:

Both as expected do not fire calypso_new_user_site_creation (because they both lost the state for isNewUser)
Small discrepancy: Stepper shows calypso_signup_complete { is_new_user: null, is_new_site: true } while in Start is_new_user is set to false.

MicBosi

~~Besides the small issue with is_new_user being null instead of false mentioned here,~~ everything looks good to me. [Edit: there is no issue]

MicBosi · 2024-12-09T14:50:22Z

We debugged the apparent issue with @escapemanuele and it turned out to be testing error on my side when logging back in, I should have swapped the .live domain into the URL sent to my email - since I didn't it was picking up the non-fixed Calypso version.

escapemanuele added 2 commits December 5, 2024 15:41

New HC to 75% of users

6396222

Simplify new user site tracking

b6b8ee3

escapemanuele requested a review from a team as a code owner December 5, 2024 17:56

matticbot added [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. labels Dec 5, 2024

escapemanuele requested review from MicBosi, skim1220 and a team December 5, 2024 17:56

escapemanuele self-assigned this Dec 5, 2024

alshakero requested changes Dec 5, 2024

View reviewed changes

alshakero reviewed Dec 5, 2024

View reviewed changes

Remove unrelated changes

9833642

escapemanuele added 3 commits December 6, 2024 17:29

Set is_new_user to false instead of null

1c31254

Simplify logic

1c949ef

Switch to Id and localStorage

767884a

escapemanuele requested a review from alshakero December 6, 2024 17:38

alshakero approved these changes Dec 6, 2024

View reviewed changes

MicBosi approved these changes Dec 9, 2024

View reviewed changes

escapemanuele merged commit 9343aba into trunk Dec 9, 2024
12 checks passed

escapemanuele deleted the stepper-signup-complete-tracking branch December 9, 2024 14:51

github-actions bot removed the [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. label Dec 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stepper Tracking: Simplify `calypso_new_user_site_creation` #97130

Stepper Tracking: Simplify `calypso_new_user_site_creation` #97130

escapemanuele commented Dec 5, 2024

github-actions bot commented Dec 5, 2024 •

edited by matticbot

Loading

matticbot commented Dec 5, 2024 •

edited

Loading

matticbot commented Dec 5, 2024 •

edited

Loading

alshakero left a comment

alshakero Dec 5, 2024

alshakero Dec 5, 2024

alshakero Dec 5, 2024

alshakero left a comment

alshakero Dec 5, 2024

escapemanuele Dec 6, 2024

MicBosi Dec 6, 2024

escapemanuele commented Dec 6, 2024

MicBosi commented Dec 6, 2024

MicBosi commented Dec 6, 2024

alshakero left a comment

MicBosi commented Dec 9, 2024

MicBosi left a comment •

edited

Loading

MicBosi commented Dec 9, 2024 •

edited

Loading

Stepper Tracking: Simplify calypso_new_user_site_creation #97130

Stepper Tracking: Simplify calypso_new_user_site_creation #97130

Conversation

escapemanuele commented Dec 5, 2024

Proposed Changes

Why are these changes being made?

Testing Instructions

github-actions bot commented Dec 5, 2024 • edited by matticbot Loading

matticbot commented Dec 5, 2024 • edited Loading

matticbot commented Dec 5, 2024 • edited Loading

alshakero left a comment

Choose a reason for hiding this comment

alshakero Dec 5, 2024

Choose a reason for hiding this comment

alshakero Dec 5, 2024

Choose a reason for hiding this comment

alshakero Dec 5, 2024

Choose a reason for hiding this comment

alshakero left a comment

Choose a reason for hiding this comment

alshakero Dec 5, 2024

Choose a reason for hiding this comment

escapemanuele Dec 6, 2024

Choose a reason for hiding this comment

MicBosi Dec 6, 2024

Choose a reason for hiding this comment

escapemanuele commented Dec 6, 2024

MicBosi commented Dec 6, 2024

MicBosi commented Dec 6, 2024

alshakero left a comment

Choose a reason for hiding this comment

MicBosi commented Dec 9, 2024

MicBosi left a comment • edited Loading

Choose a reason for hiding this comment

MicBosi commented Dec 9, 2024 • edited Loading

Stepper Tracking: Simplify `calypso_new_user_site_creation` #97130

Stepper Tracking: Simplify `calypso_new_user_site_creation` #97130

github-actions bot commented Dec 5, 2024 •

edited by matticbot

Loading

matticbot commented Dec 5, 2024 •

edited

Loading

matticbot commented Dec 5, 2024 •

edited

Loading

MicBosi left a comment •

edited

Loading

MicBosi commented Dec 9, 2024 •

edited

Loading