[Cluster] `py-spy` does not work on Ray cluster on GCP using the Autoscaler YAML and adding suggested commands for setup.

[simonsays1980]

What happened + What you expected to happen

What happened

I started a Ray cluster using the autoscaler for GCP. I wanted to trace some memory problems usign py-spy. I got an error Permission denied (os error 13). I used in my setup_commands in the YAML:

- python -m pip install py-spy
- sudo chown root:root `which py-spy`
- sudo chmod u+s `which py-spy`

I guess there is still some problems with the root not being able to access the user's py-spy. Also the password from user ray is not available anywhere. Which is it?

What you expected to happen

That using py-spy on a Ray cluster works seemlessly when using the dashboard's Trace Back option.

Versions / Dependencies

Python 3.11
Ubuntu 22.04.4 LTS
Ray Nightly Sept. 4th 2024

Reproduction script

Use the full YAML for GCP: https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/gcp/example-full.yaml

Use any python script to execute on the cluster. Then try to click in the dashboard under Actors on Stack Trace.

Issue Severity

Medium: It is a significant difficulty but I can work around it.