[SPARK-54587][PYTHON] Consolidate all runner_conf related code

### What changes were proposed in this pull request?

Refactor runner_conf in `worker.py`. Put all the related logic together so the code using it can just access its attributes.

### Why are the changes needed?

This is part of the effort to consolidate the worker protocol. We need to clean up the code a bit first.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

`pyspark-sql` passes locally. The rest we need to see CI results.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #53315 from gaogaotiantian/consolidate-runner-conf.

Authored-by: Tian Gao <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>

Author: gaogaotiantian