[SPARK-51591][PYTHON][CONNECT] Fix ThreadPoolExecutor failure in python 3.13 daily test

[zhengruifeng]

What changes were proposed in this pull request?

Fix ThreadPoolExecutor failure in python 3.13 daily build

Why are the changes needed?

in the last 3 weeks, python 3.13 daily build keeps failing with

  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1823, in _handle_error
    raise error
  File "/__w/spark/spark/python/pyspark/sql/connect/client/core.py", line 1534, in _execute_and_fetch_as_iterator
    for b in generator:
             ^^^^^^^^^
  File "<frozen _collections_abc>", line 360, in __next__
  File "/__w/spark/spark/python/pyspark/sql/connect/client/reattach.py", line 146, in send
    self._release_all()
    ~~~~~~~~~~~~~~~~~^^
  File "/__w/spark/spark/python/pyspark/sql/connect/client/reattach.py", line 241, in _release_all
    self._release_thread_pool.submit(target)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 171, in submit
    raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown

see https://github.com/apache/spark/actions/runs/13955602888/job/39065942222

In ThreadPoolExecutor.submit, this only happens when the pool is already shutdown:

    def submit(self, fn, /, *args, **kwargs):
        with self._shutdown_lock, _global_shutdown_lock:
            if self._broken:
                raise BrokenThreadPool(self._broken)

            if self._shutdown:
                raise RuntimeError('cannot schedule new futures after shutdown')
            if _shutdown:
                raise RuntimeError('cannot schedule new futures after '
                                   'interpreter shutdown')

            f = _base.Future()
            w = _WorkItem(f, fn, args, kwargs)

            self._work_queue.put(w)
            self._adjust_thread_count()
            return f

Does this PR introduce any user-facing change?

no

How was this patch tested?

PR builder with

default: '{"PYSPARK_IMAGE_TO_TEST": "python-313", "PYTHON_TO_TEST": "python3.13"}'

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

merged to master

[zhengruifeng]

post-merge check: https://github.com/apache/spark/actions/runs/14028046790

3.13 daily test is green now