GH-28859: [Doc][Python] Use only code-block directive and set up doctest for the python user guide (#48619)

### Rationale for this change

In many places in the Python User Guide the code exampels are written with IPython directive (elsewhere code-block is used). IPython directives are converted to IPython format (`In` and `Out` during the doc build). This can lead to slower builds.

### What changes are included in this PR?

IPython directives are converted to runnable code-block (with `>>>` and `...`) and pytest doctest support for `.rst` files is added to the `conda-python-docs` CI job. This means the code in the Python User Guide is tested separately to the building  of the documentation.

### Are these changes tested?

Yes, with the CI.

### Are there any user-facing changes?

Changes to the Python User Guide examples will have to be tested with `pytest --doctest-glob='*.rst' docs/source/python/file.rst`

* GitHub Issue: #28859

Lead-authored-by: AlenkaF <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: tadeja <tadeja@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>

Author: AlenkaF

ci/scripts/python_test.sh

@@ -70,3 +70,8 @@ export PYARROW_TEST_S3
 
 # Testing PyArrow
 pytest -r s ${PYTEST_ARGS} --pyargs pyarrow
+
+# Testing RST documentation examples (if PYTEST_RST_ARGS is set)
+if [ -n "${PYTEST_RST_ARGS}" ]; then
+  pytest ${PYTEST_RST_ARGS} ${arrow_dir}/docs/source/python
+fi

compose.yaml

@@ -1530,6 +1530,7 @@ services:
       BUILD_DOCS_CPP: "ON"
       BUILD_DOCS_PYTHON: "ON"
       PYTEST_ARGS: "--doctest-modules --doctest-cython"
+      PYTEST_RST_ARGS: "--doctest-glob=*.rst"
     volumes: *conda-volumes
     command:
       ["/arrow/ci/scripts/cpp_build.sh /arrow /build &&

docs/source/developers/python/development.rst

@@ -127,6 +127,24 @@ for ``.py`` files or
 for ``.pyx`` and ``.pxi`` files. In this case you will also need to
 install the `pytest-cython <https://github.com/lgpage/pytest-cython>`_ plugin.
 
+Testing Documentation Examples
+-------------------------------
+
+Documentation examples in ``.rst`` files under ``docs/source/python/`` use
+doctest syntax and can be tested locally using:
+
+.. code-block::
+
+   $ pushd arrow/python
+   $ pytest --doctest-glob="*.rst" docs/source/python/file.rst # checking single file
+   $ pytest --doctest-glob="*.rst" docs/source/python # checking entire directory
+   $ popd
+
+The examples use standard doctest syntax with ``>>>`` for Python prompts and
+``...`` for continuation lines. The ``conftest.py`` fixture automatically
+handles temporary directory setup for examples that create files.
+
+
 Debugging
 =========