RecursionError: maximum recursion depth exceeded while calling a Python object

[educhana]

Hi,

I've a problem using the spark backend when using an already created SparkSession. This is the simplest code to reproduce:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

from sqlframe import activate
activate(engine="spark", conn=spark)


from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

# Traceback on last getOrCreate:
# 
#   .........
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 121, in builder
#     return cls.Builder()
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 132, in __init__
#     self.builder = SparkSession.builder
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlglot\helper.py", line 44, in __get__
#     return classmethod(self.fget).__get__(None, owner)()  # type: ignore
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 121, in builder
#     return cls.Builder()
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 132, in __init__
#     self.builder = SparkSession.builder
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlglot\helper.py", line 44, in __get__
#     return classmethod(self.fget).__get__(None, owner)()  # type: ignore
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 121, in builder
#     return cls.Builder()
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 132, in __init__
#     self.builder = SparkSession.builder
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlglot\helper.py", line 44, in __get__
#     return classmethod(self.fget).__get__(None, owner)()  # type: ignore
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 121, in builder
#     return cls.Builder()
#   File "c:\ProgramData\anaconda3\envs\leanval-lib\lib\site-packages\sqlframe\spark\session.py", line 128, in __init__
#     def __init__(self):
#   File "_pydevd_bundle\\pydevd_cython.pyx", line 1697, in _pydevd_bundle.pydevd_cython.SafeCallWrapper.__call__
# RecursionError: maximum recursion depth exceeded while calling a Python object

seems that in SparkSession.builder tries to get the builder which is itself, thus the error.

Software versions:
sqlframe: 3.13.0
Any Python, tested on: >3.10
Any Spark version, tested on: >3.2

[eakmanrq]

This is going to be tricky to resolve. I need to figure out a way to replace the pyspark imports for the user's code but then still allow SQLFrame to refer to the PySpark imports when executing statements. Going to remove the activate option from Spark docs until I figure it out (changing imports still works).