[SPARK-50131][SQL] Add IN Subquery DataFrame API

[ueshin]

What changes were proposed in this pull request?

Adds IN Subquery DataFrame API.

The existing Column.isin will take Dataset/DataFrame and use it as IN subquery.

For example:

>>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob"), (8, "Mike")], ["age", "name"])
>>> df.createOrReplaceTempView('t')

>>> # SELECT * FROM t WHERE age IN (SELECT * FROM range(6)) ORDER BY age
>>> df.where(df.age.isin(spark.range(6))).orderBy("age").show()
+---+-----+
|age| name|
+---+-----+
|  2|Alice|
|  5|  Bob|
+---+-----+

For multiple values to be compared, sf.struct will be used to tuple them:

>>> from pyspark.sql import functions as sf

>>> # SELECT * FROM t WHERE (age, name) IN (SELECT id, 'Bob' FROM range(6))
>>> df.where(sf.struct(df.age, df.name).isin(spark.range(6).select("id", sf.lit("Bob")))).show()
+---+----+
|age|name|
+---+----+
|  5| Bob|
+---+----+

Why are the changes needed?

The IN subquery was missing in DataFrame API.

Does this PR introduce any user-facing change?

Yes, Column.isin will take Dataset/DataFrame to use it as IN subquery.

How was this patch tested?

Added the related tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[zhengruifeng]

for the case col.isin(df), when the col is a struct column and df only contains a struct column, what will happen?