Port important fix(es) from original #1
Description
Looking at the spark repo, it looks like at least one relevant fix has been made since this code was extracted. It might be a good idea to merge it. I wonder if the spark utility can be directly used rather than propagating the fixes.
The fix: apache/spark@06b6a07