Live Apache Spark Code Reviews - Scala Dataset API & PySpark K8s support

Continuing the PySpark on K8s PR review and a quick check through the outstanding issues from folks who are OK with me reviewing there PRs in public. As always if you're working in this space please ask me to review your PRs ( email or twitter DM ) and I'll try and take a look.