Master Databricks and Apache Spark Step by Step: Lesson 35 - How to use SparkR (R on Spark)

Learn how to use the R programming language on Databricks and Apache Spark using the SparkR package.

Join my Patreon Community and Watch this Video without Ads!

Twitter: @BryanCafferky

Video on how to upload files to Databricks and create tables see:

Slides & Notebook

[jjonos]

Thanks for this great video. I use more of R than Python for Geospatial analysis so this came in VERY handy.When I ran the bit with SQL USE awproject, I kept getting an exception error. What's causing this?

[kamrunnisalietz]

Hello Mr Bryan. I didn't found any other way to connect you. And i am positive that you can give me the answer / right direction of my problem. So trying to get connected here. I am novice in Databricks and notebook. However, i created a jupyter notebook file that basically used for getting data from SSAS Tabular cube using DAX query and sending data to datalake as a csv file. when i run from notebook. it works however, whenever i am using it from Databricks using pipeline.. it creates issues about no module found 'clr', 'system' etc. Can you please help me how i can resolve this issue?

Thanks!

ModuleNotFoundError: No module named 'System'

ModuleNotFoundError Traceback (most recent call last)
<command-3989510516593721> in <module>
21 from azure.datalake.store import core, lib, multithread
22 #import clr
---> 23 from System.Reflection import Assembly
24
25 from System.Data import DataTable

in import_patch(name, globals, locals, fromlist, level)
156 # Import the desired module. If you’re seeing this while debugging a failed import,
157 # look at preceding stack frames for relevant error information.
--> 158 original_result = python_builtin_import(name, globals, locals, fromlist, level)
159
160 is_root_import = thread_local._nest_level == 1

ModuleNotFoundError: No module named 'System'