Databricks | search
Click here to view Databricks news from 60+ newspapers.
Bookmark or Share- Newest 'databricks' Questions - Stack OverflowAzure Databricks Single Node Cluster 140 GB, 72 cores, Databricks Runtime version 12.2 LTS (includes Apache Spark 3.3.2, Scala 2.12) I have a dataframe with 330 (312 numeric) columns and 300,000 rows and I am trying to impute the medians, but memory utilization is maximized and it cannot finish. median_values={col:df_new....
- Databricks: How do I get path of current notebook?The issue is that Databricks does not have integration with VSTS. A workaround is to download the notebook locally using the CLI and then use git locally. I would, however, prefer to keep everything in Databricks. If I can download the .ipynb to the dbfs, then I can use a system call to push the notebooks to VSTS using git. –
- Printing secret value in Databricks - Stack OverflowFirst, install the Databricks Python SDK and configure authentication per the docs here. pip install databricks-sdk Then you can use the approach below to print out secret values. Because the code doesn't run in Databricks, the secret values aren't redacted. For my particular use case, I wanted to print values for all secrets in a given scope.
- Connecting C# Application to Azure Databricks - Stack OverflowThe Datalake is hooked to Azure Databricks. The requirement asks that the Azure Databricks is to be connected to a C# application to be able to run queries and get the result all from the C# application. The way we are currently tackling the problem is that we have created a workspace on Databricks with a number of queries that need to be executed.
- How to use python variable in SQL Query in Databricks?Also like 2 other ways to access variable will be 1. the spark.sql way as you mentioned like spark.sql(f"select * from tdf where var={max_date2}") 2. will be to create a temp table with that value and use that table like spark.createDataFrame([(max_date2,)],"my_date string").createOrReplaceTempView("vartable") and use value from vartable in your query Also if you are thinking that changing ...
More
