Databricks | search
Click here to view Databricks news from 60+ newspapers.
Bookmark or Share- Assign a variable a dynamic value in SQL in Databricks / SparkDatabricks Runtime 14.1 and higher now properly supports variables.-- DBR 14.1+ DECLARE VARIABLE dataSourceStr STRING = "foobar"; SELECT * FROM hive_metastore.mySchema.myTable WHERE dataSource = dataSourceStr; -- Returns where dataSource column is 'foobar'
- Printing secret value in Databricks - Stack OverflowFirst, install the Databricks Python SDK and configure authentication per the docs here. pip install databricks-sdk Then you can use the approach below to print out secret values. Because the code doesn't run in Databricks, the secret values aren't redacted. For my particular use case, I wanted to print values for all secrets in a given scope.
- Installing multiple libraries 'permanently' on Databricks' cluster ...Easiest is to use databricks cli's libraries command for an existing cluster (or create job command and specify appropriate params for your job cluster) Can use the REST API itself, same links as above, using CURL or something. Could also use terraform to do this if you want a full CI/CD automation.
- databricks - How to get the cluster's JDBC/ODBC parameters ...Databricks documentation shows how get the cluster's hostname, port, HTTP path, and JDBC URL parameters from the JDBC/ODBC tab in the UI. See image: (source: databricks.com) Is there a way to get the same information programmatically? I mean using the Databricks API or Databricks CLI.
- Databricks: How do I get path of current notebook?The issue is that Databricks does not have integration with VSTS. A workaround is to download the notebook locally using the CLI and then use git locally. I would, however, prefer to keep everything in Databricks. If I can download the .ipynb to the dbfs, then I can use a system call to push the notebooks to VSTS using git. –
More
