spark-cli.mdx

Spark connection profile

After installing Elementary's dbt package upon running dbt deps, you can generate Elementary's profile for usage with edr by running the following command within your project:

dbt run-operation elementary.generate_elementary_cli_profile

The command will print to the terminal a partially filled template of the profile that's needed for edr to work.

## SPARK ##
## By default, edr expects the profile name 'elementary'.      ##
## Configure the database and schema of elementary models.     ##
## Check where 'elementary_test_results' is to find it.        ##

elementary:
  outputs:
    default:
      type: spark
      method: [thrift, http, or odbc]
      host: [hostname]
      port: [port]
      user: [username]
      schema: [schema name] # elementary schema, usually [schema name]_elementary
      threads: [1 or more]
      # token: [optional, used with http method]

We support the same format and connection methods as dbt. Please refer to dbt's documentation of Spark profile for further details.