Skip to content

Update Salesforce Connector To Create Source Models #9491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cwarden wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update Salesforce Connector To Create Source Models #9491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72d2db0
salesforce: switch to OAuth flows and warehouse-style form
cwarden May 22, 2026
151e35a
salesforce: switch to Bulk API 2.0 and add information schema
cwarden May 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 91 additions & 28 deletions docs/docs/developers/build/connectors/data-source/salesforce.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,55 +9,118 @@ sidebar_position: 65

## Overview

[Salesforce](https://www.salesforce.com/) is a leading cloud-based Customer Relationship Management (CRM) platform designed to help businesses connect with and understand their customers better. It offers a comprehensive suite of applications focused on sales, customer service, marketing automation, analytics, and application development. Salesforce enables organizations of all sizes to build stronger relationships with their customers through personalized experiences, streamlined communication, and predictive insights. Rill can ingest data from Salesforce as a source by utilizing the Bulk API, which requires a Salesforce username and password (and, in some cases, a token, depending on the org configuration) to authenticate against a Salesforce org.
[Salesforce](https://www.salesforce.com/) is a leading cloud-based Customer Relationship Management (CRM) platform. Rill ingests data from Salesforce by issuing SOQL queries against the Bulk API. Authentication uses the Salesforce OAuth 2.0 endpoints exposed by a Connected App (or, for the client credentials and JWT flows, an External Client App).

The Salesforce connector follows the same shape as other warehouse connectors: credentials live in a connector file under `connectors/`, and each query is its own model file under `models/`.

## Local credentials
## Authentication

When using Rill Developer on your local machine, you will need to provide your credentials via a connector file. We would recommend not using plain text to create your file and instead use the `.env` file. For more details on your connector, see [connector YAML](/reference/project-files/connectors) for more details.
The Salesforce connector supports three OAuth flows. All three require a [Connected App](https://help.salesforce.com/s/articleView?id=sf.connected_app_overview.htm) in your Salesforce org; the client credentials and JWT flows also accept an [External Client App](https://help.salesforce.com/s/articleView?id=xcloud.ecapps_intro.htm). The flow you choose determines which other fields you need to provide.

:::tip Updating the project environmental variable
| Flow | Required fields |
| --- | --- |
| Username / Password (OAuth) | `username`, `password`, `client_id`, `client_secret` |
| Client Credentials | `client_id`, `client_secret` |
| JWT Bearer | `username`, `client_id`, `key` |

The connector picks a flow based on which credentials are populated: JWT wins when `key` is set; otherwise a `username` plus `password` selects the OAuth password flow; otherwise a `client_secret` selects the client credentials flow.

:::note SOAP login is deprecated

Earlier versions of this connector used the SOAP login endpoint when a username and password were supplied. Salesforce is decommissioning the SOAP login endpoint, so the connector now uses the OAuth password flow instead. This requires the Connected App's Client Secret in addition to the existing Client ID.

Note that the OAuth password flow only works with a Connected App; External Client Apps do not support it. The client credentials and JWT flows work with either.

If you've already deployed to Rill Cloud, you can either [push/pull the credential]( /guide/administration/project-settings/variables-and-credentials#pushing-and-pulling-credentials-to--from-rill-cloud-via-the-cli) from the CLI with:
```
rill env push
rill env pull
```
:::

Alternatively, you can include the credentials directly in the underlying source YAML by adding the `username` and `password` parameters. For example, your source YAML may contain the following properties (these can also be configured through the UI during source creation):
### Connector file

Place your credentials in a connector file at `connectors/<name>.yaml`. Reference secret values from `.env`.

#### Username / Password (OAuth)

```yaml
type: "model"
connector: "salesforce"
endpoint: "login.salesforce.com"
username: "user@example.com"
password: "MyPasswordMyToken"
soql: "SELECT Id, Name, CreatedDate FROM Opportunity"
sobject: "Opportunity"
type: connector
driver: salesforce

endpoint: login.salesforce.com
username: user@example.com
password: "{{ .env.connector.salesforce.password }}"
client_id: "<Client ID>"
client_secret: "{{ .env.connector.salesforce.client_secret }}"
```

:::tip Did you know?
#### Client Credentials

If this project has already been deployed to Rill Cloud and credentials have been set for this source, you can use `rill env pull` to [pull these cloud credentials](/developers/build/connectors/credentials/#rill-env-pull) locally (into your local `.env` file). Please note that this may override any credentials you have set locally for this source.
```yaml
type: connector
driver: salesforce

:::
endpoint: login.salesforce.com
client_id: "<Client ID>"
client_secret: "{{ .env.connector.salesforce.client_secret }}"
```

## Deploy to Rill Cloud
#### JWT Bearer

When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors) for more information.
```yaml
type: connector
driver: salesforce

If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI:
endpoint: login.salesforce.com
username: user@example.com
client_id: "<Client ID>"
key: "{{ .env.connector.salesforce.key }}"
```
rill env push

PEM keys contain newlines, which break `.env` parsing if stored raw. The UI's file picker base64-encodes the uploaded key automatically; when hand-editing `.env`, base64-encode the PEM file yourself:

```sh
base64 < key.pem | tr -d '\n' >> .env
```

Raw PEM written inline in the connector YAML (without an `.env` reference) is also accepted.

:::note
## Models

Leave the `key` and `client_id` fields blank unless you are using JWT (described in the next section [below](#jwt)).
A Salesforce model file references the connector by name and supplies the SOQL query. Rill issues the query through the Salesforce [Bulk API 2.0](https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/bulk_api_2_0.htm)

```yaml
type: model
materialize: true

connector: salesforce

soql: |
SELECT Id, Name, CreatedDate
FROM Opportunity

output:
connector: duckdb
```

Use `soql:` for the query. `sql:` is also accepted as an alias for parity with other warehouse drivers (the connector explorer in the UI writes the query into `sql:`). Add `queryAll: true` to include soft-deleted records.

SOQL itself does not accept `SELECT *`, but the connector rewrites the `SELECT * FROM <SObject>` shape (which the explorer's "Table" mode emits) into an explicit field list discovered from the SObject's describe response. Compound types like `Address` and `Location` are skipped — their atomic sub-components (e.g. `BillingStreet`, `BillingCity`) are queried instead.

## Local credentials

When using Rill Developer on your local machine, provide credentials via a connector file as shown above. Keep secrets in `.env` rather than the connector YAML. See [connector YAML](/reference/project-files/connectors) for more details.

:::tip Updating the project environmental variable

If you've already deployed to Rill Cloud, you can either [push/pull the credential]( /guide/administration/project-settings/variables-and-credentials#pushing-and-pulling-credentials-to--from-rill-cloud-via-the-cli) from the CLI with:
```
rill env push
rill env pull
```
:::

### JWT
## Deploy to Rill Cloud

When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. See the [connector YAML reference docs](/reference/project-files/connectors) for more information.

Authentication using JWT instead of a password is also supported. Set `client_id` to the **Client Id** (also known as the _Consumer Key_) of the Connected App to use, and set `key` to contain the PEM-formatted private key to use for signing.
If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), update them by pushing the `Deploy` button or by running:
```
rill env push
```
14 changes: 10 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.13.1
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.6.4
github.com/ClickHouse/clickhouse-go/v2 v2.41.0
github.com/ForceCLI/force v1.1.0
github.com/ForceCLI/force v1.10.1
github.com/Masterminds/sprig/v3 v3.3.0
github.com/MicahParks/keyfunc v1.9.0
github.com/NYTimes/gziphandler v1.1.1
Expand Down Expand Up @@ -142,6 +142,7 @@ require (
)

require (
git.sr.ht/~jackmordaunt/go-toast v1.1.2 // indirect
github.com/apache/arrow/go/v12 v12.0.1 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.0.8 // indirect
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
Expand All @@ -152,6 +153,8 @@ require (
github.com/duckdb/duckdb-go-bindings/lib/linux-amd64 v0.10502.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/linux-arm64 v0.10502.0 // indirect
github.com/duckdb/duckdb-go-bindings/lib/windows-amd64 v0.10502.0 // indirect
github.com/esiqveland/notify v0.13.3 // indirect
github.com/gen2brain/beeep v0.11.2 // indirect
github.com/go-openapi/swag/cmdutils v0.25.4 // indirect
github.com/go-openapi/swag/conv v0.25.5 // indirect
github.com/go-openapi/swag/fileutils v0.25.5 // indirect
Expand All @@ -163,15 +166,21 @@ require (
github.com/go-openapi/swag/stringutils v0.25.5 // indirect
github.com/go-openapi/swag/typeutils v0.25.5 // indirect
github.com/go-openapi/swag/yamlutils v0.25.5 // indirect
github.com/godbus/dbus/v5 v5.1.0 // indirect
github.com/in-toto/attestation v1.1.2 // indirect
github.com/jackmordaunt/icns/v3 v3.0.1 // indirect
github.com/moby/moby/api v1.54.1 // indirect
github.com/moby/moby/client v0.4.0 // indirect
github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/prometheus/otlptranslator v1.0.0 // indirect
github.com/rs/zerolog v1.28.0 // indirect
github.com/segmentio/encoding v0.5.4 // indirect
github.com/sergeymakinen/go-bmp v1.0.0 // indirect
github.com/sergeymakinen/go-ico v1.0.0-beta.0 // indirect
github.com/sigstore/sigstore v1.10.4 // indirect
github.com/sigstore/sigstore-go v1.1.4 // indirect
github.com/tadvi/systray v0.0.0-20190226123456-11a2b8fa57af // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v4 v4.0.0-rc.4 // indirect
gotest.tools/gotestsum v1.8.2 // indirect
Expand Down Expand Up @@ -200,7 +209,6 @@ require (
github.com/BurntSushi/toml v1.4.0 // indirect
github.com/ClickHouse/ch-go v0.69.0 // indirect
github.com/DefangLabs/secret-detector v0.0.0-20250403165618-22662109213e // indirect
github.com/ForceCLI/config v0.0.0-20230217143549-9149d42a3c99 // indirect
github.com/ForceCLI/inflect v0.0.0-20130829110746-cc00b5ad7a6a // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.31.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.51.0 // indirect
Expand All @@ -209,7 +217,6 @@ require (
github.com/Masterminds/semver/v3 v3.4.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/ProtonMail/go-crypto v1.3.0 // indirect
github.com/ViViDboarder/gotifier v0.0.0-20140619195515-0f19f3d7c54c // indirect
github.com/acarl005/stripansi v0.0.0-20180116102854-5a71ef0e047d // indirect
github.com/alicebob/gopher-json v0.0.0-20230218143504-906a9b012302 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
Expand Down Expand Up @@ -347,7 +354,6 @@ require (
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/mitchellh/copystructure v1.2.0 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/mitchellh/reflectwalk v1.0.2 // indirect
github.com/moby/buildkit v0.29.0 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
Expand Down
Loading
Loading