> ## Documentation Index
> Fetch the complete documentation index at: https://kb.aampe.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Estimated setup time: 1 hour

## Prerequisites

* Databricks workspace with Unity Catalog enabled
* Metastore created and attached to your workspace
* Appropriate admin privileges to create service principals and grant permissions

## Integration Methods

### Option 1: Direct Read Access via Service Principal

This method allows Aampe to directly access your Unity Catalog tables using a dedicated service principal with OAuth authentication.

#### Required Permissions

The Aampe service principal needs the following Unity Catalog privileges:

* USE CATALOG - Access to view metadata in the catalog
* USE SCHEMA - Access to view metadata in schemas
* SELECT - Read access to tables and views
* BROWSE - Optional: View object metadata in Catalog Explorer

#### Step-by-step Procedure

1. Create the Service Principal

**Option A: Databricks-managed Service Principal**

1. Go to your Databricks account console
2. Navigate to **User management** > **Service principals**
3. Click **Add service principal**
4. Provide a name (e.g., aampe-integration-sp)
5. Click **Add**

**Option B: Azure Entra ID Service Principal (Azure Databricks only)**

1. Create an app registration in Azure Portal
2. Note the Application (client) ID
3. Add the service principal to your Databricks account
4. Generate OAuth Credentials
5. In the service principal's details page, go to the **Secrets** tab
6. Under **OAuth secrets**, click **Generate secret**
7. Set the secret's lifetime (maximum 730 days)
8. Copy and securely store:
   * **Client ID** (same as Application ID)
   * **OAuth Secret** (shown only once)
9. Click **Done**
10. Grant Unity Catalog Permissions

Run the following SQL commands in a Databricks notebook or SQL editor:

```
-- Grant catalog-level permissions
GRANT USE CATALOG ON CATALOG <catalog_name> TO <service_principal_id>;

-- Grant schema-level permissions
GRANT USE SCHEMA ON SCHEMA <catalog_name>.<schema_name> TO <service_principal_id>;

-- Grant table-level read permissions
GRANT SELECT ON TABLE <catalog_name>.<schema_name>.<table_name> TO <service_principal_id>;

-- Alternatively, grant SELECT on all tables in a schema
GRANT SELECT ON SCHEMA <catalog_name>.<schema_name> TO <service_principal_id>;
```

Replace:

* \<catalog\_name>: Your Unity Catalog name
* \<schema\_name>: Your schema/database name
* \<table\_name>: Specific table names (or use schema-level grant)
* \<service\_principal\_id>: The Application ID of your service principal

5. Provide Connection Details to Aampe

Share the following information with the Aampe team:

* **Workspace URL**: `https://<workspace-name>.cloud.databricks.com`
* **Client ID**: Your service principal's Application ID
* **OAuth Secret**: The generated secret (transfer securely)
* **Catalog Name**: Your Unity Catalog name
* **Schema Names**: List of schemas containing relevant tables
* **Table Names**: Specific tables to sync (if not syncing entire schemas)
* **Workspace ID**: Your Databricks workspace ID
* **Account ID**: Your Databricks account ID (for account-level access)

### Option 2: Push to Cloud Storage

For large-scale data transfers, you can export data to cloud storage that Aampe can access.

#### AWS S3 Setup

1. Create External Location

   ```
   -- Create storage credential
   CREATE STORAGE CREDENTIAL IF NOT EXISTS aampe_storage_cred
   WITH (
     AWS_IAM_ROLE = 'arn:aws:iam::<account-id>:role/<role-name>'
   );

   -- Create external location
   CREATE EXTERNAL LOCATION IF NOT EXISTS aampe_export_location
   URL 's3://<bucket-name>/aampe-export/'
   WITH (STORAGE CREDENTIAL aampe_storage_cred);
   ```
2. Grant Permissions

   ```
   -- Grant write access to the service principal
   GRANT CREATE EXTERNAL TABLE ON EXTERNAL LOCATION aampe_export_location 
   TO <service_principal_id>;

   GRANT WRITE FILES ON EXTERNAL LOCATION aampe_export_location 
   TO <service_principal_id>;
   ```
3. Set up Automated Export

Create a Databricks Job or Delta Live Tables pipeline to export data

## Data Models

Ensure your exported data conforms to the Aampe <u>Data Models</u> specification.