Creating a catalog on S3 compatible cloud providers on Apache Polaris

Creating a catalog on AWS S3

Mon, 01 Jan 0001 00:00:00 +0000

When creating a catalog based on AWS S3 storage only the role-arn is a required parameter. However, usually one also provides the region and external-id.

Note: the name quickstart_catalog from the example below is referenced in other Getting Started examples, but of course, it can be any valid catalog name.

 1CLIENT_ID=root
 2CLIENT_SECRET=s3cr3t
 3DEFAULT_BASE_LOCATION=s3://example-bucket/my_data
 4ROLE_ARN=arn:aws:iam::111122223333:role/ExampleCorpRole
 5REGION=us-west-2
 6EXTERNAL_ID=12345678901234567890
 7
 8./polaris \
 9 --client-id ${CLIENT_ID} \
10 --client-secret ${CLIENT_SECRET} \
11 catalogs \
12 create \
13 --storage-type s3 \
14 --default-base-location ${DEFAULT_BASE_LOCATION} \
15 --role-arn ${ROLE_ARN} \
16 --region ${REGION} \
17 --external-id ${EXTERNAL_ID} \
18 quickstart_catalog

Creating a catalog on Apache Ozone

Mon, 01 Jan 0001 00:00:00 +0000

When creating a catalog based on Apache Ozone storage it is important to configure the endpoint property to point to your own storage cluster. If the endpoint property is not set, Polaris will attempt to contact AWS storage services (which is certain to fail in this case).

Note: the --no-sts CLI option instructs Polaris to avoid calling STS endpoints with AssumeRoles requests. This means that both Polaris and its clients (engines) will have to use distinct (local) credentials for accessing storage. Engines should not request credential vending in this case (should not use the X-Iceberg-Access-Delegation=vended-credentials header).

Creating a catalog on MinIO

Mon, 01 Jan 0001 00:00:00 +0000

When creating a catalog based on MinIO storage it is important to configure the endpoint property to point to your own MinIO cluster. If the endpoint property is not set, Polaris will attempt to contact AWS storage services (which is certain to fail in this case).

Note: the region setting is not required by MinIO, but it is set in this example for the sake of simplicity as it is usually required by the AWS SDK (used internally by Polaris). One can also set the AWS_REGION environment variable in the Polaris server process and avoid setting region as a catalog property.