Getting Started with Apache Polaris and RustFS

Tags:

⚠️ Warning

Disclaimer: This guide uses mc from MinIO OSS for local testing only. MinIO OSS is in maintenance mode, and MinIO container images may no longer receive updates or security fixes. For production setups, rc should be used.

Overview🔗

This example uses RustFS as a storage provider with Polaris.

Spark is used as a query engine. This example assumes a local Spark installation. See the Spark Notebooks Example for a more advanced Spark setup.

Starting the Example🔗

Build the Polaris server image if it’s not already present locally:

1./gradlew \
2   :polaris-server:assemble \
3   :polaris-server:quarkusAppPartsBuild --rerun \
4   -Dquarkus.container-image.build=true

Start the docker compose group by running the following command from the root of the repository:
```
1docker compose -f site/content/guides/rustfs/docker-compose.yml up
```

Connecting From Spark🔗

 1bin/spark-sql \
 2    --packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.10.1,org.apache.iceberg:iceberg-aws-bundle:1.10.1 \
 3    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
 4    --conf spark.sql.catalog.polaris=org.apache.iceberg.spark.SparkCatalog \
 5    --conf spark.sql.catalog.polaris.type=rest \
 6    --conf spark.sql.catalog.polaris.uri=http://localhost:8181/api/catalog \
 7    --conf spark.sql.catalog.polaris.token-refresh-enabled=false \
 8    --conf spark.sql.catalog.polaris.warehouse=quickstart_catalog \
 9    --conf spark.sql.catalog.polaris.scope=PRINCIPAL_ROLE:ALL \
10    --conf spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation=vended-credentials \
11    --conf spark.sql.catalog.polaris.credential=root:s3cr3t

Note: s3cr3t is defined as the password for the root user in the docker-compose.yml file.

Running Queries🔗

Run inside the Spark SQL shell:

1USE polaris;
2
3CREATE NAMESPACE ns;
4
5CREATE TABLE ns.t1 AS SELECT 'abc';
6
7SELECT * FROM ns.t1;
8-- abc

RustFS Endpoints🔗

Note that the catalog configuration defined in the docker-compose.yml contains different endpoints for the Polaris Server and the client (Spark). Specifically, the client endpoint is http://localhost:9000, but endpointInternal is http://rustfs:9000.

This is necessary because clients running on localhost do not normally see service names (such as rustfs) that are internal to the docker compose environment.