Apache Polaris 1.4.0 Documentation on Apache Polaris

Admin Tool

Mon, 01 Jan 0001 00:00:00 +0000

Polaris includes a tool for administrators to manage the metastore.

The tool is available as a Docker image: apache/polaris-admin-tool. It can also be downloaded as part of the binary distribution.

📝 Note

The tool must be built with the necessary database drivers to access the metastore database. The default build includes drivers for the PostgreSQL and NoSQL (MongoDB) backends.

Usage🔗

Please make sure the admin tool and Polaris server are with the same version before using it. To run the standalone JAR, use the following command:

Command Line Interface

Mon, 01 Jan 0001 00:00:00 +0000

In order to help administrators quickly set up and manage their Polaris server, Polaris provides a simple command-line interface (CLI) for common tasks.

The basic syntax of the Polaris CLI is outlined below:

polaris [options] COMMAND ...

options:
--host
--port
--base-url
--client-id
--client-secret
--access-token
--realm
--header
--profile
--proxy
--debug

COMMAND must be one of the following:

catalogs
principals
principal-roles
catalog-roles
namespaces
privileges
profiles
policies
repair
setup

Each command supports several subcommands, and some subcommands have actions that come after the subcommand in turn. Finally, arguments follow to form a full invocation. Within a set of named arguments at the end of an invocation ordering is generally not important. Many invocations also have a required positional argument of the type that the command refers to. Again, the ordering of this positional argument relative to named arguments is not important.

Realm

Mon, 01 Jan 0001 00:00:00 +0000

This page explains what a realm is and what it is used for in Polaris.

What is it?🔗

A realm in Polaris serves as logical partitioning mechanism within the catalog system. This isolation allows for multitenancy, enabling different teams, environments or organizations to operate independently within the same Polaris deployment.

Key Characteristics🔗

Isolation: Each realm encapsulates its own set of resources, ensuring that operations, policies in one realm do not affect others.

Entities

Mon, 01 Jan 0001 00:00:00 +0000

This page documents various entities that can be managed in Apache Polaris.

Catalog🔗

A catalog is a top-level entity in Polaris that may contain other entities like namespaces and tables. These map directly to Apache Iceberg catalogs.

For information on managing catalogs with the REST API or for more information on what data can be associated with a catalog, see the CreateCatalogRequest OpenAPI.

Storage Type🔗

All catalogs in Polaris are associated with a storage type. Valid Storage Types are S3, Azure, and GCS. The FILE type is also additionally available for testing. Each of these types relates to a different storage provider where data within the catalog may reside. Depending on the storage type, various other configurations may be set for a catalog including credentials to be used when accessing data inside the catalog.

Policy

Mon, 01 Jan 0001 00:00:00 +0000

The Polaris Policy framework empowers organizations to centrally define, manage, and enforce fine-grained governance, lifecycle, and operational rules across all data resources in the catalog.

With the policy API, you can:

Create and manage policies
Attach policies to specific resources (catalogs, namespaces, tables, or views)
Check applicable policies for any given resource

What is a Policy?🔗

A policy in Apache Polaris is a structured entity that defines rules governing actions on specified resources under predefined conditions. Each policy contains:

Generic Table

Mon, 01 Jan 0001 00:00:00 +0000

The generic tables are non-Iceberg tables. Table can be multiple formats including Delta, CSV, etc. With this framework, you can:

Create a generic table under a namespace
Load a generic table
Drop a generic table
List all generic tables under a namespace

❗Important

Generic tables are in beta. Please use it with caution and report any issue if encountered.

What is a Generic Table?🔗

A generic table is an entity that defines the following fields:

Telemetry

Mon, 01 Jan 0001 00:00:00 +0000

Metrics🔗

Metrics are published using Micrometer; they are available from Polaris’s management interface (port 8182 by default) under the path /q/metrics. For example, if the server is running on localhost, the metrics can be accessed via http://localhost:8182/q/metrics.

Metrics can be scraped by Prometheus or any compatible metrics scraping server. See: Prometheus for more information.

Additional tags can be added to the metrics by setting the polaris.metrics.tags.* property. Each tag is a key-value pair, where the key is the tag name and the value is the tag value. For example, to add a tag environment=prod to all metrics, set polaris.metrics.tags.environment=prod. Many tags can be added, such as below:

Polaris Spark Client

Mon, 01 Jan 0001 00:00:00 +0000

Polaris provides a Spark client to manage non-Iceberg tables through Generic Tables.

📝 Note

The Spark client can manage Iceberg tables and non-Iceberg tables.

Users who only use Iceberg tables do not need to use this client and can use the Iceberg-provided Spark client.

This page documents how to connect Spark with Polaris Service using the Polaris Spark client.

Quick Start with Local Polaris Service🔗

If you want to quickly try out the functionality with a local Polaris service, simply check out the Polaris repo and follow the instructions in the Spark plugin getting-started README.

Polaris Evolution

Mon, 01 Jan 0001 00:00:00 +0000

This page discusses what can be expected from Apache Polaris as the project evolves.

Using Polaris as a Catalog🔗

Polaris is primarily intended to be used as a Catalog of Tables and Views. As such, it implements the Iceberg REST Catalog API and its own REST APIs.

Revisions of the Iceberg REST Catalog API are controlled by the Apache Iceberg community. Polaris attempts to accurately implement this specification. Nonetheless, optional REST Catalog features may or may not be supported immediately. In general, there is no guarantee that Polaris releases always implement the latest version of the Iceberg REST Catalog API.

Building a Custom Server Based on Polaris

Mon, 01 Jan 0001 00:00:00 +0000

This page highlights a few keys points for building custom servers based on Apache Polaris.

Polaris produces several jars. These jars may be used to build custom catalog server according to the terms of the license included in Polaris distributions.

General Principles🔗

Create a module in your project (Maven, Gradle or any other build tool) for a Quarkus Application.
Include polaris-runtime-service as a runtime or compile-time dependency.
Include / exclude other dependencies according to the project’s needs (e.g., an alternative JDBC driver).
Do not include polaris-runtime-defaults or polaris-server as dependencies.
Provide your own application.properties file for your Quarkus build.
Define your own Quarkus Application Name and Version (do not use “Apache Polaris” as the application name).

Background🔗

The polaris-server module is intended to be a particular way to build a Polaris server. It includes dependencies that are considered essential for the Apache Polaris distribution. This is not necessarily the case for downstream builds, which may want to exclude some features and include others (perhaps custom implementations of some Polaris SPI interfaces).