Storage Layout

This page explains where Polaris places namespaces, tables, and views in object storage, and how the settings that change this layout interact. Several of these settings have overlapping effects, and some combinations are rejected, so it is worth reading this page before changing any of them.

This page covers internal catalogs. Federated catalogs are served by a remote catalog, which owns the table locations, so none of the rules on this page apply to them. External catalogs without a federation connection store their tables in Polaris, but the locations are set by the system that updates them rather than generated by Polaris. Only some of the rules below apply to them, and the sections below note where they differ.

All feature flags mentioned here are described in the configuration reference. Most of them can be set for a whole realm with polaris.features."<FLAG>", or for a single catalog with the catalog property listed in the summary of settings at the end of this page.

Default layout🔗

By default, Polaris uses a structured layout that mirrors the namespace hierarchy:

• The catalog’s default-base-location property is the root of the catalog. If the catalog’s storage configuration does not list any allowedLocations, the default base location is the only allowed location.
• A namespace’s location is its parent’s location followed by the namespace name. The parent is either the catalog or the enclosing namespace.
• A table’s or view’s location is its namespace’s location followed by the table or view name.
• Within a table’s location, Iceberg writes metadata files under metadata/ and data files under data/.

Polaris always ends namespace and table locations with a slash. For example, with default-base-location set to s3://bucket/warehouse and a table sales.emea.orders:

s3://bucket/warehouse/                                  catalog
s3://bucket/warehouse/sales/                            namespace sales
s3://bucket/warehouse/sales/emea/                       namespace sales.emea
s3://bucket/warehouse/sales/emea/orders/                table sales.emea.orders
s3://bucket/warehouse/sales/emea/orders/metadata/00000-<uuid>.metadata.json
s3://bucket/warehouse/sales/emea/orders/data/<partition>/<file>.parquet

With the default settings, Polaris also enforces the following rules:

• A client may supply its own table or view location in the location field, but it must be inside the namespace’s location and inside the catalog’s allowed locations.
• A client may also set the write.data.path and write.metadata.path table properties, but they must be inside the table’s own location and inside the catalog’s allowed locations.
• A table’s metadata files must be inside the table’s location.
• A new table, view, or namespace location may not overlap the location of a sibling, that is, another table, view, or namespace directly under the same parent. For sibling tables, their write.data.path and write.metadata.path are included in the check, so two tables in the same namespace cannot share a write path.
• A namespace may only be created with a location property if that location is the default one.

Requests that break these rules fail with 403 Forbidden. In internal catalogs, a table or view location outside the namespace, or a write path outside the table’s location, requires ALLOW_UNSTRUCTURED_TABLE_LOCATION (see Unstructured table locations); without it, the request is rejected.

External catalogs do not depend on this flag: their table locations and write paths only need to be inside the catalog’s allowed locations. Polaris does not generate table locations in external catalogs, so the settings below that only affect generated locations have no effect there.

Unique table locations🔗

When a table is dropped and a new table with the same name is created, the default layout gives both tables the same location. Setting DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED to true appends a random suffix to the table or view name in generated locations, so that every new table gets its own directory:

s3://bucket/warehouse/sales/emea/orders-<32 hexadecimal characters>/

The suffix is only added when Polaris generates the location. Client-supplied locations are used as given.

Custom namespace locations🔗

To create or update a namespace with a location property that is different from its default location, enable the behavior change ALLOW_NAMESPACE_CUSTOM_LOCATION. Tables and nested namespaces created later in that namespace are placed under its custom location.

Unstructured table locations🔗

Setting ALLOW_UNSTRUCTURED_TABLE_LOCATION to true lets table and view locations, and the write.data.path and write.metadata.path table properties, be anywhere inside the catalog’s allowed locations, instead of only inside their namespace’s location or, for write paths, the table’s own location. Generated locations still follow the default layout unless the object storage prefix described below is enabled.

Related settings:

• ALLOW_CLIENT_SPECIFIED_TABLE_LOCATION, which is true by default, controls whether clients may supply locations at all. When it is false, requests that set a location or a write path property are rejected, whatever the other flags are.
• ALLOW_EXTERNAL_METADATA_FILE_LOCATION separately allows metadata files outside the table’s location. ALLOW_UNSTRUCTURED_TABLE_LOCATION does not affect this check.

Object storage prefix for table locations🔗

Object stores such as Amazon S3 scale request rates by key prefix. When many tables in the same namespace share a long common prefix, such as s3://bucket/warehouse/sales/emea/, their traffic is concentrated on a narrow part of the key space.

Setting DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED to true changes the generated location of new tables and views. Instead of placing them under their namespace’s location, Polaris places them directly under the catalog’s default-base-location, behind a hash of the table identifier, followed by the namespace levels and the table name:

s3://bucket/warehouse/1111/1111/0100/01010000/sales/emea/orders/

The hash values in the examples on this page are illustrative. The hash is computed from the table identifier, so it is the same for a given identifier and differs between tables.

Things to be aware of:

• All files of a table share the same hash prefix. This spreads different tables across the key space, but does not spread the files of a single busy table. For that, use Iceberg’s object store layout, described in the next section.
• The prefix is only added when Polaris generates the location. Client-supplied locations are used as given.
• The prefixed location is built from the catalog’s default-base-location, so a custom namespace location is not used for these tables.
• Existing tables and views are not moved.
• When DEFAULT_UNIQUE_TABLE_LOCATION_ENABLED is also enabled, the table name in the prefixed location gets the unique suffix.

Because prefixed tables are not inside their namespace’s location, this flag only works together with other settings. If they are not set, creating a table or view without a location fails with an error that names the missing setting:

• ALLOW_UNSTRUCTURED_TABLE_LOCATION must be true.
• Unless ALLOW_TABLE_LOCATION_OVERLAP is true, which disables overlap protection and is not recommended, OPTIMIZED_SIBLING_CHECK must be true so that table and namespace locations are checked across the catalog rather than only within the namespace.

For example, for a catalog in a new realm, set the catalog properties:

polaris.config.default-table-location-object-storage-prefix.enabled=true
polaris.config.allow.unstructured.table.location=true

and enable the optimized sibling check for the realm:

polaris.features."OPTIMIZED_SIBLING_CHECK"=true
polaris.features."ALLOW_OPTIMIZED_SIBLING_CHECK"=true

Only set ALLOW_OPTIMIZED_SIBLING_CHECK for new realms. See Overlap checks for the reason.

Iceberg object store layout🔗

Iceberg has its own layout option, enabled with the table property write.object-storage.enabled. It is independent of Polaris: it applies only to data files, and adds a hash computed from each file’s name under the table’s data location. Metadata files are not affected.

To enable it for all new tables in a catalog, set the catalog property table-default.write.object-storage.enabled=true.

s3://bucket/warehouse/sales/emea/orders/data/0011/0100/1011/11101010/<partition>/file1.parquet
s3://bucket/warehouse/sales/emea/orders/data/0011/0001/0001/00000001/<partition>/file2.parquet

The two options work at different levels and can be combined:

With both enabled, a table’s data files look like this:

s3://bucket/warehouse/1111/1111/0100/01010000/sales/emea/orders/data/0011/0100/1011/11101010/<partition>/file1.parquet

Sharing a data location between tables🔗

If write.data.path points outside the table’s location, Iceberg’s object store layout puts the per-file hash directly under write.data.path, followed by the last namespace level and the table name:

<write.data.path>/<file hash>/<namespace>/<table>/<partition>/<file>

Pointing write.data.path at a short shared location, such as the warehouse root, therefore spreads the data files of all tables over the key space. Polaris validates write.data.path and checks it for overlaps, though, so this layout needs two relaxations:

• ALLOW_UNSTRUCTURED_TABLE_LOCATION, because without it write.data.path must be inside the table’s own location.
• ALLOW_TABLE_LOCATION_OVERLAP, because two overlap checks reject this layout otherwise, and this flag skips both of them (see Overlap checks):
  • The check when a table is created or updated. It includes sibling tables’ write paths, so sibling tables cannot share a write.data.path, and it rejects a location above other tables and namespaces.
  • For catalogs served by LocalIcebergCatalog, the check before vending storage credentials. It refuses credentials for a table whose write path overlaps another table or namespace.

Disabling overlap protection removes a safeguard that keeps tables from writing into each other’s locations, so only use this layout when storage isolation is enforced outside Polaris.

Overlap checks🔗

Unless ALLOW_TABLE_LOCATION_OVERLAP is enabled, Polaris rejects a table or view whose location, or a table whose write path, is inside, or contains, the location of another table, view, or namespace. The same applies to namespaces, controlled by ALLOW_NAMESPACE_LOCATION_OVERLAP. View locations are only checked while the behavior change VALIDATE_VIEW_LOCATION_OVERLAP is true, which is the default.

By default, the check lists the siblings of the new entity, that is, the entities directly under the same parent. It compares the new location with each sibling’s location and, for sibling tables, with their write.data.path and write.metadata.path. That is enough for the default layout, where each entity’s location is inside its parent’s, but not for unstructured locations.

OPTIMIZED_SIBLING_CHECK first queries a location index kept by the JDBC and NoSQL metastores. The index covers the locations of tables, views, and namespaces across the catalog, but not tables' write paths. When ALLOW_UNSTRUCTURED_TABLE_LOCATION is enabled or the catalog is external, and the index finds no conflict, Polaris still lists the siblings to check their write paths, so those checks stay limited to siblings under the same parent. Otherwise write paths are inside their table’s location and the index alone suffices. Listing siblings can slow validation in namespaces with many entities.

OPTIMIZED_SIBLING_CHECK is a realm-level setting with no catalog property. Realms created by older Polaris versions may not have that index populated, so Polaris refuses to start with OPTIMIZED_SIBLING_CHECK enabled unless ALLOW_OPTIMIZED_SIBLING_CHECK is also set, as an acknowledgment that the realm’s index is complete.

Before vending storage credentials for a table, Polaris also checks that none of the table’s write paths outside its location overlaps another table or namespace. The vended credentials cover these paths, and an overlapping entity may have been created after the table. If a write path overlaps, the request fails with 403 Forbidden. This only affects tables whose write paths can be outside their location, that is, with ALLOW_UNSTRUCTURED_TABLE_LOCATION enabled or in an external catalog. The check is skipped when ALLOW_TABLE_LOCATION_OVERLAP is enabled.

In external catalogs, overlaps are checked when a table is registered, but not when a table is created or updated by a notification.

Summary of settings🔗

See also🔗

• Configuring Polaris for Production
• Configuration reference