Structured and Unstructured Table Locations

Polaris supports two modes that decide where Iceberg tables and views may be located in a catalog:

• Structured locations, the default. A table’s or view’s location must be inside the location of its namespace, and a table’s write.data.path and write.metadata.path must be inside the table’s own location.
• Unstructured locations. Table and view locations, and write paths, may be anywhere inside the catalog’s allowed locations.

In both modes, all locations must be inside the catalog’s allowed locations. Unstructured locations are enabled by setting the feature ALLOW_UNSTRUCTURED_TABLE_LOCATION or the catalog property polaris.config.allow.unstructured.table.location to true.

This page describes what each mode checks, how the effective mode of a catalog is determined, and what happens to existing tables and views when it changes. Switching from structured to unstructured locations does not affect existing entities. Switching back can make existing entities fail, and is described in detail below.

This page is about Iceberg tables and views. Generic tables are only checked when they are created with an explicit base location. They are not validated on load or drop the way Iceberg tables are on commit and credential vending.

External catalogs always allow unstructured locations, whatever the setting is, so the namespace and table location checks on this page do not apply to them.

Structured locations🔗

With structured locations, Polaris checks that a table’s or view’s location is inside its namespace’s location, and that a table’s write.data.path and write.metadata.path are inside the table’s own location, in these cases:

• Creating or registering a table or view, and changing the location of a table or view or the write paths of a table.
• Renaming a table or view into another namespace. The locations must be inside the destination namespace’s location.
• Committing to an existing table. The new metadata file and any other referenced locations must be inside both the table’s namespace location and the table’s own location. This applies to every commit, including appends.
• Vending credentials for an existing table, when a client loads the table with access delegation or refreshes its credentials.

A failed check returns HTTP 403 (ForbiddenException). Listing tables and loading a table without vended credentials are not affected.

The overlap checks and the metadata file check described below also apply. Since every location is inside its namespace, comparing a new location with the entities under the same parent, including their write paths, is enough to detect overlaps.

Unstructured locations🔗

With unstructured locations, the checks above only require locations and write paths to be inside the catalog’s allowed locations. The other location checks still apply:

• Overlap checks, unless ALLOW_TABLE_LOCATION_OVERLAP is enabled. By default, they only compare a location with entities under the same parent, so a table placed outside its namespace can overlap a table or namespace elsewhere in the catalog without being detected. Enabling OPTIMIZED_SIBLING_CHECK checks overlaps with the locations of tables, views, and namespaces across the whole catalog. Checks against other tables’ write paths stay limited to siblings under the same parent.
• Metadata files must be inside the table’s location, unless ALLOW_EXTERNAL_METADATA_FILE_LOCATION is enabled.

Locations generated by Polaris still follow the namespace hierarchy, unless the object storage prefix described below is enabled.

The effective mode of a catalog is determined like for any other feature, see How the effective value of a feature is determined. The catalog property is polaris.config.allow.unstructured.table.location, and the default is false. The deprecated catalog property allow.unstructured.table.location is still honored when the current one is not set.

Switching from structured to unstructured locations🔗

Enabling unstructured locations does not affect existing tables and views. Every location that is valid with structured locations is also valid with unstructured locations, so only new requests are checked differently.

Switching from unstructured to structured locations🔗

The change does not validate existing entities🔗

Polaris does not scan, move, or repair table and view locations when the setting changes. The configuration change itself always succeeds. Tables and views that follow the structured rules keep working. The others fail the structured location checks the next time one of those operations runs.

Validating every table and view is not required before the change. You can enforce structured locations immediately and fix incompatible entities afterwards, or find and fix them first, as described in Finding and fixing incompatible locations.

Changes that switch a catalog to structured locations🔗

The effective value of a catalog changes from true to false in any of these cases:

Example: operations that may fail after the switch🔗

Any operation that involves a location check can fail for an entity whose locations do not follow the structured rules, including updating, renaming, or vending credentials for it. The check is against the entity’s current parent, so it fails whenever the parent’s location does not contain the entity’s locations. Creating a new entity is not affected by the switch itself as its locations are checked against the rules in effect when it is created, whichever mode that is.

For example:

1. With unstructured locations enabled, create table ns1.t1 without a location. Polaris places it inside the location of ns1, for example s3://bucket/warehouse/ns1/t1/.
2. While unstructured locations are still enabled, rename the table to ns2.t2. The rename only changes the table’s name and namespace: its metadata and data files stay under s3://bucket/warehouse/ns1/t1/. With structured locations, this rename would itself fail, because the table’s location is not inside ns2. It only succeeds here because unstructured locations are enabled.
3. Disable unstructured locations for the catalog. Nothing is checked at this point.
4. From now on, loading ns2.t2 with vended credentials, updating it, or renaming it again fails with HTTP 403, because its location is not inside the location of ns2.

The same happens, for example, to a table created with an explicit location outside its namespace, or with a write.data.path outside its own location. Tables that follow the structured rules are not affected.

Interaction with object storage prefix for table locations🔗

DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED (the catalog property polaris.config.default-table-location-object-storage-prefix.enabled) places new Iceberg tables and views under the catalog’s default-base-location with a hash prefix, not under their namespace location. That layout requires unstructured locations to be enabled and, unless ALLOW_TABLE_LOCATION_OVERLAP is enabled, OPTIMIZED_SIBLING_CHECK as well, so that overlaps are checked across the whole catalog and not only within a namespace.

If you disable unstructured locations while the prefix flag stays enabled:

• Every existing Iceberg table or view created under the prefix layout sits outside its namespace. Commits and credential vending for those entities start failing with HTTP 403.
• New Iceberg tables or views created without a client-supplied location fail at create time with IllegalStateException, not HTTP 403, because Polaris refuses to generate a prefixed location without unstructured locations.

Turn off the prefix flag before or at the same time as disabling unstructured locations. Disabling only unstructured locations leaves every prefixed table incompatible and blocks new default-location creates while the prefix flag remains on.

Finding and fixing incompatible locations🔗

When the switch happens, Polaris does not check or report location incompatibilities for the existing entities of the catalogs concerned. The failures only happen later, whenever a location check runs for an incompatible entity, for example when it is updated or renamed. If you want to avoid these failures, you have to find and fix the incompatible entities yourself. To find them, check for each table and view, as returned when loading it:

• That the table or view location is inside the location of its namespace. If the namespace has no location property, its location is built from the closest parent that has one (or the catalog’s default-base-location) plus the names of the namespaces below that parent. For example, namespace a.b without a location in a catalog with default-base-location s3://bucket/warehouse is located at s3://bucket/warehouse/a/b.
• That the write.data.path and write.metadata.path table properties, if they are set, are inside the table’s location.

Typical sources of incompatible locations are tables renamed across namespaces, tables created with an explicit location or with write paths outside the table’s location, tables registered or migrated from another catalog, and Iceberg tables or views created while DEFAULT_LOCATION_OBJECT_STORAGE_PREFIX_ENABLED was enabled (every such table sits outside its namespace).

To fix an incompatible table or view:

• Rename it back into a namespace whose location contains its files.

• Copy the data to a location inside the namespace, for example with a CREATE TABLE ... AS SELECT statement, and drop the old table.

• Move the table to a new location while keeping its history, using Iceberg’s rewrite_table_path Spark procedure. It writes copies of the table’s metadata files, with every path that starts with source_prefix replaced by target_prefix, to a staging location and produces a list of the files to copy. It does not copy any file itself. Copy the listed files to the target location with a tool such as DistCp, then register the table at the new location with the register_table procedure, and drop the old table. For example:

  CALL catalog_name.system.rewrite_table_path(
      table => 'ns2.t2',
      source_prefix => 's3://bucket/warehouse/ns1/t1',
      target_prefix => 's3://bucket/warehouse/ns2/t2'
  );
  

• Keep unstructured locations enabled for the catalog that needs them, by setting the catalog property to true.

• If the catalog uses the object storage prefix for default table locations, disable polaris.config.default-table-location-object-storage-prefix.enabled (or move tables off the prefixed layout) before or together with turning unstructured locations off.

If operations start failing with HTTP 403 after the change, setting the catalog property back to true restores the previous behavior while you fix the affected entities.

Related pages🔗

• Storage Layout
• Configuring Polaris for Production
• Configuration Reference