Amazon Aurora PostgreSQL Integrates DuckDB Engine to Enable Direct Queries on Data Lakes Without ETL

SEATTLE — In a major architectural leap for enterprise database management, Amazon Web Services (AWS) has announced a powerful new capability for Amazon Aurora PostgreSQL. By embedding the high-performance analytical engine DuckDB directly into the database service, AWS now allows developers and database administrators to query operational data and data lake assets simultaneously.

The integration effectively bridges the historical divide between transactional databases and analytical data repositories. Organizations can now run unified queries across live operational tables—including uncommitted writes—and massive historical archives stored in Apache Iceberg and Apache Parquet formats on Amazon S3 and S3 Tables, all without building complex, costly, and resource-intensive Extract, Transform, Load (ETL) pipelines.

Main Facts

The newly released capability is designed to remove friction from modern application architectures, particularly those supporting real-time dashboards, contextual transaction enrichment, and generative AI agents.

• Direct Data Lake Querying: Aurora PostgreSQL can now query Apache Iceberg and Apache Parquet datasets stored in Amazon S3, S3 Tables, and Iceberg REST Catalog (IRC)-compatible repositories via AWS Glue Data Catalog federation.
• DuckDB Integration: Following the integration of DuckDB—the analytical database project maintained by DuckLabs, whose team recently joined Amazon—Aurora leverages the engine’s core efficiency. DuckDB executes analytical scans directly within the Aurora instance, eliminating additional network hops and data duplication.
• Zero ETL Architecture: By utilizing foreign tables with automatic schema inference, applications can query operational and historical data side-by-side using standard PostgreSQL syntax, drivers, and client tools like psql.
• Advanced Optimizations: The system incorporates query optimizations such as predicate pushdown and column pruning, ensuring that only relevant data blocks are read from S3. Frequently accessed data is automatically cached locally within the Aurora instance.
• Availability and Pricing: The feature is supported on Amazon Aurora PostgreSQL major versions 17 (starting with version 17.11) and 18 (starting with version 18.6). It is available globally across all commercial AWS Regions and AWS GovCloud (US) Regions at no additional software charge, with customers paying only for incremental Aurora compute and standard Amazon S3 request costs.

Chronology of Development

The journey toward bridging transactional databases and data lakes has been a central focus of enterprise database engineering for over a decade.

• The Traditional Era of Silos: Historically, enterprise data was strictly partitioned. Relational databases like PostgreSQL handled fast, row-based transactional (OLTP) workloads, while data lakes stored massive columnar files (Parquet, ORC, and later Iceberg) for batch processing and business intelligence.
• The Rise of Reverse ETL: As applications required historical context—such as fraud detection models evaluating a live transaction against five years of history—architects resorted to reverse ETL pipelines. These pipelines regularly copied, transformed, and duplicated data into operational databases, driving up cloud infrastructure costs and introducing latency and synchronization headaches.
• The Advent of Open Table Formats: The popularization of open table formats like Apache Iceberg brought transactional guarantees to data lakes, making them more reliable. However, querying them still traditionally required specialized query engines (like Amazon Athena or Amazon EMR) rather than direct operational access.
• The DuckLabs Acquisition and Integration: A pivotal turning point arrived when the team behind DuckDB joined Amazon. Rather than forcing external data virtualization layers, AWS engineers embedded the lightweight, vector-optimized analytical execution engine directly inside the Aurora PostgreSQL memory and execution space.
• General Availability (Late 2026): AWS officially rolled out the capability for Aurora PostgreSQL versions 17 and 18, enabling seamless integration with S3, S3 Tables, and external IRC-compatible catalogs through AWS Glue.

Supporting Data and Technical Implementation

To understand the practical mechanics of the new Aurora PostgreSQL capability, consider a standard financial application tracking customer accounts.

Setting Up the Extension

Database administrators begin by creating an Aurora PostgreSQL cluster, attaching an appropriate AWS Identity and Access Management (IAM) role with the AuroraAnalytics feature enabled to grant access to S3 and the Glue Data Catalog, and initializing the extension via SQL:

CREATE EXTENSION aurora_analytics;

Pointing to Data Lakes

Instead of manually defining column schemas for large external datasets, administrators can point directly to a Parquet file or an Iceberg table. Aurora reads the metadata directly:

CREATE FOREIGN TABLE transaction_history ()
SERVER aurora_analytics_server
OPTIONS (
    location 's3://<my-bucket>/finance/transaction_history.parquet',
    format 'parquet'
);

For large-scale data lakes containing hundreds of tables, engineers can skip individual definitions entirely using a bulk-creation command: IMPORT FOREIGN SCHEMA.

Executing Unified Queries

Once the foreign table is established, applications can execute complex joins and unions that pull real-time data from local Aurora tables alongside archival data from S3 in a single command:

SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
  AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;

Under the hood, DuckDB handles the high-speed analytical scan of the columnar Parquet data on S3, while Aurora manages the operational table reads. Furthermore, engineers can monitor performance metrics—such as rows scanned, bytes read from S3, and local cache hits—using the built-in diagnostic function aurora_analytics_stat_statements().

For workloads demanding single-digit-millisecond latency, data from the lake can be materialized permanently into native Aurora tables using standard commands like CREATE TABLE AS SELECT or MERGE INTO, providing a flexible path for frequently accessed "hot" data.

Official Responses and Perspectives

AWS engineering leadership emphasizes that this release marks a fundamental shift in how developers think about data architecture boundaries.

According to product statements released by the AWS Aurora team, the primary motivation was to eliminate the operational tax of data duplication. "Previously, if your application needed to combine recent transactional data in Aurora with historical records stored in Amazon S3, a common approach was to build reverse ETL pipelines that duplicated data, increased infrastructure costs, and required ongoing engineering effort to keep everything synchronized," notes the engineering team.

This challenge has been sharply magnified by the rapid enterprise adoption of artificial intelligence. "This challenge only grows as you increasingly embed AI agents into your applications, where it is impractical to predict and pre-replicate every dataset an agent might need," the team explained.

By integrating DuckDB, AWS has created a conduit where AI agents and standard operational applications can reason over live, uncommitted data and petabytes of archival data simultaneously through a unified interface. By building this architecture around DuckDB, AWS has also ensured that future open-source enhancements to the analytics engine will automatically flow into Aurora and other AWS services, establishing a sustainable path for long-term performance gains.

Implications for the Enterprise

The release of direct Apache Iceberg and Parquet querying inside Amazon Aurora PostgreSQL carries profound implications for software engineering, data engineering, and overall IT expenditure.

1. Simplification of Data Architectures

For years, organizations maintained strict organizational and technical boundaries between data engineering teams (managing data lakes and ETL pipelines) and application development teams (managing operational databases). This feature blurs those lines, allowing standard application developers to leverage deep historical archives using simple SQL, without writing complex integration code or relying on data engineering teams to provision intermediate data warehouses.

2. Cost Reduction and Efficiency

Data duplication is one of the leading hidden drivers of cloud storage and compute bloat. By querying data in place on Amazon S3 and leveraging DuckDB’s columnar vectorization, enterprises avoid the storage costs of duplicate data warehouses and the compute costs of running continuous, heavy ETL jobs. Furthermore, offloading analytical scans to read replicas ensures that primary transactional workloads remain undisturbed.

3. Unlocking the Potential of Generative AI

As businesses rush to deploy autonomous AI agents capable of contextual reasoning, the velocity and breadth of data access become critical. An AI customer-support agent can now query a customer’s live shopping cart while simultaneously analyzing five years of purchasing history across external Iceberg catalogs in a single round-trip query—all without human engineers having to anticipate and pre-load those datasets into a dedicated analytical store.

Summary

Amazon Aurora PostgreSQL’s new native data lake querying capability represents a maturation of cloud-native database design. By blending the transactional reliability of PostgreSQL with the analytical velocity of DuckDB and the massive, cost-effective storage of S3 and Apache Iceberg, AWS has provided enterprises with a streamlined, zero-ETL path forward.