AWS Bridges Operational Databases and Data Lakes: Amazon Aurora PostgreSQL Gains Direct Querying Capabilities Powered by DuckDB Integration

SEATTLE — In a major development poised to reshape enterprise data architectures, Amazon Web Services (AWS) has announced a powerful new capability for Amazon Aurora PostgreSQL. Organizations can now directly query operational data stored within Aurora alongside historical data housed in data lakes—using Apache Iceberg and Apache Parquet formats—through their existing PostgreSQL applications and tools.

By bypassing the traditional, complex Extract, Transform, Load (ETL) pipelines that have long plagued data management strategies, this launch aims to streamline application development, reduce infrastructure overhead, and eliminate the latency and cost of data duplication. Powered by the newly integrated DuckDB query engine—following DuckLabs joining Amazon—this update offers a unified interface for real-time dashboards, historical transaction enrichment, and modern artificial intelligence agents that require simultaneous access to live and archived information.

Main Facts

The core innovation of this release centers on bridging the traditional divide between online transactional processing (OLTP) databases and analytical data lakes.

• Direct Data Lake Access: Aurora PostgreSQL users can now query Apache Parquet and Apache Iceberg files stored directly in Amazon S3, S3 Tables, and catalogs compatible with the Iceberg REST Catalog (IRC) standard via the AWS Glue Data Catalog.
• DuckDB Integration: The efficiency of the DuckDB analytical query engine has been embedded directly within Aurora PostgreSQL. DuckDB executes heavy analytical scans locally under the hood, while Aurora handles the live operational workloads, including uncommitted writes, without additional network hops.
• Zero ETL Architecture: By eliminating reverse ETL pipelines, organizations no longer need to constantly sync, duplicate, or predict datasets needed for downstream consumption.
• Version Availability: This feature is currently supported on Amazon Aurora PostgreSQL major versions 17 (starting with version 17.11) and 18 (starting with version 18.6).
• Cost Structure: The capability is available at no additional software charge across all commercial AWS Regions and AWS GovCloud (US) Regions. Users pay only for the incremental Aurora compute consumed by queries and standard Amazon S3 request costs.

Chronology and Technical Evolution

The path toward unifying operational databases and analytical storage has been a multi-decade industry challenge. Historically, relational databases were built for high-throughput, low-latency transactional operations, whereas data lakes were engineered for massive, cost-effective analytical scans.

The Traditional ETL Bottleneck

For years, organizations seeking a 360-degree view of their business operations had to deploy complex data pipelines. If an application required combining recent financial transactions sitting in an Aurora PostgreSQL database with five years of historical logs in Amazon S3, engineers had to build, monitor, and maintain ETL workflows. These pipelines introduced several friction points:

1. Data Duplication: Storing the same data across multiple systems inflated cloud storage costs.
2. Synchronization Lags: Real-time operational decisions were often made on stale data because batch synchronization windows caused delays.
3. Engineering Overhead: Maintaining pipelines consumed valuable engineering hours that could otherwise be spent on core product development.

The AI Imperative

This architectural strain intensified dramatically with the mainstream adoption of generative artificial intelligence and autonomous AI agents. Modern AI applications require dynamic, context-aware reasoning over live operational states and deep historical archives. Because it is practically impossible to predict and pre-replicate every conceivable dataset an AI agent might query on the fly, traditional pre-ETL approaches became unsustainable.

The DuckLabs Acquisition and Integration

The catalyst for the current breakthrough came when DuckLabs—the core team behind the high-performance analytical database project DuckDB—joined Amazon. AWS seized the opportunity to embed DuckDB directly into the Aurora PostgreSQL engine. Instead of moving data across services, the processing logic is brought directly to where the data resides, laying the technical foundation for this zero-ETL, federated querying capability.

Supporting Data and Technical Implementation

Setting up and executing cross-lake queries in Aurora PostgreSQL has been designed to fit seamlessly into existing PostgreSQL workflows. The process relies on standard SQL extensions, foreign data wrappers, and intelligent query optimizations.

Getting Started: Step-by-Step

To activate the capability, database administrators follow a streamlined workflow:

1. Cluster Setup: Provision an Aurora PostgreSQL cluster on version 17.11+ or 18.6+.
2. IAM Configuration: Attach an IAM role equipped with the AuroraAnalytics feature permission to grant Aurora secure access to Amazon S3 and the AWS Glue Data Catalog.
3. Extension Activation: Connect to the database using any standard PostgreSQL client (such as psql) and run the command:

   CREATE EXTENSION aurora_analytics;

4. Foreign Tables: Define foreign tables pointing to data lake assets. Thanks to automatic schema inference from Parquet and Iceberg metadata, administrators can create tables with empty parentheses:

   CREATE FOREIGN TABLE transaction_history ()
   SERVER aurora_analytics_server
   OPTIONS (
      location 's3://<my-bucket>/finance/transaction_history.parquet',
      format 'parquet'
   );

   For bulk deployments, a single IMPORT FOREIGN SCHEMA statement automatically maps an entire AWS Glue Data Catalog database.

Federated Catalogs and Advanced Querying

Beyond single buckets, users can query external IRC-compatible catalogs through AWS Glue Data Catalog federation. By registering an external catalog once with Glue, a single SQL query can join live transactional data from Aurora with Iceberg tables spread across multiple disparate catalogs.

Sample Unified Query:

SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
  AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;

In this execution, DuckDB handles the heavy analytical scan of the multi-year Parquet file in S3, while Aurora processes the recent operational rows—returning a unified result set instantly.

Optimization and Materialization

To ensure high performance at scale, Aurora applies advanced database optimizations, including predicate pushdown and column pruning, ensuring that only relevant data bytes are read from S3. Frequently accessed data is automatically cached locally within the Aurora instance, and performance metrics can be inspected using aurora_analytics_stat_statements().

Furthermore, for workloads demanding single-digit-millisecond latency, users can materialize data lake subsets into native Aurora tables using standard commands like CREATE TABLE AS SELECT or MERGE INTO, offloading analytical scans from primary writers to read replicas.

Official Responses and Industry Perspective

AWS leadership emphasizes that this capability represents a fundamental shift in how databases interact with modern data lakes.

While speaking on the architecture of the new feature, AWS engineering representatives highlighted the philosophy behind the DuckDB integration: "We’re excited to bring the speed and simplicity of DuckDB directly into Aurora PostgreSQL, so you and your agents can query and combine operational and Iceberg data using the familiar PostgreSQL applications, tools, and endpoints already in use. By building this capability around DuckDB, future improvements to the open source engine can continue to bring performance and functionality gains to Aurora and other AWS services."

Industry analysts have similarly noted that the move addresses a critical friction point in enterprise IT. As cloud data estates grow exponentially, organizations have struggled with the administrative overhead of managing separate operational and analytical data silos. By blurring the line between transactional databases and object storage through open table formats like Apache Iceberg, AWS is providing a pragmatic path forward that respects existing developer tooling while embracing modern, open-source analytical innovations.

Implications for the Enterprise

The introduction of direct data lake querying within Amazon Aurora PostgreSQL carries profound implications for software engineering teams, data architects, and artificial intelligence developers.

1. Simplified Application Architecture

Developers are no longer forced to design complex dual-write patterns or maintain fragile reverse-ETL mechanisms just to surface historical context inside operational applications. A single SQL connection to Aurora is now sufficient to access petabytes of archival data.

2. Supercharging AI Agents

As enterprises pivot toward autonomous AI agents capable of executing complex business workflows, access to comprehensive data is paramount. Agents can now query live operational queues alongside years of historical customer interactions without waiting for data engineering teams to provision custom pipelines or data marts.

3. Cost Efficiency and Governance

By keeping data in place within Apache Iceberg and Parquet formats on Amazon S3—and leveraging AWS Glue for catalog management—organizations avoid unnecessary data duplication. This not only lowers cloud storage expenditures but also simplifies data governance, ensuring that compliance and auditing policies can be applied cleanly at the data lake layer.

4. Zero Migration Risk

Because the capability is exposed entirely through standard PostgreSQL syntax, extensions, and wire protocols, organizations can adopt the feature immediately without rewriting existing application logic or replacing established database tools.

As enterprises continue to demand faster insights and tighter integrations between transactional systems and artificial intelligence, features like direct Aurora-to-data-lake querying signal a new era of cohesive, highly efficient cloud database architecture.