AWS Unveils Direct Data Lake Querying for Amazon Aurora PostgreSQL Powered by Embedded DuckDB

SEATTLE — In a major development for database management and data lake architectures, Amazon Web Services (AWS) has announced a powerful new capability for Amazon Aurora PostgreSQL. The feature allows developers and database administrators to directly query operational data alongside analytical data stored in Apache Iceberg and Apache Parquet formats using standard PostgreSQL applications and tools.

By embedding the high-performance analytical engine DuckDB directly into Aurora PostgreSQL, AWS has eliminated the traditional requirement for complex Extract, Transform, Load (ETL) pipelines and reverse-ETL syncing mechanisms. The announcement marks one of the most prominent integrations of DuckDB technology following the recent acquisition of DuckLabs—the team behind the DuckDB project—by Amazon.

Available immediately across all commercial and AWS GovCloud (US) Regions at no additional software charge, this capability empowers enterprises to unify their operational transactions and historical archives under a single, familiar SQL interface.

Main Facts

The core innovation of the new Aurora PostgreSQL capability lies in its ability to bridge the gap between high-throughput operational databases and cost-effective cloud data lakes without data duplication.

• Direct Multi-Source Querying: Users can execute a single SQL query that joins live operational data (including uncommitted writes) inside Aurora with massive historical archives stored in Amazon S3, S3 Tables, or tables managed by the AWS Glue Data Catalog and Iceberg REST Catalog (IRC)-compatible endpoints.
• DuckDB Integration: The efficiency of the DuckDB analytical engine is embedded directly within the database instance. Query processing occurs entirely within Aurora, avoiding external network hops, intermediate staging storage, or data duplication.
• Supported Versions: The feature is natively supported on Amazon Aurora PostgreSQL version 17 (starting with 17.11) and version 18 (starting with 18.6).
• Automatic Schema Inference: Utilizing commands such as CREATE FOREIGN TABLE with empty schema definitions or bulk-importing via IMPORT FOREIGN SCHEMA, Aurora automatically extracts and infers schemas directly from Parquet and Iceberg metadata files.
• Advanced Optimizations: The engine automatically applies internal optimizations such as predicate pushdown and column pruning, ensuring that only relevant rows and columns are scanned from Amazon S3. Frequently accessed objects can also be cached locally within the Aurora instance to drastically accelerate repeat queries.
• Cost Structure: There are no licensing or standalone feature fees. Customers pay only for the incremental Aurora compute resources consumed by the analytical queries and standard Amazon S3 API request costs.

Chronology of Development

The journey toward bridging transactional databases and analytical data lakes has been a long-standing challenge in enterprise software engineering. To understand the significance of the new Aurora PostgreSQL release, it is helpful to trace the technical evolution:

• The Era of Siloed Architectures: Historically, relational databases like PostgreSQL were optimized for low-latency, high-concurrency transactional processing (OLTP), while data lakes on Amazon S3 handled large-scale analytical processing (OLAP). The two systems remained strictly separated.
• The Rise of Complex ETL Pipelines: To enable business intelligence or customer insights, organizations built brittle, expensive ETL and reverse-ETL pipelines to copy data from operational databases into data lakes—and vice versa—incurring significant infrastructure maintenance costs and data synchronization delays.
• The Rise of Lakehouse Architectures and Open Table Formats: The adoption of open storage formats like Apache Parquet and open table formats like Apache Iceberg allowed organizations to decouple storage from compute, centralizing historical data in cloud object storage. However, querying this data still required specialized engines like Amazon Athena, Amazon Redshift, or Apache Spark, leaving operational databases out of the direct analytical loop.
• DuckLabs Joins Amazon: In a strategic talent and technology acquisition, the engineering team behind DuckDB joined Amazon, bringing advanced, vector-optimized in-memory analytical processing capabilities into the fold.
• The Integration Milestone: AWS successfully embeds DuckDB directly into the architecture of Amazon Aurora PostgreSQL. This allows database instances to perform high-speed analytical scans on external data lake files while continuing to serve core application traffic.

Supporting Data and Technical Architecture

The architecture of the new feature is designed to maintain high performance and operational transparency for developers.

To operationalize the capability, administrators execute a straightforward setup process:

1. Create an Aurora PostgreSQL cluster running version 17.11+ or 18.6+.
2. Attach an AWS Identity and Access Management (IAM) role containing the AuroraAnalytics policy to grant Aurora secure read access to Amazon S3 and the AWS Glue Data Catalog.
3. Enable the extension within the database via a simple command:

   CREATE EXTENSION aurora_analytics;

4. Define foreign tables pointing directly to S3 storage locations or Glue databases.

Practical Execution Example

In a typical financial services use case, an enterprise might maintain the last seven days of active customer transactions in an operational recent_transactions table, while archiving five years of older transaction history as Parquet files in an S3 bucket.

Instead of moving historical data into the database or writing complex application-layer joins, an engineer can point a foreign table directly to the S3 bucket:

CREATE FOREIGN TABLE transaction_history ()
SERVER aurora_analytics_server
OPTIONS (
    location 's3://my-enterprise-bucket/finance/transaction_history.parquet',
    format 'parquet'
);

When executing a unified query, DuckDB processes the analytical scan over the S3 Parquet files under the hood, while Aurora handles the live operational table data:

SELECT merchant, category, amount, transaction_date, 'recent' AS source
FROM recent_transactions
WHERE customer_id = 'C-1001'
UNION ALL
SELECT merchant, category, amount, transaction_date, 'historical' AS source
FROM transaction_history
WHERE customer_id = 'C-1001'
  AND transaction_date >= CURRENT_DATE - INTERVAL '5 years'
ORDER BY transaction_date DESC
LIMIT 15;

Performance Monitoring and Materialization

Engineers can inspect the efficiency of these cross-system queries using the built-in aurora_analytics_stat_statements() diagnostic function. This function exposes critical runtime metrics, including total rows scanned, exact bytes read from Amazon S3, and local cache hit ratios.

Furthermore, for workloads requiring single-digit-millisecond operational latency, data engineers can easily materialize data from the lake directly into native Aurora tables using standard SQL commands such as CREATE TABLE AS SELECT, INSERT INTO ... SELECT, or MERGE INTO. These materialized tables can then be queried across any writer or read-replica node within the Aurora cluster, effectively offloading heavy analytical scans from primary operational workloads.

Official Responses and Industry Perspective

While AWS engineering teams have long emphasized the need to reduce infrastructure friction, the integration of DuckDB marks a definitive shift toward embedding specialized analytics engines directly into general-purpose transactional databases.

According to technical documentation and release communications authored by AWS product specialists, the overarching goal is to remove the "operational tax" traditionally imposed by multi-system architectures.

"Whether you’re powering real-time dashboards, enriching transactions with historical context, or building AI agents that reason over both live and archived data, you can now do it all through a single, familiar interface," AWS engineering representatives noted in the official product announcement.

Industry analysts view the move as a direct response to the increasing complexity of modern data stacks. As enterprises accelerate their deployment of generative artificial intelligence (AI) and autonomous AI agents, the demand for instant, frictionless access to both live transactional state and deep historical records has skyrocketed. Pre-replicating every conceivable dataset into a transactional database for an AI agent is mathematically and economically unfeasible; direct querying over open formats like Iceberg and Parquet solves this bottleneck elegantly.

Implications for Developers and Enterprises

The introduction of direct data lake querying within Amazon Aurora PostgreSQL carries profound implications for software architecture, infrastructure spending, and application design.

1. Simplification of Application Architecture

Developers no longer need to write, test, and maintain complex reverse-ETL pipelines or sync scripts just to provide historical context to transactional applications. By treating S3-backed data lakes as native foreign tables, applications can query vast repositories of historical information using standard PostgreSQL drivers, ORMs (Object-Relational Mappers), and BI tools.

2. Accelerated AI and Agentic Workflows

Autonomous AI agents require broad situational awareness, pulling from live system states as well as historical trends, logs, and logs of past customer interactions. This native integration enables AI applications to execute complex reasoning tasks over multi-year datasets without requiring custom vector databases or dedicated data-staging workflows.

3. Cost and Storage Efficiency

Organizations can avoid the costly anti-pattern of data duplication. By keeping historical records in low-cost Apache Iceberg or Parquet formats on Amazon S3—while querying them on-demand via DuckDB—companies only pay for active compute when queries are actively running.

4. Zero-Migration Friction for Existing PostgreSQL Tools

Because the feature relies entirely on standard PostgreSQL syntax and foreign table abstractions, existing enterprise tooling, monitoring solutions, and migration scripts continue to work without modification. Database administrators can scale out read workloads across Aurora replicas, ensuring high availability and robust performance even under heavy analytical query loads.