What Amazon, Google Cloud, and Apache Iceberg are teaching us about the next generation of data platforms—and why the small-file problem matters to retail

The Hook: The Pipeline Works. So Why Is the Query Still Slow?

Imagine I am building a customer-behavior pipeline for a retailer.

Every time someone searches for a product, views a product detail page, adds something to their cart, or checks availability, an event enters the system.

The business wants to analyze this behavior almost immediately.

The architecture seems straightforward:

Application → Event Stream → Data Lakehouse → Analytics and AI

I can use Kafka or a managed streaming service to capture events. I can store the data in Apache Iceberg tables. Spark or a SQL engine can process it. The resulting data can feed dashboards, personalization models, and AI applications.

But there is a problem.

The data arrives quickly, yet querying it becomes increasingly expensive.

Why?

Because continuous ingestion can create thousands of small files.

Every file adds overhead. Query engines have to discover files, read metadata, plan scans, and process the relevant data. As the table grows, these costs can become significant.

I can increase the ingestion interval to produce larger files, but that makes the data less fresh.

I can reduce the interval, but that may create even more small files.

I can build a separate compaction pipeline, but now I have another job to deploy, monitor, troubleshoot, and pay for.

This is one of the less glamorous problems in data engineering. It is also one of the most important.

In August 2026, Amazon Web Services introduced a new capability called streaming tables in Amazon Kinesis Data Streams. It continuously delivers stream records into Apache Iceberg tables on Amazon S3 Tables, with managed delivery and inline compaction. The announcement directly addresses the operational complexity and small-file problems associated with streaming ingestion. Amazon Web Services, Inc.

Google Cloud has also been investing in managed Iceberg infrastructure, including streaming ingestion, table maintenance, and interoperability between BigQuery and other Iceberg-compatible engines. Its April 8, 2026 announcement described these capabilities as part of a broader effort to make open-format lakehouses more practical for production workloads. Google Cloud

I think these developments point toward an important change.

The next step in data engineering is not simply making data arrive faster. It is making fresh data affordable and practical to query.

1. Why This Matters: Streaming Is Only Half the Problem

For years, data engineering discussions often focused on the ingestion layer.

Can the system process millions of events?

Can it handle traffic spikes?

Can it meet a latency target?

Can it recover after a failure?

Those questions still matter. But they do not describe the complete problem.

A successful streaming pipeline must deliver data into a form that downstream systems can actually use.

Consider a retail application generating customer events.

At the beginning, the pipeline may work perfectly. Events arrive, records are written to object storage, and analysts query the resulting tables.

As traffic grows, however, the workload changes.

A high-volume stream can generate many small Parquet files. The table’s metadata and file layout become increasingly important. Query engines may need to process substantial metadata before they even start reading the relevant records.

The pipeline is delivering data, but the analytical experience is getting worse.

AWS identified this exact tension in its August 31, 2026 engineering article about Kinesis streaming tables. The company described how high-volume streaming ingestion can produce small files and how coordinating writes across stream shards can create commit conflicts, freshness delays, and degraded query performance. Amazon Web Services, Inc.

This creates three separate engineering problems.

Problem 1: Freshness

How quickly should newly produced data become queryable?

For customer-behavior analytics, five minutes may be acceptable.

For some inventory or fraud-detection use cases, it may not be.

Problem 2: File layout

How should the system organize incoming records so that queries remain efficient?

The answer depends on query patterns, file sizes, partitioning, and how frequently data changes.

Problem 3: Operational complexity

Who is responsible for delivery retries, write coordination, schema validation, compaction, and monitoring?

If every team builds these capabilities independently, engineering effort gets duplicated.

This is why managed streaming-to-lakehouse services are worth paying attention to.

They target the infrastructure between ingestion and consumption—the part of the architecture that can become expensive to operate without delivering much direct business differentiation.

My observation: A pipeline is not truly successful just because its ingestion metrics look healthy. It also needs to produce data that downstream systems can consume at an acceptable cost and latency.

2. What’s Actually Happening: Three Developments Worth Watching

A. Amazon Is Making Streaming-to-Iceberg Delivery a Managed Capability

On August 31, 2026, Amazon Web Services published its engineering explanation of Kinesis streaming tables.

The service continuously delivers records from an Amazon Kinesis Data Streams stream to Apache Iceberg tables backed by Amazon S3 Tables.

AWS says the service handles delivery, write coordination, retries, and inline compaction. The resulting tables can be queried using services such as Amazon Athena, Amazon Redshift, and Apache Spark. The tables are also registered in AWS Glue Data Catalog. Amazon Web Services, Inc.

The architectural difference is significant.

Previously, a team building its own streaming-to-Iceberg pipeline might have needed to operate:

• A streaming consumer or connector.

• Compute infrastructure for delivery.

• Schema conversion logic.

• Retry and failure-handling mechanisms.

• A strategy for coordinating table writes.

• Compaction jobs.

• Monitoring and operational alerts.

A managed delivery capability can remove much of that plumbing.

AWS also reports that streaming tables can reduce data-delivery costs by up to 50% and downstream query costs by up to 30% compared with the alternatives described in its announcement. These are AWS-reported savings, not guaranteed results for every workload. Actual savings depend on ingestion volume, existing infrastructure, query patterns, and pricing. Amazon Web Services, Inc.

The important point is not the headline percentage.

It is the fact that the cloud provider is treating delivery and file optimization as part of the managed streaming service.

That is a useful direction for data platforms.

B. Google Cloud Is Bringing More Lakehouse Management Into the Platform

Google Cloud is approaching a similar problem from the lakehouse side.

In its April 8, 2026 announcement, Google described read-and-write interoperability between BigQuery and Iceberg-compatible engines, including Spark, Flink, and Trino, using its managed Iceberg REST Catalog.

The announcement also covered managed table maintenance, compaction, garbage collection, and streaming support. Google described its Vortex streaming infrastructure as enabling high-throughput ingestion with near-zero read latency for supported BigQuery-managed Iceberg tables. Some of the newly announced capabilities for Google-managed REST Catalog tables were in preview at the time. Google Cloud

The direction is similar to AWS, although the products and implementation details differ.

Google is trying to make it easier to combine:

• Open table formats.

• Managed storage and metadata.

• Multiple processing engines.

• Streaming ingestion.

• Automated maintenance.

• Centralized governance.

This matters because a lakehouse is rarely just a storage decision.

It is an operating model.

A company might use Spark for transformations, BigQuery for analytics, and another engine for specialized processing. The more engines share the same tables, the more important consistent metadata, access controls, and maintenance become.

Google’s announcement also highlighted the challenge of choosing appropriate file sizes, partitioning strategies, and sort orders for different workloads. These decisions affect performance even when the underlying table format is open and interoperable. Google Cloud

An open table format does not eliminate engineering work.

It changes where that work happens.

C. Apache Iceberg Is Improving the Metadata and Catalog Layer

The open-source project matters just as much as the managed services.

On May 19, 2026, the Apache Iceberg project released version 1.11.0.

One notable improvement was expanded remote scan planning through the REST Catalog protocol. Instead of requiring every client to fetch and interpret table manifests to determine which files to read, a catalog server can plan scans and return the relevant tasks.

The release extended this approach to incremental scans for Structured Streaming workloads and metadata tables.

It also introduced freshness-aware metadata loading and idempotency-key support for mutating catalog operations. The latter helps ensure that retried operations such as commits, creates, and drops do not execute twice because of network timeouts. Apache Iceberg

These are not flashy features.

But they address real operational concerns.

Metadata overhead can affect query planning. Network retries can create difficult failure scenarios. Streaming workloads need efficient ways to identify changes without repeatedly processing everything.

The broader lesson is that the lakehouse ecosystem is improving at several layers simultaneously.

Cloud providers are managing more of the infrastructure.

Open-source projects are improving the protocols and metadata operations.

Data engineers can spend less time rebuilding generic plumbing—but only if they understand the constraints of the managed capabilities they adopt.

3. The Business Impact: What Does This Mean for Retail?

Let’s bring this back to retail.

Imagine a home-improvement retailer collecting events from its website and mobile application.

A customer searches for a cordless drill, views several products, checks battery compatibility, and adds a drill to their cart.

At the same time, the retailer receives inventory updates from stores and fulfillment systems.

The business wants to understand product demand, customer intent, and availability without waiting for an overnight batch job.

A streaming-to-Iceberg architecture could look like this:

Customer events and inventory updates
↓
Event ingestion
↓
Streaming transformations where needed
↓
Apache Iceberg tables on object storage
↓
SQL analytics, feature pipelines, dashboards, and AI applications

Not every event needs the same processing path.

Raw product-view events may be valuable for historical analysis and model training.

Aggregated customer behavior may feed personalization features.

Inventory changes may support near-real-time availability analysis.

A separate low-latency service may still be needed to serve a customer-facing inventory request.

The important distinction is between making data queryable quickly and making a decision synchronously.

Those are different requirements.

Where managed streaming tables could help

Customer behavior analytics

A team could analyze recent searches, clicks, and cart activity without maintaining a separate delivery pipeline solely to make those events available in Iceberg.

AI and ML feature preparation

Fresh event data can be useful for preparing features for downstream machine learning workloads. AWS explicitly identifies AI/ML feature pipelines and personalization among the intended use cases for streaming data delivered to Iceberg. Amazon Web Services, Inc.

However, a queryable Iceberg table is not automatically an online feature store. A model that needs millisecond-level feature retrieval may still require a separate serving system.

Operational analysis

Retail teams could analyze inventory events, order activity, and customer demand with less delay than a conventional batch-only workflow.

Fraud and anomaly analysis

Recent transactions and event patterns can be made available to analytical systems more quickly. Whether this meets a particular detection requirement depends on the latency target and the additional processing involved.

What the business actually gains

The business does not benefit merely because a pipeline has fewer components.

It benefits when engineers can deliver reliable data more quickly, reduce duplicated operational work, and control the cost of analytical access.

This is why I would evaluate a managed streaming capability against business outcomes rather than just infrastructure features.

I would ask:

• How much earlier does data become usable?

• How much operational work disappears?

• Does query performance remain stable as volume grows?

• How much does the entire pipeline cost?

• Can downstream teams trust the data?

Those questions tell me much more than whether the service is labelled real-time.

4. The Engineering Perspective: Managed Does Not Mean Constraint-Free

This is where I think engineers need to be careful.

The new managed capabilities are useful, but they do not remove every design decision.

In fact, understanding their limits is part of using them correctly.

4.1 Five Minutes Is Not Millisecond-Level Real Time

AWS documents a configurable data-freshness interval of 300 to 900 seconds for Kinesis streaming tables. The default is 300 seconds, or five minutes.

The service buffers records until the freshness interval is reached, validates and converts them, compacts and writes the files, and then commits the data to Iceberg so it becomes queryable. AWS Documentation

That is a useful capability.

But it is not the same as a system that updates an online recommendation feature in a few milliseconds.

For a retail analytics dashboard, a five-minute delay may be perfectly reasonable.

For a checkout decision requiring current stock information, it may be too slow.

For a recommendation engine, the answer depends on how much business value is gained from incorporating a new event immediately.

I would define freshness requirements before selecting the architecture.

For example:

Use case Possible freshness requirement Design consideration Historical customer analysis Hours or days Batch processing may be sufficientProduct-interest analyticsMinutes Managed streaming-to-lakehouse delivery may fitRecent customer-behavior features Seconds to minutesDepends on model and serving requirements Checkout inventory validation Potentially milliseconds to secondsOften requires an operational serving path Historical ML training dataHours or days Optimize for completeness and cost

These are illustrative engineering targets, not universal service-level objectives.

The right choice depends on the business process.

4.2 The Service Does Not Replace Your Transformation Layer

This is one of the most important limitations.

AWS documents that streaming-table delivery applies no transformations: records are delivered as-is from the source stream. It also supports append-only delivery, does not support schema evolution in an existing delivery, and does not backfill records written before the delivery becomes active. Each delivery creates a new destination table rather than writing into an existing table. AWS Documentation

Those details matter.

Suppose I need to:

• Deduplicate product-view events.

• Enrich them with product attributes.

• Calculate a rolling two-hour click-to-view ratio.

• Join events to inventory data.

• Apply business rules before exposing data to analysts.

A direct streaming delivery does not perform those transformations for me.

I may still need Apache Flink, Spark Structured Streaming, Dataflow, or another processing engine.

The architecture could therefore have two separate responsibilities:

Delivery layer: Reliably moves records into a queryable lakehouse table.

Transformation layer: Applies business logic, enriches records, computes aggregates, and produces curated datasets.

That separation is useful.

It prevents a managed delivery service from being mistaken for a complete data-processing platform.

4.3 Schema Governance Is Still Essential

AWS requires a schema registered in AWS Glue Schema Registry for streaming-table delivery, and a dead-letter queue is also required. Records that fail validation are sent to the dead-letter queue. AWS Documentation

This is not a minor configuration detail.

Consider a product-view event that changes from:

product_id, customer_id, event_timestamp

to a new schema with different field names or incompatible types.

If producers change the contract without coordinating with downstream consumers, the pipeline can fail to process records as expected.

The service cannot make poor event contracts disappear.

I would establish explicit rules for:

• Schema compatibility.

• Event versioning.

• Required fields.

• Timestamp conventions.

• Invalid-record handling.

• Dead-letter queue monitoring.

• Replay and recovery.

I would also test schema changes before deploying them.

A dead-letter queue is useful only if someone monitors it and the team has a process for investigating and recovering failed records.

4.4 Compaction Has a Cost

Compaction improves query performance by combining smaller files into fewer, larger files.

AWS documents that S3 Tables can automatically compact files, with a default target file size of 512 MB and configurable targets between 64 MB and 512 MB. It also explains that maintenance operations can incur additional costs. AWS Documentation

That means compaction should not be treated as free background magic.

The goal is not simply to produce the largest possible files.

The goal is to organize data in a way that matches the workload.

A table used for broad historical scans may benefit from a different layout than one frequently filtered by date, store, and product category.

I would measure:

• Average file size.

• Number of files created.

• Query scan volume.

• Query latency.

• Compaction cost.

• Storage growth.

• Snapshot retention.

I would then adjust table layout based on observed query patterns.

The same principle applies to partitioning.

Partitioning by a high-cardinality field can create inefficient layouts. Partitioning only by time may not be sufficient for every query. Sorting can help, but it also introduces trade-offs.

These decisions should follow the access pattern, not a generic best-practice checklist.

4.5 Build for the Failure You Will Actually Encounter

Managed services reduce the amount of infrastructure a team operates directly.

They do not eliminate the need for monitoring and recovery.

For Kinesis streaming tables, I would monitor:

• Source-stream ingestion health.

• Delivery status.

• Event-to-queryable-data latency.

• Dead-letter queue volume.

• Schema validation failures.

• Query latency and scanned bytes.

• Storage and maintenance costs.

I would also test what happens when a delivery fails, when an event violates the schema, or when a source stream needs to be recreated.

The recovery plan matters because AWS documents that delivery does not backfill records written before the delivery becomes active. AWS Documentation

If I need historical backfill, I must design that path separately.

This is one of the most common mistakes I would want to avoid: assuming that because the normal delivery path is managed, recovery and historical completeness are automatically covered.

5. My Take: The Best Architecture Is the One That Removes the Right Complexity

I like the direction these developments are taking.

For years, data engineers have spent significant time building the infrastructure required to move records between systems.

That work is necessary, but not all of it differentiates the business.

If a managed service can reliably handle delivery, write coordination, and compaction, I would seriously consider using it rather than maintaining a custom pipeline without a clear reason.

But I would not adopt it simply because it is new.

I would first compare it with the existing architecture.

If my current Flink pipeline performs complex transformations, manages state, joins multiple streams, and enforces business rules, replacing it with a direct delivery service would not solve the same problem.

If my existing pipeline only moves records into Iceberg and manages compaction, a managed alternative could be worth evaluating.

The distinction is important.

I want to eliminate operational complexity, not eliminate engineering control where the business still needs it.

I also think open table formats will become more important as companies use different engines for different workloads.

Apache Iceberg helps separate the table format from the processing engine. Google Cloud’s April 2026 announcement, for example, described read-and-write interoperability across BigQuery and other Iceberg-compatible engines. Google Cloud

That can give teams more flexibility.

But openness alone does not guarantee portability.

Catalog behavior, governance, maintenance, schema evolution, and service-specific limitations still matter.

An architecture can use an open table format while remaining dependent on particular cloud services.

That is not necessarily bad.

It simply needs to be a deliberate decision.

The insight I would take into my next design review

I would stop asking only how quickly the pipeline can ingest data.

I would ask how much it costs to make each newly ingested record useful.

That includes ingestion, transformation, compaction, metadata operations, storage, query execution, and maintenance.

A pipeline that ingests data in seconds but makes every downstream query expensive may be a worse design than one that delivers data in a few minutes and keeps query costs predictable.

The right balance depends on the use case.

But measuring the full path gives us a much better basis for choosing.

Key Lessons for Data Engineers

1. Treat freshness as a business requirement.
Define the acceptable delay before choosing batch, micro-batch, or streaming.

2. Do not confuse delivery with transformation.
A service that writes events to Iceberg does not automatically deduplicate, enrich, or aggregate them.

3. Treat compaction as part of the architecture.
File layout affects query performance, storage behavior, and operating cost.

4. Keep schema contracts explicit.
Managed delivery cannot compensate for incompatible producers and consumers.

5. Design historical recovery separately.
Understand backfill behavior, failure recovery, and how to reconstruct missing data.

6. Compare the full cost, not just the ingestion cost.
Include delivery, compute, compaction, storage, and downstream queries.

7. Use managed services where they remove generic plumbing.
Keep custom processing where the business logic genuinely requires it.

Final Takeaway

I believe the next phase of data engineering will be defined less by how quickly we can move events and more by how efficiently we can turn those events into usable information.

Amazon’s streaming tables, Google’s managed Iceberg capabilities, and Apache Iceberg’s improvements to metadata and catalog operations are different developments, but they address related problems.

They make parts of the data lifecycle easier to operate.

For retail, that matters because customer behavior, inventory, orders, and product information can all benefit from fresher analytical data but not every use case needs the same latency or architecture.

My conclusion is straightforward:

Don’t optimize for the fastest pipeline. Optimize for the lowest sustainable cost of delivering trustworthy, usable data at the freshness the business actually needs.

That is a much more useful engineering goal than simply making everything real time.

And it is the goal I would use to evaluate the next data platform I build.

References

1. AWS Big Data Blog — August 31, 2026

Deliver real-time data to streaming tables for Apache Iceberg with Amazon Kinesis Data Streams

Explains the managed delivery architecture, inline compaction, write coordination, pricing model, and intended use cases.

Read the AWS engineering article

2. AWS Documentation — Kinesis Data Streams

How streaming table delivery works

Technical details covering buffering, schema validation, record conversion, inline compaction, destination tables, data freshness, dead-letter queues, and encryption.

Read the streaming-table documentation

3. AWS Documentation — Kinesis Data Streams

Delivery quotas and limits

Documents important constraints, including append-only delivery, lack of schema evolution and historical backfill, and restrictions on cross-account and cross-region delivery.

Read the delivery constraints

4. AWS Documentation — Amazon S3 Tables

Maintenance for tables

Explains compaction strategies, target file sizes, snapshot management, and the associated maintenance trade-offs.

Read the table-maintenance documentation

5. Google Cloud Blog — April 8, 2026

Openness without compromises for your Apache Iceberg lakehouse

Describes interoperability between BigQuery and Iceberg-compatible engines, managed maintenance, streaming support, and table-management improvements.

Read the Google Cloud engineering article

6. Apache Iceberg — May 19, 2026

Apache Iceberg 1.11.0 Release

Covers improvements to REST Catalog remote scan planning, incremental scans, metadata loading, idempotency, and performance.

Read the Apache Iceberg release notes

Editorial note: AWS’s cost-reduction figures are vendor-reported estimates. Google’s performance figures are based on its own benchmarks. Neither should be treated as an independently verified guarantee for a production workload.

The recurring research task has been paused after this scheduled run, so another weekly article will not be generated automatically unless the schedule is re-enabled.