What Amazon, Target, Lowe’s and cloud platforms are teaching us about batch pipelines, streaming, vector search and AI agents

There is a pattern I keep noticing in retail technology.

A customer searches for a drill. Then looks at a few batteries. Adds a tool to the cart. Removes it. Changes the delivery location. Opens the app again ten minutes later.

From the customer’s point of view, these are tiny interactions.

From a data engineering point of view, they are state changes.

The system has to understand what just happened, decide whether that signal matters, update the customer’s context, and potentially change what appears on the next screen.

This is where retail personalization is heading.

For years, the standard architecture was fairly predictable: collect events, build a data lake or warehouse, train a model, generate recommendations in batch, and serve the results.

That architecture is not disappearing.

But it is being extended.

Amazon is combining a batch recommendation foundation with real-time signals. Target has rebuilt parts of retail discovery around a unified system combining transactional, graph, vector and search capabilities. Lowe’s is using AI across shopping, associate workflows and store operations. And cloud providers are increasingly describing enterprise data platforms as infrastructure for AI agents rather than just systems for reporting and analytics. (Amazon Web Services, Inc.)

The interesting part is not the AI model.

The interesting part is the data architecture underneath it.

Why This Matters Right Now

Retail has always been sensitive to timing.

A recommendation generated yesterday is still useful for some products.

A recommendation generated from what a customer searched for ten seconds ago can be much more useful for others.

That creates a fundamental engineering question:

How fresh does a signal actually need to be before its business value changes?

That question sounds simple, but it affects almost every part of a data platform.

Do I need Kafka or Pub/Sub?

Do I need streaming feature computation?

Can I keep batch pipelines?

Should I use a vector database?

Where should embeddings live?

Do I need an online feature store?

Can my recommendation service survive if the real-time path fails?

Should an AI agent query several systems, or should I build a unified context layer?

These aren’t abstract architecture questions anymore.

Retailers are deploying systems where search, recommendation, inventory, customer context and AI assistance increasingly overlap.

Target, for example, said in June 2026 that AI-driven traffic to its site had increased 2,000% year over year in the first quarter, compared with nearly 400% growth across retail sites overall. The company also said customers could interact with Target shopping experiences through Google Search/AI Mode and Gemini, Microsoft Copilot and ChatGPT. (Target Corporation)

That changes the input side of the retail system.

The customer is no longer interacting only with the retailer’s website.

The interaction may begin inside a conversational AI system.

The retailer still needs to answer questions such as:

• Which products are available?

• Which products fit the customer’s intent?

• Which products are compatible?

• What is the customer’s context?

• What price and fulfillment options apply?

• What information can safely be exposed?

The data platform suddenly becomes part of the customer experience itself.

1. What’s Actually Happening

Amazon: Don’t throw away batch

One of the most useful examples comes from Amazon’s engineering team.

In a July 27, 2026 AWS engineering article, the team described a recommendation architecture serving millions of customers across global marketplaces at tens of thousands of requests per second.

The interesting choice was not to replace batch processing with streaming.

They built a batch-first system.

The architecture uses a centralized data lake, Airflow orchestration, Glue processing, SageMaker for ML workloads and DynamoDB for serving. Later, when the team needed fresher signals, it added MemoryDB with vector search and real-time model inference. (Amazon Web Services, Inc.)

The distinction is important.

Slow-moving signals such as purchase history and catalog relationships can still be calculated in batch.

Fast-moving signals such as recent searches, cart additions and in-session behavior can be handled on the real-time path.

Amazon’s team describes this as combining batch and real-time rather than choosing one over the other. (Amazon Web Services, Inc.)

That matches how I would approach this problem in production.

A streaming architecture everywhere sounds attractive until the infrastructure bill arrives.

If a customer’s annual purchase history changes once a year, there is very little business value in maintaining a streaming pipeline for it.

But if a customer searches for “black bathroom vanity” and then clicks five related products, waiting until tomorrow to incorporate that behavior is obviously too late.

The right architecture depends on signal velocity.

Target: Reduce the number of places where context can break

Target is taking a different architectural direction.

In an August 4, 2026 engineering case study published by Google Cloud, Target described an older discovery architecture based on separate Elasticsearch clusters, inverted indexes and NoSQL systems.

That architecture worked, but maintaining consistency between transactional data, vector information and search indexes created operational overhead and disconnected context. (Google Cloud)

Target chose Spanner Graph as a unified foundation for transactional data, graph relationships, vectors and search.

The goal was not simply database consolidation.

The more interesting goal was context consolidation.

Target described using the platform for product entities, relationships, semantic information and embeddings, with the AI layer consuming structured context from the same foundation. (Google Cloud)

There is a subtle lesson here.

Many modern AI systems have a data synchronization problem disguised as an AI problem.

You can build an excellent model.

But if the product catalog says one thing, the search index says another, the vector database is stale and the transactional system has a different inventory state, the model is not going to rescue the architecture.

The model will simply generate an intelligent answer from inconsistent information.

That is still a bad answer.

2. Retailers Are Moving From Recommendations to Context

I think this is the bigger change.

Traditional personalization asks:

“What product should I show this customer?”

AI-driven retail experiences ask a much broader question:

“What does this customer need right now, and what should the system do next?”

Those questions are related, but they lead to different architectures.

Consider Lowe’s.

Lowe’s reported in its May 20, 2026 earnings call that its Mylow AI shopping assistant was handling more than one million customer inquiries per month, and that customers using Mylow had a reported online conversion rate three times that of customers who did not use it. Those are company-reported figures, not an independent measurement. (Lowe’s Corporate)

The same call described Mylow Companion for store associates, with more than five million questions asked through the associate tool since launch. Lowe’s also said its combined associate and customer AI interactions were around two million per month. (Lowe’s Corporate)

This is important because retail AI is no longer just a recommendation widget.

It is becoming an interface over enterprise data.

A customer can ask a question.

An associate can ask another question.

An AI system may retrieve product information.

Another system might verify availability.

Another may provide pricing.

Another may determine eligibility.

The system starts looking less like a recommendation engine and more like a data-powered decision system.

That creates a new requirement:

The data must not only be correct. It must be available at the speed of the interaction.

3. The Business Problem Behind the Architecture

It is easy for engineers to start this discussion with technology.

Kafka.

Flink.

Vector databases.

Feature stores.

Graph databases.

Embeddings.

Streaming SQL.

But the business question comes first.

Imagine a retailer has three types of customer signals.

Slow signals

Purchase history.

Long-term category preference.

Customer segment.

Historical affinity.

These might change over days or weeks.

Batch is usually sufficient.

Medium-speed signals

Recent browsing history.

Products viewed in the last few hours.

Current category interest.

These might justify hourly or near-real-time updates.

Fast signals

Current search query.

Current cart.

Recent clicks.

Current store location.

Current inventory availability.

These can change in seconds.

Trying to force all three into a single processing model is where systems become unnecessarily expensive.

Amazon’s architecture makes this very clear: batch-generated recommendations remain the foundation, while real-time inference and vector search are added only where freshness creates additional value. (Amazon Web Services, Inc.)

That is a better engineering principle than simply saying:

“Everything needs to be real time.”

It doesn’t.

Freshness is a business requirement

This is one of the most useful ways I think about personalization.

Don’t start with:

“How do we build real-time data?”

Start with:

“Which customer signals lose value when they become stale?”

That gives you a much better architecture.

4. What This Means for Data Engineers

This shift creates several engineering implications that are easy to underestimate.

1. Event schemas matter more

Once user behavior becomes an input to real-time decisions, an event is no longer just an analytics record.

It becomes an operational input.

A product_view event may influence:

• recommendations

• ranking

• search

• customer context

• marketing decisions

• AI agent responses

That means event contracts need stronger discipline.

Schema evolution, idempotency, event ordering, timestamps, late-arriving events and deduplication all become product concerns.

A bad event isn’t just a bad dashboard.

It can produce a bad customer experience.

2. Streaming does not automatically mean better

This is where engineering teams can get carried away.

Streaming introduces additional infrastructure and failure modes.

Now I have to reason about:

• late events

• duplicate events

• out-of-order events

• state growth

• checkpointing

• replay

• backpressure

• schema compatibility

• operational monitoring

• disaster recovery

For some signals, this complexity is worth paying for.

For others, it isn’t.

Amazon’s engineering team explicitly recommends starting with batch and adding real-time processing only when there is a clear need for freshness. It also recommends routing signals according to how quickly they change. (Amazon Web Services, Inc.)

That is a much more practical principle than “stream everything.”

3. Serving architecture becomes part of data engineering

Traditional data engineering often ends at the warehouse.

Personalization systems don’t.

The last mile matters.

A model can have excellent offline metrics and still produce a poor user experience if serving latency is too high.

Amazon’s architecture uses precomputed recommendations in DynamoDB and then applies real-time filtering and re-ranking at request time. The real-time path can additionally generate embeddings and perform vector search for fresh signals. (Amazon Web Services, Inc.)

This is a hybrid design.

Compute expensive work before the request.

Reserve the request path for information that genuinely needs to be fresh.

That pattern can save both latency and infrastructure cost.

4. Failure handling has to be designed into the experience

This is one of my favorite details from the AWS architecture.

The recommendation system doesn’t assume that every component will always work.

If the batch pipeline fails, the previous recommendations remain available.

If the real-time path fails, batch-generated recommendations can still be served.

AWS also discusses fallback options for real-time embedding failures, such as cached results, previously generated recommendations or default content. (Amazon Web Services, Inc.)

That is how I think production AI systems should be designed.

Don’t ask only:

“What happens when everything works?”

Ask:

“What does the customer see when the smartest component is unavailable?”

That question exposes whether the system is actually resilient.

5. The Data Platform Is Becoming an Application Dependency

Google Cloud’s 2026 discussion around “agentic data” makes an argument that I find technically important.

The bottleneck for production AI systems may not be the model.

It may be access to reliable enterprise context.

Google Cloud wrote in July 2026 that organizations need to move beyond storing data toward activating trusted context, and noted that AI agents can create nonlinear infrastructure load because one prompt can trigger multiple queries and actions across systems. (Google Cloud)

Whether you agree with Google’s product framing or not, the underlying engineering problem is real.

An agent can make many more data requests than a human user.

A human might inspect three pages.

An agent could potentially inspect dozens of records, call several services, perform searches and execute actions.

That changes the economics of data access.

A data warehouse designed primarily for human analytics is not automatically a good runtime dependency for an autonomous system.

That means modern data architecture increasingly needs two characteristics at once:

analytical depth and operational responsiveness.

And that brings us back to the architecture question.

Maybe the answer is not one giant platform.

Maybe it is a carefully designed combination of:

• durable analytical storage

• event streams

• curated feature or context layers

• low-latency serving systems

• vector or graph retrieval

• strong governance

• intelligent caching

• observability

The common denominator is not the technology.

It is consistency of business context.

My Take

The most interesting thing happening in retail data engineering right now is not that companies are adopting AI.

Retailers have been using machine learning for years.

The bigger change is that AI is increasing the value of data freshness.

That is a subtle difference.

A traditional recommendation engine can operate mostly from historical behavior.

An AI shopping assistant needs to understand the current situation.

What product is the customer asking about?

What is available?

What alternatives exist?

What has the customer already considered?

What is compatible?

What is the current context?

What action is possible?

That means the data platform is slowly moving closer to the application.

As someone working around retail personalization and data platforms, this is the part I find most interesting. The hardest engineering decisions are often not about the model at all. They are about deciding which signals deserve real-time treatment, how long they remain valid, where they should live, and what the system should do when the fresh path is unavailable.

I also don’t think every retailer should copy Target’s unified database architecture or Amazon’s exact stack.

Architecture is a response to constraints.

Amazon’s recommendation problem is different from Target’s product-discovery problem.

A home-improvement retailer has different semantics from a fashion retailer.

A marketplace has different requirements from a first-party inventory model.

The common lesson is more general:

Design around the velocity and business value of information, not around the popularity of a technology.

That is where I think many data platforms still have room to mature.

Key Lessons for Data Engineers

1. Treat freshness as a contract

Don’t define a pipeline simply as batch or streaming.

Define how fresh each signal actually needs to be.

A useful architecture might look like:

Weeks → batch

Hours → micro-batch

Minutes → streaming

Seconds → online state

The boundaries should come from business value.

2. Don’t rebuild the entire platform for real time

Keep the stable batch foundation.

Add real-time capabilities where they materially improve the customer experience.

Amazon’s architecture is a good example of incremental evolution rather than a complete rewrite. (Amazon Web Services, Inc.)

3. The serving layer deserves as much attention as the model

A brilliant recommendation that arrives too late is not a brilliant recommendation.

Latency, caching, re-ranking, fallback behavior and availability matter.

4. Data consistency becomes more important as AI becomes more autonomous

When several systems disagree about product state, inventory, price or customer context, AI can amplify the inconsistency.

More intelligence does not compensate for contradictory data.

5. Build for failure

A production recommendation system should have a useful answer even when one dependency is down.

Fallbacks are not an afterthought.

They are part of the product design.

One Insight I Would Keep

The next generation of personalization systems may not be defined by their recommendation model at all.

They may be defined by their freshness architecture.

Two retailers could use the same foundation model, the same embedding model and even similar ranking algorithms.

One could still deliver a much better experience because its data platform understands that a search made five seconds ago is fundamentally different from a purchase made six months ago.

That is the part of AI engineering I think deserves much more attention.

**The competitive advantage may not be who has the smartest model.

It may be who can get the right business context to the model at the right moment, reliably and economically.**

References

Amazon Web Services — July 27, 2026
Building a scalable personalized recommendation system on AWS: From batch to real-time
Amazon’s engineering team describes a batch-first recommendation architecture, its real-time extension, reusable Airflow operators, serving design, failure handling and the trade-off between freshness and cost. (Amazon Web Services, Inc.)
Read the AWS engineering article

Amazon Web Services — May 27, 2026
Real-time personalized recommendations with Amazon SageMaker and Valkey
Technical deep dive into real-time embedding generation, vector search and serving considerations. (Amazon Web Services, Inc.)
Read the AWS Database Blog article

Google Cloud / Target Engineering — August 4, 2026
How Target is enhancing retail discovery and cutting database maintenance by 50% with Spanner Graph
Target engineers explain the move from fragmented search/vector/transactional systems toward a unified data model for retail discovery and AI-assisted experiences. (Google Cloud)
Read the Target engineering case study

Target — June 18, 2026
How Target is Shaping Shopping Experiences in Conversational AI
Target reported strong growth in AI-driven traffic and described shopping experiences across multiple conversational AI platforms. (Target Corporation)
Read Target’s announcement

Lowe’s — May 20, 2026
Q1 2026 Earnings Call
Lowe’s discussed Mylow, Mylow Companion, AI adoption, customer inquiries and its broader approach to AI in shopping and operations. (Lowe’s Corporate)
Read the Lowe’s earnings transcript

Lowe’s — February 25, 2026
Q4 2025 Earnings Call
Lowe’s discussed real-time data for AI-enabled shelf replenishment and its use of AI in store operations.
Read the Q4 earnings transcript

Google Cloud — July 23, 2026
Your AI agents are ready. Is your data?
Google Cloud’s discussion of agentic workloads, enterprise context and the infrastructure implications of AI agents. (Google Cloud)
Read the Google Cloud article