1. Introduction

A few weeks ago, someone asked me a question:

"How can I use AI to improve my inventory if I have similar or duplicate products, but their names aren't written exactly the same way?"

To explore how to solve this problem, we'll use video games as an example.

⚠️ First spoiler: that's the type of data we'll be working with in this article.

• Call of Duty: Black Ops and Call of Duty: Black Ops 2 have almost identical titles, but they are different games. They are not duplicates.
• Black Ops 2 and Black Ops II are exactly the same game, even though a computer sees them as two different text strings.
• FIFA 14 for PlayStation 3 and FIFA 14 for Xbox 360 have the same title, but the platform is different, so they may represent distinct commercial products.

So, what does it really mean for two products to be duplicates?

• Two records may describe the same product even if their names are different.
• Nearly identical names may refer to different products.
• The same product may be sold in different presentations or formats, which we may need to keep separate depending on the business rules.

Each use case may require different business rules. Some differences may seem obvious to a person, but when we're dealing with thousands of records, we can't rely on someone detecting them manually.

⚠️ Second spoiler: AI isn't going to solve everything.

We could send all the information about each product to an AI model and ask it to determine whether it's a duplicate. But processing an entire catalog this way can be expensive, and we can't guarantee that the responses will always be consistent.

In other projects, I have worked with text normalization, regular expressions, and similarity techniques. These remain useful tools, and in many cases, they are sufficient. For this MVP, I wanted to explore a different approach: using rules to handle straightforward cases, while reserving AI for more ambiguous scenarios that cannot be resolved by a rule.

That's why, before we start building the solution, there are three questions I want to answer:

1. How can we distinguish truly duplicate products from those that are simply similar?
2. When does AI add value, and when is it better to use traditional rules?
3. Can we build an MVP on AWS using a serverless architecture while keeping costs under control?

To answer these questions, we'll build a solution that combines AI with traditional rules. We'll walk through the process, from data analysis and normalization to architecture design, comparison logic, and evaluation of the results.

Because detecting duplicates isn't simply about finding similar names. First, we need to understand what we're comparing and what it actually means for two products to be the same.

2. The Business Problem

Poor product data quality is not an isolated problem. Various studies highlight the difficulties companies face in keeping information consistent, up to date, and accessible across their systems.
For example, in 2026, GS1 UK published a report on product information management in the construction sector, based on research involving more than 300 professionals and 15 prominent figures from the industry.
Some of its findings include:

• £3.8 billion estimated potential annual losses resulting from inefficient product data management in the UK construction industry.
• 60% of respondents say that inefficiencies in managing this information are hindering progress. The problem is not limited to construction. The 2024 Product Information Management Report by RIWI, based on a survey of more than 300 professionals, found that 37% identified bringing products to market as one of their main pain points.

What consequences can this have for a product catalog?

When products are not identified consistently, several problems can arise:

• Stock: units of the same product may be spread across multiple records, making it difficult to determine actual inventory levels.
• Pricing: comparing offers for the same product across suppliers or sales channels can become more difficult.
• Search: users may encounter multiple listings for the same product or fail to find it when searching with a different variation of its name.
• Reviews: ratings and reviews may be distributed across different listings of the same product.

That is why the objective of this project is more specific: to explore how to detect potential duplicates in a catalog, distinguish them from products that should remain separate, and evaluate whether combining AI with traditional rules can address part of the problem without driving up costs.

3. How Do We Manage Product Information?

When a company sells products through multiple channels, keeping information up to date and consistent can become a challenge. Product Information Management (PIM) systems are designed to manage this process.

3.1. 🗂️ What Is a PIM?

A PIM allows companies to centralize, enrich, organize, and distribute product information from a single location. It can manage product names, descriptions, technical specifications, images, and other attributes needed to sell a product. Adobe explains here how these systems work.

Source: Author's own diagram.

Consider, for example, a company that sells video games through its own website and marketplaces such as Amazon or eBay. Product information may come from different suppliers and in different formats, so it needs to be organized before being published across each channel.
Although this article introduces the concept of PIM, depending on the organization, product information may be managed in an Excel spreadsheet, a database table, enterprise systems such as a CRM or SAP MM, or specialized solutions like a PIM.
The idea is to explore a solution that can be accessed through an API, for example, to help perform data quality checks and detect potential duplicates, regardless of where the product information is managed.

4. MVP Design Principles

Before starting to build the solution, I had to make several decisions: which data to use, which part of the catalog to process, and how to combine AI with traditional rules without driving up costs.
These are the principles that defined the scope and design of the MVP:

1. A public, realistic dataset.

I used the product metadata from Amazon Reviews 2023, in the Video Games category. I chose this category because it covers many different scenarios: the same game can be sold for different platforms (Nintendo, PlayStation, Xbox or PC), in different editions (standard, deluxe or collector’s), in physical or digital format, or as a refurbished product.

2. A limited scope based on an external criterion.

For the MVP, I selected a subset of products based on the 20 best-selling video game franchises according to Wikipedia. To identify the records, I used regular expressions that look for the names of these franchises and their variants in the dataset titles. This keeps the data volume manageable and makes the selection criterion reproducible.

3. Cost as a design variable.

I chose a serverless AWS architecture, with on-demand processing and low-cost AI models. Every technical decision is also evaluated for its impact on cost, from data preparation to model calls.

4. Rules first, AI where it adds value.

To avoid unnecessary costs, I first analyze the data and apply deterministic rules to the cases that can be solved simply. The language model is reserved for cases that require interpreting text, recognizing abbreviations or evaluating ambiguous matches.

5. Explicit business criteria.

Deciding whether two products are the same, a variant or different products is not just a technical question: it depends on the business. In this MVP, those criteria are defined in the prompts and in the pipeline rules, with examples that guide the model. They are a proposal for this case study: in a production environment, they must be reviewed and validated with the business teams of each organization.

6. A pipeline to evaluate, not a finished system.

This MVP is a practical approach to building a data pipeline for detecting duplicate products. The goal is not a perfect system, but to check whether the approach is viable and to measure its technical behavior: how long it takes, how much it costs, which share of the decisions is resolved with rules and which needs a model, and how extraction quality evolves across prompt versions. I also review specific cases to understand where it succeeds and where it fails.

5. Architecture

Before going into the details of each step, it is worth looking at the complete solution. The goal of the architecture is to transform a disorganized product catalog into one where each listing is associated with the actual product it represents, making that information available for real-time queries.
To achieve this, the solution combines batch processing, which processes the entire catalog at once, with a real-time component that handles new products or search queries as they arrive.

Source: Architecture diagram created by the author.

The image above shows the solution architecture, organized into three parts:

• Input: starts with the Amazon Reviews 2023 dataset. We apply the selection criteria defined for the MVP and the data quality rules identified during the EDA, discarding records that do not meet the requirements or fall outside the project's scope.
• Pipeline: processes the products from start to finish. It extracts their attributes using a language model, converts them into vectors, searches for similar products, and determines whether they are duplicates, variants, or different products. Finally, it assigns each product a product identifier and a family identifier, storing the results in DynamoDB, a NoSQL database.
• Inference: allows us to query the catalog or evaluate new products. It receives a natural language search query, such as FIFA 14 PS3 digital code, or the details of a new product, and determines whether it matches an existing product, is a variant, or is a new product. The response includes the corresponding identifiers. One principle guided all our decisions: a fully serverless architecture that allows on-demand processing without managing dedicated servers. For a catalog that is updated periodically, as in this case, this approach allows us to adjust resource consumption to actual needs, with costs primarily associated with data storage, operations performed, and resources used during each execution.

The following AWS services are grouped according to the diagram's legend:

🟩 Storage

• Amazon S3: stores the data from each stage in its own folder, allowing us to trace each product and reprocess data from any step.
• Amazon S3 Vectors: stores product vectors and performs similarity searches without requiring us to manage a vector database.
• Amazon DynamoDB: stores each product with its product and family identifiers, enabling millisecond key-based queries for the real-time component.

🟨 Compute / AI

• Amazon Bedrock, with three models, each selected for a specific task:
• Nova Micro: extracts the attributes of each product. It is the most cost-effective model and is used for all attribute extraction calls.
• Titan Text Embeddings V2: converts each product into a vector.
• Nova Lite: handles ambiguous cases. It has stronger reasoning capabilities than Nova Micro and is used only for a small proportion of product pairs.

🟪 Pipeline

• AWS Lambda: executes each processing step as an independent function.
• AWS Step Functions: orchestrates the pipeline steps, retries temporary failures, and displays the status of each execution.

🟥 Security

• AWS IAM: each Lambda function has its own role with the minimum permissions required (for example, reading only from the input folder and writing only to the output folder).
• AWS KMS: encrypts data at rest.

🟦 Logs

• Amazon CloudWatch: centralizes logs from all Lambda functions and Step Functions. It is the first place to look when a step fails, and it also allows us to examine metrics such as execution duration or memory usage for each run.

6. Use Case Development

6.1. 🔍 What Is EDA?

Exploratory Data Analysis (EDA) is a process that helps us better understand a dataset before applying models or making decisions. Through different exploration and visualization techniques, we can identify patterns, discover relationships between variables, detect anomalies, and assess data quality.
Although we often associate EDA with the early stages of Machine Learning projects, the concept is not new. Statistician John W. Tukey formalized it in his book Exploratory Data Analysis, published in 1977.

🎯 Why is it important?

Because it helps us understand the data before working with it, identify outliers, detect inconsistencies, and question our initial assumptions. Instead of assuming that the data has the expected structure or quality, we explore it to discover what it is actually telling us.

🤖 And what about generative AI projects?

It can be tempting to send raw data to a model with a prompt and hope for the best. But every unnecessary field adds tokens that we pay for with each call, and every inconsistent data point is a potential error that the model can propagate. EDA helps us decide which data to send, what to correct beforehand using simple rules, and which part of the catalog to process.
For this use case, I performed ten checks:

Source: Author's own infographic.

The findings that had the greatest impact on the design were:

• main_category does not describe the product: it refers to the store department. The platform is taken from categories.
• There are hidden nulls: thousands of products have an empty categories list.
• Not everything in Games is actually a game: there are controllers, statues, and bundles.
• Accessory titles are full of keywords: so comparing titles directly does not work.
• Some fields seem useful, but are not: Pricing is a legal notice, and Date First Available is not the release date.

These findings lead to an initial conclusion: before looking for duplicates, we need to understand what each field represents and distinguish the products that can actually be compared.

📓 Would you like to explore the full analysis? The eda_video_games.ipynb notebook, including the code and charts, is available in the project repository. There, you will also find the files needed to continue implementing the architecture.

6.2. ☁️ Infrastructure Deployment

With the data analyzed, the next step is to create the resources required by the pipeline.
Ideally, we would define the entire infrastructure as code using Terraform, AWS SAM, or CDK and deploy it with a single command. To keep this MVP simple, I created the resources through the AWS Console and included the scripts and AWS CLI commands in the repository to reproduce the deployment.

🧰 Prerequisites

Before getting started, you will need:

• An AWS account with access to Amazon Bedrock in us-east-1 and the Nova Micro, Nova Lite, and Titan Text Embeddings V2 models enabled.
• AWS CLI v2, configured with the permissions required to create the resources.
• The meta_Video_Games.jsonl file from the Amazon Reviews 2023 dataset.
• Clone the repository and navigate to the project directory:

git clone https://github.com/RominaElenaMendezEscobar/aws-games-products.git cd aws-games-products 

📁 Repository Structure

The resources are organized by service under aws/. Each Lambda function has its own directory containing the code, permissions policy (policy-role.json), and required configuration.

aws/ ├── S3/ → data bucket ├── S3Vectors/ → vector bucket and index ├── DynamoDB/ → products table ├── lambda/ → 8 functions + deployment script └── step-function/ → pipeline definition and its role 

The deployment follows this order: storage → Lambda functions → Step Functions, as each stage depends on resources created earlier.

6.2.1. 🟩 Storage

6.2.1.1 Amazon S3 · game-catalog-dedup

This bucket stores the intermediate data generated during the different pipeline stages.

bash aws/S3/create_s3_bucket.sh 

6.2.1.2 Amazon S3 Vectors · game-catalog-vectors / index games-v2

This is where the product embeddings are stored. First, we create the vector bucket and then the index, which uses 1,024-dimensional vectors compatible with the Titan Text Embeddings V2 configuration and cosine distance.
The title and base_title fields are configured as non-filterable metadata: they are preserved in the results but cannot be used as filtering criteria in the index.

aws s3vectors create-vector-bucket \ --vector-bucket-name game-catalog-vectors aws s3vectors create-index \ --vector-bucket-name game-catalog-vectors \ --index-name games-v2 \ --data-type float32 \ --dimension 1024 \ --distance-metric cosine \ --metadata-configuration '{"nonFilterableMetadataKeys": ["title", "base_title"]}' 

6.2.1.3 Amazon DynamoDB · game-catalog-products

The table stores products identified by parent_asin and uses on-demand capacity. Two global secondary indexes allow us to query the results by identifier:

• canonical_id-index: all listings representing the same product.
• family_id-index: all variants of the same game.

bash aws/DynamoDB/create_dynamodb_table.sh 

6.2.2. 🟥 Permissions and Configuration

Each function has its own IAM role with least-privilege permissions. The policies are defined in the policy-role.json file in each directory and specify which services each Lambda can access. Each role also includes the AWS managed policy AWSLambdaBasicExecutionRole, so the function can send logs to CloudWatch. The deployment script (6.2.3) creates the roles and attaches both policies.
Before deploying, review the ARNs in each policy-role.json and replace <account_number> with your own AWS account ID. Also check the resource names and the Region.

6.2.3. 🟪 Lambda Functions

All eight functions use Python 3.12. To simplify deployment, I prepared a script that creates each function's IAM role from its policy-role.json file, packages the code, and creates the Lambda function or updates it if it already exists.

The script receives the function’s folder, its name, the memory and the timeout, and optionally a Lambda Layer ARN and the ephemeral storage size:

./aws/lambda/deploy_lambda.sh <folder> <function-name> <memory-MB> <timeout-s> [layer-arn] [ephemeral-MB] 

These are the commands to deploy the eight functions:

cd aws/lambda ./deploy_lambda.sh <folder> <function-name> <memory-MB> <timeout-s> [layer-arn] [ephemeral-MB] ./deploy_lambda.sh ETL01-game-catalog-etl game-catalog-etl 2048 900 <pandas-layer-arn> 2048 ./deploy_lambda.sh SP01-game-catalog-prepare-batch game-catalog-prepare-batch 1024 180 "" 1024 ./deploy_lambda.sh SP02-game-catalog-extract-ondemand game-catalog-extract-ondemand-access 512 900 ./deploy_lambda.sh SP03-game-catalog-parse-results game-catalog-parse-results 1024 300 "" 1024 ./deploy_lambda.sh SP04-game-catalog-embed game-catalog-embed 512 900 ./deploy_lambda.sh SP05-game-catalog-decide-pairs game-catalog-decide-pairs 1024 900 ./deploy_lambda.sh SP06-game-catalog-build-clusters game-catalog-build-clusters 1024 900 ./deploy_lambda.sh RT01-game-catalog-resolve-product game-catalog-resolve-product 512 60 

⚠️ The ETL Lambda function uses pandas, which is not included in the default Lambda runtime. It requires the AWS-managed AWSSDKPandas-Python312 layer (in the Lambda console: Layers → Add a layer → AWS layers) and 2 GB of ephemeral storage, as it writes the cleaned catalog to /tmp before uploading it to S3.
The configuration depends on the task: the ETL Lambda requires more memory to process the dataset, while functions that invoke models may need longer execution times. The real-time query Lambda has a shorter timeout.

6.2.4. 🟪 Step Functions · game-catalog-dedup-pipeline

Finally, we create the Standard state machine game-catalog-dedup-pipeline, which coordinates the six processing stages. Its role has permission to invoke only the six Lambda functions used by the pipeline (LambdaInvokeScopedAccessPolicy.json).

aws iam create-role --role-name game-catalog-dedup-pipeline-role \ --assume-role-policy-document file://aws/step-function/trust-policy.json aws iam put-role-policy --role-name game-catalog-dedup-pipeline-role \ --policy-name LambdaInvokeScopedAccessPolicy \ --policy-document file://aws/step-function/LambdaInvokeScopedAccessPolicy.json aws stepfunctions create-state-machine \ --name game-catalog-dedup-pipeline \ --definition file://aws/step-function/game-catalog-dedup-pipeline.json \ --role-arn <step-functions-role-arn> \ --type STANDARD \ --region us-east-1 

6.2.5. 📥 Upload the Raw Data

aws s3 cp data/meta_Video_Games.jsonl s3://game-catalog-dedup/raw/ 

6.3. 🛠️ Data Preparation

After exploring the dataset, the next step is to prepare it for the duplicate detection pipeline.
The EDA helped us understand its structure, identify data quality issues, and determine which attributes could be useful. Based on these findings, I divided the preparation process into two stages: first, cleaning and filtering the data; then, selecting the catalog we would use for the MVP.

6.3.1. 🧹 Data Cleaning and Filtering (ETL)

A Lambda function applies the rules defined during the EDA before any AI model is involved. The goal is to prevent incomplete records or unnecessary fields from affecting subsequent processing stages.
The main operations are:

• Discard products without titles or categories. Without this information, it becomes more difficult to identify the product and determine whether it falls within the experiment's scope.
• Remove columns without useful information. This reduces the amount of data we need to store and process, avoiding the need to carry empty fields into subsequent stages.
• Convert prices to numeric values. When available, this conversion makes prices easier to analyze and ensures a consistent data type.
• Filter the Games category. We keep only records belonging to this dataset category, reducing the initial dataset before selecting the franchises.

These operations do not identify duplicates yet. They prepare the data so that subsequent stages can compare products using more consistent information.

This stage is important because it defines the information the model will receive: if a piece of data arrives incomplete or incorrect, the error carries over into the extraction, the search, and the decision-making process.

6.3.2. 🎯 Why the 20 Best-Selling Video Game Franchises?

As mentioned in the MVP design principles, I used the 20 best-selling video game franchises as a reference to define the scope of the dataset.
This choice was intentional: I wanted to work with franchises that would allow us to explore different duplicate and variant scenarios, such as annual releases, titles published for different consoles, re-releases, and versions with similar names.
For example, FIFA has successive releases that should not be confused with one another, while the Mario and The Legend of Zelda franchises include games released for different Nintendo consoles. In these cases, similar titles do not necessarily identify the same product.
Additionally, limiting the catalog helps control the volume of data and the cost of model invocations during experimentation.

The dataset is progressively reduced at each stage:

1. 📦 Original dataset: 137,269 products
2. 🧹 Products with titles and categories: 124,624
3. 🎮 Products in the Games category: 46,530
4. 🏆 Products from the 20 best-selling franchises: 3,740

6.3.3. 📦 Storing Each Stage's Results Separately

Each stage stores its results in a separate Amazon S3 folder (raw/,clean/,mvp/…). In addition, the pipeline stages store their outputs in an execution-specific subfolder, such as attributes/top20-v8/.
This separation offers two practical advantages:
First, it allows us to reuse the results of a stage without repeating the entire processing workflow.
Second, it makes it easier to compare different pipeline executions and identify what has changed between versions.
This proved especially useful when refining the prompt: I could rerun subsequent stages using the already prepared data, without repeating the catalog cleaning process every time I modified the model instructions.

6.4. 🧠 Prompt Design and Evaluation

Once the data is prepared, the next step is to define how we will use AI. The goal is not to ask a model to find duplicates directly across the entire catalog, but rather to break the problem down into more specific tasks.
To achieve this, I designed two prompts: one to normalize the attributes of each product and another to determine whether two candidate products are duplicates, variants, or different products.
However, before selecting the models and writing the instructions, there were several requirements the solution needed to meet.

6.4.1. 📋 What Do We Expect from AI?

The first requirement was consistency. If two listings describe the same video game, we need the model to generate normalized attributes that allow us to recognize them as the same product, even when their original titles are different.
We also needed structured output that followed a predefined schema and could be processed programmatically without requiring manual interpretation.

Source: Author's own infographic.

The main requirements were:

• Normalize products: extract the base title, platform, edition, format, region, publisher, and product type.
• Maintain structured output: return valid JSON with the fields and values expected by the pipeline.
• Apply business rules: distinguish an edition from a re-release, a physical product from a digital one, and a video game from an accessory.
• Avoid unnecessary assumptions: do not invent attributes when the available information is insufficient to determine them.
• Control costs and execution time: choose the appropriate model for each task and avoid unnecessary model calls.

6.4.2. ✂️ Two Prompts for Two Different Tasks

During testing, I decided to separate the process into two tasks: normalizing each product's attributes and evaluating candidate pairs that require further analysis. I also found that I could use different models depending on the task: Nova Micro for attribute extraction and Nova Lite for resolving ambiguous cases.

6.4.2.1. How Do We Invoke a Model from AWS Lambda?

One of the advantages of Amazon Bedrock is that we can invoke a model through its Converse API without having to manage the underlying model infrastructure. In this case, the prompt is sent as a system message, while the product listing is sent as a user message.
The basic implementation in Python is straightforward:

response = bedrock.converse( modelId="us.amazon.nova-micro-v1:0", system=[{"text": SYSTEM_PROMPT}], messages=[{"role": "user", "content": [{"text": json.dumps(build_payload(product))}]}], inferenceConfig={"temperature": 0, "maxTokens": 300}, ) attributes = json.loads(response["output"]["message"]["content"][0]["text"]) 

The build_payload function sends only the fields I selected during the EDA: title, store, categories, four keys from details, and three product features. This way, we avoid sending unnecessary information to the model.
This snippet shows the basic model invocation. The complete implementation also needs to handle response validation and error management.

6.4.2.2. SYSTEM_PROMPT: Product Normalization (Nova Micro)

This prompt uses Amazon Nova Micro and runs on every product in the catalog. Its purpose is to transform an original product listing into a set of normalized attributes that we can use in subsequent stages.
For example, a product listing might contain:

{ "title": "GTA : Vice City - platinum [Playstation 2]", "categories": [ "Video Games", "Legacy Systems", "PlayStation Systems", "PlayStation 2", "Games" ] } 

The expected output would have a structure similar to this:

{ "product_type": "game", "base_title": "Grand Theft Auto: Vice City", "platform": "PlayStation 2", "edition": "Standard", "format": "physical", "region": "unknown", "publisher": "Rockstar Games", "confidence": "high" } 

In this example, the model normalizes the title and platform, while other attributes are determined according to the rules defined in the prompt. For instance, the Platinum label is treated as a budget re-release and normalized to Standard for this MVP. Other values may be inferred from the available information, so it is important to consider the limitations of these inferences.
The key is not simply to clean up the title, but to ensure that equivalent products are represented consistently without losing important distinctions, such as the platform.
📍 This prompt is used by the game-catalog-prepare-batch (batch) and game-catalog-resolve-product (on-demand) Lambda functions, both for the initial catalog processing and for resolving subsequent queries.

6.4.2.3. DECISION_PROMPT: Candidate Pair Comparison (Nova Lite)

This prompt uses Amazon Nova Lite and is reserved for ambiguous cases that require additional evaluation.
After normalization and candidate retrieval, not every match is straightforward. Two titles may be very similar while referring to different video games.

For example:

A: Mario & Sonic at the Olympic Games [Nintendo Wii] B: Mario and Sonic at the Olympic Winter Games - Nintendo Wii 

Although they share many words, these titles refer to different games. The presence of Winter is a meaningful distinction that we should not ignore.
The prompt must return a structured decision, for example:

{ "relation": "different", "reason": "The base titles of the games are different." } 

📍 This prompt is used by the game-catalog-decide-pairs (batch) and game-catalog-resolve-product (on-demand) Lambda functions. By reserving Nova Lite for ambiguous pairs, we avoid using a more expensive model for every record in the catalog.

6.4.3. 🏗️ How I Built the Extraction Prompt

The SYSTEM_PROMPT was organized into five parts:

Source: Author's own diagram.

• Role: a product data specialist who responds only in JSON.
• Purpose: explains that the output will be used to detect duplicates and that consistency matters more than preserving the original title.
• Rules: each rule comes from an EDA finding or an observed error: labels in brackets, product condition (“Renewed”) is not an edition, budget re-releases are considered the same product, sequel numbers are part of the title.
• Schema: the fields and their allowed values, including a predefined list of platforms.
• Examples: two complete cases using franchises outside the evaluated dataset, to assess whether the model generalizes rather than simply copying the answers.

6.4.4. 🧪 Testing Plan in Amazon Bedrock

To develop and test the prompts, I used the Amazon Bedrock Playground, which allows you to change the prompt, model, and parameters and see the response immediately, without writing a single line of code. Its comparison mode displays two models side by side using the same input, along with token consumption and latency.
This was especially useful during experimentation: I could modify a rule, rerun a problematic case, and check whether the result improved or a new error appeared.

Source: Screenshot captured by the author from the Amazon Bedrock console (AWS).

To evaluate the models' behavior and improve the prompts, I organized the tests across several dimensions:

🤖 Model Comparison

I tested Amazon Nova Micro and Nova Lite with equivalent inputs to evaluate their performance on extraction and comparison tasks.
The final decision was to use Nova Micro to normalize all products and reserve Nova Lite for ambiguous pairs. This way, the model with stronger reasoning capabilities is used only when it can provide additional value.

🌡️ Generation Parameters

I used a temperature of 0 to reduce response variability during testing. This is useful when comparing prompt versions, although it does not guarantee that every execution will produce identical results.

📦 One Product per Call or Multiple Products Together

Grouping ten products into a single call could have reduced the number of extraction calls from approximately 3,740 to 374. However, during testing, I observed that the model could mix information between records, incorrectly modify titles, or return results without properly associating them with their identifiers.
For this reason, I kept one call per product and used eight concurrent calls. This decision increases the number of requests but makes it easier to maintain the correspondence between each input and its result.

🌍 Queries in...