Automated Retail Space Analysis: Segmenting Carts & Shelves with AI on Matrice AI

Turning Retail Environments into Actionable Data Points Date: May 30, 2025

Understanding customer flow and store layout effectiveness is paramount in the dynamic retail environment. Traditional methods often provide limited insights. With modern AI-driven computer vision, specifically instance segmentation developed on the Matrice AI platform, retailers can now precisely identify and delineate Shopping-trolleys and Shelf areas in real-time. This foundational capability unlocks a wealth of data for optimizing store operations, enhancing customer experience, and paving the way for future automated solutions.

This article walks through an end-to-end retail asset segmentation pipeline—from dataset curation to model deployment using the Matrice AI platform—focusing on the “Shopping cart analysis dataset v1.0” and a compact yet powerful YOLOv8-Seg (small) model.

1. Why Segmentation of Shopping Trolleys & Shelves Matters in Retail

The precise, real-time instance segmentation of shopping trolleys and shelf areas is fundamental for data-driven retail optimization:

  • Enhanced Customer Flow Analysis: Tracking trolley movements provides deep insights into popular paths, dwell times in specific zones, and overall store navigation patterns, helping to identify bottlenecks or underutilized areas.

  • Optimized Store Layout & Merchandising: Accurate shelf segmentation aids in planogram compliance, assessing product visibility, and understanding the spatial relationship between shelf areas and customer traffic.

  • Improved Operational Efficiency: Knowing trolley locations can assist in managing availability, reducing abandonment, and optimizing staff allocation for assistance or retrieval.

  • Foundation for Advanced Checkout Systems: Segmenting the shopping trolley is a critical first step for future systems designed to identify individual items within it for automated or expedited checkout.

  • Enhanced Security & Loss Prevention: Monitoring trolley movements in relation to exits or specific zones can provide data for loss prevention strategies.

Automated segmentation of these key retail assets provides the granular data needed for smarter, more responsive store management.

2. Benefits of AI in Retail Space Analysis

AI-powered instance segmentation solutions, developed on Matrice AI, offer significant advantages for analyzing retail environments:

  • Accurate Trolley and Shelf Segmentation: Delivers precise pixel-level masks for Shopping-trolleys and Shelf areas, enabling reliable spatial data collection.

  • Real-time Environmental Understanding: Continuous monitoring provides live data on trolley distribution and shelf area status.

  • Actionable Analytics for Optimization: Transforms raw visual data into insights for improving store layouts, product placement, and operational workflows.

  • Scalability: AI models can be deployed across multiple store locations, with Matrice AI facilitating centralized management and updates.

  • Objective Data Collection: Eliminates manual biases and inconsistencies in assessing store dynamics.

3. Implementing Trolley & Shelf Segmentation with YOLOv8-Seg on Matrice AI

Our approach, developed using the Matrice AI platform, focuses on leveraging the capabilities of the YOLOv8-Seg model for accurate and efficient instance segmentation of Shopping-trolleys and Shelf areas.

Dataset Preparation: “Shopping cart analysis dataset v1.0”

A robust and well-curated dataset, managed and potentially annotated using Matrice AI’s data tools, is the cornerstone of this project.

  • Dataset Name: Shopping cart analysis dataset

  • Version: v1.0

  • Total Images: 1,641

  • Split:

    • Training images: 1,437

    • Validation images: 103

    • Test images: 101

  • Classes: The dataset is annotated for instance segmentation of two primary classes: Shelf and Shopping-trolley.

    Developer Tip: Segmenting Shelf areas provides crucial contextual information, helping the model better distinguish Shopping-trolleys and understand the overall store environment.

  • Annotation Type: YOLO masks (polygon annotations for instance segmentation).

  • Primary Metric Focus: Precision (for accurate delineation of Shopping-trolley and Shelf instances).

Model Architecture: YOLOv8 Instance Segmentation Small (yolov8s_seg)

The YOLOv8s-seg model was chosen for this task due to its optimal balance of accuracy, speed, and resource efficiency, with experimentation and selection streamlined on Matrice AI.

  • Model Family: YOLOv8_Instance_Segmentation

  • Model Name: YOLOv8 Instance Segmentation Small

  • Parameters: 11.8 Million – offering a compact footprint suitable for edge deployment.

  • Key Strengths:

    • Designed for high efficiency and accuracy in instance segmentation tasks.

    • Provides pixel-level masks for each detected Shopping-trolley and Shelf instance.

    • Builds on the highly successful YOLO architecture.

  • Benchmark Performance: On the standard COCO dataset, YOLOv8-Seg variants demonstrate strong performance, with yolov8s_seg achieving a COCO maskAP (val) of 36.80%, showcasing its generalizability.

  • Training Framework: PyTorch

Model Performance: YOLOv8s-Seg

The performance of the YOLOv8s-Seg model was thoroughly evaluated on the “Shopping cart analysis dataset v1.0”. Matrice AI provides tools for comprehensive model evaluation and tracking. The model was selected for its excellent balance of segmentation precision, inference speed, and efficiency for edge deployment.

YOLOv8s-Seg Performance Highlights (“Shopping cart analysis dataset v1.0”):

Metric

Validation

Test

Precision (Primary)

0.76

0.70

mAP@50

0.42

0.46

Fitness

0.56

0.57

The YOLOv8s-Seg model achieved a strong Test Precision of 0.70 and a Test mAP@50 of 0.462 for segmenting Shopping-trolley and Shelf instances. The high precision is particularly valuable for reliable asset identification, and a Test Fitness score of 0.57 indicates good overall model health. These results underscore its suitability for real-world deployment.

Training Highlights

The YOLOv8s-seg model was trained on the “Shopping cart analysis dataset v1.0” with the following key hyperparameters, with training runs managed on Matrice AI:

Hyper-parameter

Value

Rationale

Epochs

50

Sufficient cycles for convergence on a modest dataset without over-fitting.

Batch size

16

Balances GPU memory usage (under 8 GB) with gradient stability.

Learning rate

0.001

Standard starting point; cosine LR schedule was disabled for simplicity.

Optimizer

auto (e.g., SGD with momentum 0.95)

‘auto’ allows the Ultralytics framework to select an optimal optimizer. Momentum smooths updates.

Weight decay

0.0005

Regularization technique to combat overfitting.

Primary Metric

Precision

Focus metric for evaluating segmentation quality of trolleys and shelves.

Thoughtful data augmentation techniques were employed to boost model robustness.

Model Inference Examples

The system accurately segments Shopping-trolley and Shelf areas within store environments.

Retail Space Analysis Banner

Deployment Strategy: Real-Time Inference Pipeline

A typical deployment for retail space analysis, facilitated by Matrice AI’s MLOps capabilities, involves these steps:

  1. Camera Input: Strategically placed cameras capture images/video streams of store areas.

  2. Frame Processing: Frames are grabbed and pre-processed.

  3. Edge Inference: The YOLOv8-Seg model runs on an edge device (e.g., NVIDIA Jetson Orin) to perform instance segmentation of trolleys and shelves at approximately 60 FPS (latency <35 ms/frame).

  4. Output Processing: The JSON output containing segmentation masks and class labels (Shopping-trolley, Shelf) is parsed. This data can be used for location tracking and counting these assets.

  5. System Integration: Processed data is sent to analytics dashboards or store management systems for insights on customer flow, trolley availability, and shelf space utilization.

The Edge Advantage: The 11.8M parameter footprint of YOLOv8s-seg is crucial for efficient edge deployment. Matrice AI supports optimized model exports for various edge runtimes.

4. Real-World Applications and Business Impact

The implementation of AI-driven Shopping-trolley and Shelf segmentation, using solutions developed on Matrice AI, significantly impacts retail operations:

Enhanced Store Layout Optimization

By analyzing heatmaps of trolley traffic and dwell times near specific shelf segments, retailers can optimize product placement, promotional displays, and overall store flow to enhance customer engagement and sales.

Improved Customer Journey Insights

Tracking trolley paths from entrance to checkout provides valuable data on common routes, abandoned journeys, and areas of congestion, enabling data-driven decisions to improve the in-store experience.

Efficient Resource Allocation

Real-time data on trolley distribution can help staff quickly locate available trolleys or identify areas needing attention (e.g., trolley collection points), improving operational responsiveness.

Foundation for Advanced Automation

Accurate trolley segmentation is a prerequisite for more advanced systems, such as identifying trolleys approaching checkout to prepare staff, or as a first step before performing item recognition within the segmented trolley area in future automated checkout solutions.

5. Future Developments

The field of AI in retail is rapidly advancing, and Matrice AI is committed to enabling these innovations. Future enhancements for retail space analysis could include:

  • Item-Level Recognition within Segmented Trolleys: Building a second-stage model to identify individual products inside the accurately segmented Shopping-trolley boundaries.

  • Shelf Stock Analysis: Using segmented Shelf areas as regions of interest for analyzing on-shelf availability or detecting out-of-stock situations.

  • Integration with Store Analytics Platforms: Seamlessly feeding segmentation data (trolley counts, locations, paths; shelf interactions) into broader retail analytics dashboards.

  • Behavioral Anomaly Detection: Identifying unusual trolley movements or dwell times that might indicate customer confusion or potential security concerns.

  • Multi-Camera Tracking: Fusing data from multiple cameras to track trolleys across larger store areas for comprehensive flow analysis.

6. Conclusion

Automated instance segmentation of Shopping-trolleys and Shelf areas using models like YOLOv8-Seg, developed and deployed through platforms such as Matrice AI, provides foundational intelligence for transforming retail operations. As demonstrated with the “Shopping cart analysis dataset v1.0”, precise segmentation of these key retail assets enables a deeper understanding of store dynamics, customer behavior, and operational efficiency.

This technology empowers retailers with data-driven insights to optimize layouts, enhance customer experiences, and lay the groundwork for future automation. By embracing computer vision with Matrice AI, retailers can build smarter, more responsive, and more profitable store environments.

Think CV, Think Matrice

Experience 40% faster deployment and slash development costs by 80% with Matrice AI.

Book a Demo Sign Up