Sumitomo Heavy Industries, NEC Develop AI Construction Safety System

New system analyzes excavator video and sensor data to detect near-miss incidents and automatically generate safety reports for construction site operations.  www.nec.com Construction, civil engineering, and heavy equipment operations increasingly require digital safety systems to manage risks in dynamic and unpredictable environments. Sumitomo Heavy Industries, Ltd. and NEC Corporation will begin joint development in April 2026 of an AI-based system designed to automatically identify near-miss incidents and generate reports using data from hydraulic excavators. The system aims to address a gap in construction safety technologies by providing end-to-end functionality—from data acquisition to incident reporting—within a single integrated platform. Addressing limitations in construction safety monitoring Construction sites are subject to constantly changing conditions, including weather, terrain, and workflow variations, which increase the likelihood of hazardous situations. While video monitoring and data logging are already used, existing solutions typically lack the ability to automatically extract and analyze risk scenarios across large datasets. The proposed system is designed to identify “risk scenes” from machine operation data and video footage, enabling automated detection of near-miss incidents. This reduces reliance on manual review processes, which can be time-consuming and inconsistent. AI-driven extraction and analysis of risk scenes The system architecture combines multiple AI technologies to process and interpret operational data. An extraction AI model, trained on real-world hydraulic excavator data from the SHI Group’s ICT/IoT platform “SHICuTe” (Note 1), identifies relevant risk scenes from recorded video footage. These scenes are then analyzed alongside machine sensor data using NEC’s technology, which integrates video recognition with generative AI (Note 2). The system converts this information into multimodal datasets incorporating both temporal and spatial context. By correlating this data with predefined hazard patterns—such as unsafe operator behavior, equipment misuse, or known accident scenarios—the system can automatically determine which events require reporting. Automated generation of near-miss reports Based on the analyzed data, the system generates structured near-miss reports summarizing each incident. These reports include contextual information about the event, enabling safety managers to understand underlying causes and implement preventive measures. The automation of reporting processes supports more consistent documentation and faster feedback loops, which are essential for improving safety performance across construction sites. Validation and roadmap toward deployment A proof of concept conducted in September 2025 demonstrated the system’s ability to extract and report near-miss incidents from excavator-mounted camera footage. The results confirmed that the system could identify potential accident scenarios and provide meaningful summaries of each case. The next phase of development, starting in April 2026, will focus on expanding the range of detectable incidents and improving report accuracy to align with real-world safety management requirements. Practical implementation is targeted for fiscal year 2027. Future enhancements are expected to include detection of less visible risks, such as unsafe site conditions that may not be immediately recognized by workers, as well as adaptation to site-specific operational rules. Positioning within construction safety technologies Digital safety systems in construction are evolving toward integrated platforms that combine machine data, computer vision, and AI analytics. Comparable developments are seen from companies such as Komatsu and Caterpillar, which also incorporate telematics and monitoring systems into heavy machinery. However, most existing solutions focus on data collection and monitoring rather than fully automated incident extraction and reporting. The joint development by Sumitomo Heavy Industries and NEC addresses this gap by integrating analysis and reporting into a single workflow. By combining machine data, video analytics, and generative AI, the system represents an approach to improving construction site safety through automated identification and documentation of near-miss events. Notes: 1. SHICuTe is a common platform that enables cross functional development of capabilities across the product lines of the SHI Group. It collects and stores various operational data from the group’s products connected online. "SHICuTe" is a registered trademark of Sumitomo Heavy Industries, Ltd. 2. "NEC uses generative AI (LLM) and video recognition AI to automatically generate explanatory text from video - Applied to drive recorder videos, cutting accident report creation time in half -" (Announced by NEC on December 5, 2023) Edited by Natania Lyngdoh, Induportals Editor — Adapted by AI. www.nec.com P