On the Control of the Technical Condition of Elevator Ropes Based on Artificial Intelligence and Computer Vision Technology

Introduction. Increased attention is paid to safety issues during the operation of elevator equipment^{1, 2, 3}. About 500 thousand passenger, hospital and freight elevators are being used in the Russian Federation. According to the statistics of elevator accidents, their prime causes are insufficient quality of elevators and components, as well as improper maintenance and repair.

Analysis of elevator accidents has shown that about 30 % of incidents are directly or indirectly related to the technical condition of the ropes [1]. Extremely dangerous are defects, whose growth causes elevator car fall. Passenger elevator accidents, especially tragedies with victims, are very resonant events. Administrative⁴ and criminal⁵ liability is provided for violation of safety requirements during operation of elevators.

Timely and high-quality control of the technical condition of steel ropes provides preventing dangerous incidents [2–5].

Damage to the ropes occurs due to their poor quality and errors during installation. The consequence of incorrectly performed work may be unacceptable interaction with elements of the elevator equipment, including electrical ones.

Steel ropes can be considered an indicator of the technical condition of elevator equipment. Their defects signal the pulley groove wear, rope slip on a pulley (including due to excessive lubrication), skewed winch mounts, backlash in the elements of the drive gears, etc. Possible malfunctions of the elements of the mechanical part of the elevator, depending on the defective parameters of the ropes, are presented in Table 1.

There is a significant difference in the rejection indicators and the standards for the rejection of ropes of various lifting structures^{6, 7} and elevators. In the latter case, the criteria are included in the operating manual of the specific elevator. This approach reduces the number of defect indicators, which, specifically, does not contribute to improving operational safety. In [1], the criteria and norms of rejection are given.

Materials and Methods. Ropes are products of deep processing of steel. If their safety margin is reduced to a critical value, operation is unacceptable.

The control of ropes of lifting structures by magnaflux method is widely known. It is used in the diagnosis of passenger cable cars, cargo cranes, lifters. This approach makes it possible to recognize internal defects invisible to the eye, measure the loss of the cross-sectional area of the rope, and detect local inconsistencies. As a rule, the method is used to control steel ropes with a diameter of more than 20 mm.

The average service life of elevator ropes is regulated and does not exceed five years. During this time, loading cycles do not cause loss of cross-sectional area and fatigue damage to the wires of the inner layers. For this reason, magnetic flaw detection is not used for elevator installations.

The most common method of assessing the technical condition of elevator ropes is visual inspection control (VIC)⁸. It provides assessing the technical condition of the operated elevator ropes, which are evaluated according to the following standards of rejection:

• geometry change P1(t);
• corrosion and wear P2(t);
• wire breaks (fatigue) P3(t);
• temperature effect P4(t), etc.

Here, t — current operating time of the elevator.

The rejection rate is understood as the value of Pi(t):

Pi(t) = Pio + Pi'(t). (1)

Here, Pio — scleronomous part, it does not depend on t and is associated with a single application of loads; Pi'(t) — rheonomic part, which accumulates in the process of elastoplastic cyclic deformation.

When assessing the technical condition of the rope during operation, the loss of its strength is calculated according to at least one of the rejection criteria:

Pi(t) ≤ [Pi], (2)

where i = 1/12.

Under the maintenance of passenger elevators, VIC of steel ropes is performed at intervals specified in the operations manual (as a rule, once a month).

VIC of steel ropes consists of two stages:

1. inspection of the rope and its attachment points with or without optics;
2. measurements of diameter, wear of outer wires, geometric out-of-straightness, electric shock, and attachment points.

The quality of VIC under maintenance of steel ropes is associated with the physiological capabilities of human visual perception [4–6]. The elevator can have from three to twelve parallel ropes with a complex balancing system. The quantity depends on the height and load capacity. Health assessment of ropes, balancing units and sealing off involves working at height, under difficult conditions, in tight working space, in low light⁹. The possibilities of human vision are limited. It is known that after watching a moving steel rope for more than ten minutes, the eyes get tired and perceive the numerous rejection indicators worse. High-quality, objective control of elevator ropes requires automation of the process. Analysis of literature sources, accidents and incidents, has shown that it is reasonable to use solutions based on computer vision and artificial intelligence technologies for automation [5–10].

The authors set a goal to create a program apparatus complex (PAC) of VIC for the practical implementation of visual and measurement control based on computer vision and artificial intelligence technologies¹⁰.

Research Results. In elevators, defects in steel ropes and their attachments are automatically detected and identified using special means of photo and video recording. Machine vision and artificial intelligence process images of defects. An integral assessment of the types and parameters of the detected defects in the form of a color indication is transmitted to the mobile device. PAC VIC automates and combines two control methods: visual and instrumental.

The basic functions of the PAC VIC are listed below.

1. The solution based on a deep convolutional artificial neural network automatically detects and classifies the primary defects of ropes and their sealing off.
2. A set of cameras and an image scanning algorithm with distortion compensation give a three-dimensional image of the rope for the actual determination of the metric characteristics of defects.
3. On totality of the detected defects, an integral assessment of the technical condition of the rope is performed on the basis of risk analysis.
4. The integral assessment in the form of a color indicator is periodically transmitted to the user's mobile device.

The complex consists of three elements.

1. Hardware component. This is a housing with built-in video cameras that provide viewing the rope from all sides. A special module processes the video stream, and means of communication allow you to contact the server.
2. Software, created using a stack of Pytorch, TensorFlow, OpenCV technologies, searches for and classifies defects in steel ropes based on a deep convolutional artificial neural network.
3. Mobile application informs the user about the monitoring results.

Six IP cameras with a resolution of 640.480 are used to photograph sections of movable steel ropes. They are fixed in the housing installed in the elevator shaft. The number of cameras depends on the number of ropes on which the elevator car is suspended. The cameras are connected via a wired or wireless communication channel to a single-board computer that processes images and generates a batch message for forwarding to the server. A single-board computer generates a batch message. It is transmitted via an Ethernet cable or a wireless communication channel through a modem router to the server. The latter classifies defects, determines their number and metrological parameters, outputs an integral assessment of the technical condition of the rope, interprets it as a color indicator, and in this form transmits it to the user through a mobile application.

Figures 1–3 shows the PAC VIC modules for elevator installations, the functional diagram and the hardware component of the complex.

PAC VIC identifies and classifies primary defects of steel ropes, elevator installations and balancing units using computer vision:

1. outer wire breaks;
2. surface wear of wires;
3. increase or decrease in rope diameter;
4. deformation in the form of undulation (“corkscrew”);
5. electric arc damage;
6. balancing unit defects.

The database contains photo and video images of typical defects of steel ropes and balancing units. The complex periodically or on request outputs a remote integrated assessment of the rejection indicators of ropes and defects of balancing units and transmits it to the user's mobile application.

The computational tool is used for an integral assessment of the technical condition of the steel rope and interpretation of the result using color (red, yellow, green). This makes it possible to integrate with sensors installed in elevator control and dispatch communication systems, as well as with the “Elevator Monitoring”¹¹ online maintenance control program and the “Arpicon. Online”¹² cloud service automation platform.

The proposed solution was previously tested on a laboratory elevator installation with three ropes with an organic core. Their characteristics correspond to GOST 3077-80:

• double LK-O type rope;
• structure 6.19 (1 + 9 + 9) + 1 o. s.;
• diameter of 10 mm.

Testing has shown the feasibility of the method for determining the defects listed above using the PAC VIC. The reliability of detection and qualification of defects exceeded 80%. To improve the results, the investigation on deep learning of the system continues, i.e., algorithms are being improved, the number and quality of datasets of images of typical defects of steel ropes and balancing units are increasing.

Discussion and Conclusions. Defects in the steel ropes may cause the elevator cab to fall. When maintaining elevators, visual control is not enough to fully assess the actual risks of accidents and incidents.

It is established that computer vision and artificial intelligence technologies can be used for an integral assessment of the technical condition of steel ropes, and this significantly increases the level of safety during the operation of elevator installations.

The PAC VIC of elevator ropes eliminates the risks associated with the psychophysical state of the person who determines the state of the system. Thus, it is possible to increase safety of the elevator equipment operation.

Advantages of the developed PAC VIC of elevator ropes are as follows:

• functioning in automatic mode, directly in the operated elevator;
• contactless detection of rejection indicators using photo and video surveillance (machine vision);
• automatic evaluation of rejection indicators according to five criteria online (external wire breaks, surface wear, local increase or decrease in rope diameter, undulation, temperature exposure);
• transfer of color-coded rejection indicators to the user's mobile application;
• processing of images with defects of steel ropes by artificial intelligence.

Potential consumers of PAC VIC:

• operators — organizations responsible for the technical condition of elevators;
• home unit companies, homeowners' associations;
• manufacturers, buyers and sellers of passenger elevators.