DOI 10.2478/jok-2019-0074
Paweł HYLA, Janusz SZPYTKO
AGH University of Science and Technology (Akademia Górniczo-Hutnicza)
THE VISION TECHNIQUE CONCEPT SUPPORT
CRANE SAFETY EXPLOITATION PROCESS
Koncepcja zastosowania techniki wizyjnej w zapewnieniu
bezpieczeństwa eksploatacji suwnic pomostowych
Abstract: The issue of ensuring safety in the crane operation process is the subject ofstandards and regulations: the aim is to apply solutions aimed at ensuring the health and safety of users of cranes and the work environment. The subject of the paper is the concept of using vision technology in ensuring the safety of overhead travelling type cranes. Using a camera based vision system in conjunction with dedicated sensors, it is possible to build a system for monitoring the degradation process of selected crane operating parameters and conditions of use in a specific dynamically changing environment, as well as the operator's characteristics. Using the developed standards of safe use of the crane, it is possible to generate safe behaviors resulting in the device being immobilized or the potential hazard being evaporated.
Keywords: vision techniques, work safety, cranes
Streszczenie: Zagadnienie zapewnienia bezpieczeństwa w procesie eksploatacji dźwignic
jest przedmiotem norm i przepisów: celem jest stosowanie rozwiązań ukierunkowanych na zapewnienia zdrowia i bezpieczeństwa użytkowników urządzeń dźwignicowych oraz środowiska pracy. Przedmiotem artykułu jest koncepcja zastosowania techniki wizyjnej w zapewnieniu bezpieczeństwa eksploatacji suwnic pomostowych. Z użyciem systemu kamer w połączeniu ze specjalistycznymi czujnikami możliwa jest budowa systemu nadzorowania procesu degradacji wybranych parametrów eksploatacyjnych dźwignicy oraz warunków użytkowania w określonym zmiennym dynamicznie otoczeniu, a ponadto właściwości operatora. Standardy bezpiecznego procesu użytkowania dźwignicy umożliwiają generowanie zachowań bezpiecznych skutkujących unieruchomieniem urządzenia lub odparowaniem potencjalnego zagrożenia.
1. Introduction
The modern approach to the crane operator as the subject of the weak link of the control system enforces to the implementation of preventive solutions aimed at improving the reliable of human being through avoid so-called sensory deprivation [7, 11]. First of all the modern technology allows filtering information, buffering and selective presenting the essential data improving presentation [1]. All this thinks gather together have a significant impact on the crane operator concentration and correct decisions taken from the point of view both the technological process and crane exploitation [7]. Vision system technology enables a transparent approach to issues related to obtaining and presenting information in an unambiguous, legible and redundant way in the transmitted content [10, 15, 16]. The issue is important especially in the process of supporting the crane operators in daily work. The crane operators are exposed to unfavorable effects of an active and passive nature. For example in the metallurgy, unfavorable active influences are connected with the production technology itself, which is mediated by transport devices. Passive impacts result from the nature of the devices operation, its location and additional factors such as vibrations, high noise level, temperature, toxic fumes, high pollution and air dustiness. These factors make it necessary to take any possible actions aimed at increasing not only the level of reliability of the device, but also the crane operator point of view [13].
The article presents a complex vision system dedicated for the overhead travelling crane work space and operator surveillance with vision system assistance [4, 5, 9] enriched by image analysis [6, 8, 14] technique as a supporting systems in chosen field. Presented in the paper applications were tested with a positive result on a physical laboratory model of a bridge crane with the lifting 150 kg and 1000 kg capacity.
2. Vision system dedicated for the crane - generations class
Visual perception deliver to the human being more than 80% [2] information about the environment. The growth of visual information in our surroundings has raised a demand for effective acquire and processing visual information. Human sense of sight has been developed in course of millions of years of natural evolution and near a half of man cerebral cortex is busy with processing visual information. On the base human sense of sight, at now engineers and scientist trying gain answers how to attached vision possibility to the machines or use vision high potential.2.1. Vision methods in crane - generation 1
stand 1
stplus
More and more often we hear about the new generation of products. Sources concerning this expression are extremely poor. As a basic definition in reference to techniques it possible to distinguish a group of machines, devices, vehicles constructed at a
particular stage of their development, according to a specific technology use. However this definition may be use in variety kind of applications in wide spectrum from foods products to advanced electronics.
As a first generation (1st gen.) of vision system used in the crane is possible to list first
sets of CCTV (Closed Circuit Television) mounted on the crane supporting interactions between the crane operator and the device through clean not filtered video stream. Usually a crane operator works around 40% of his time without full direct view of his payload and operating zone. A hidden work place and unknown payload dimension in combination of open workspace were winds can blow in any moment and disturbance payload trajectory need special attentions and care. One of the first solution for this issue offering a solution was use a poor resolution and low FPS (Frame Per Second) cameras. It was not very effective but can help in some situation when the load or work zone was not clearly visible from the operator actual position. Though some defect in this kind of vision use a new approach was created. They were quickly notice that this kind solution has high potential. In fig. 1 was presented the CCTV system dedicated for overhead travelling crane contain nine cameras presented view from variety device points. The presented system can be included to 1st plus generation due to use high resolution cameras with images flow.
Fig. 1. Gen. 1st plus vision system example dedicated for overhead travelling crane
However, regardless of whether we are dealing with the 1st gen. or 1st plus gen. from
psychological point of view the system presented only clean video stream is not very effective. According with survey presented in [3] vision stream has positive effect and support operator only if use in two hour long cycle with maximum four display showing the image at the same
time. Research related to the interaction of vision systems with the operator has led to further work and development of vision systems new generation in cranes.
2.2. Vision methods in crane - generation 2
In general the 2 gen. vision system must has extended possibilities and architecture than 1 gen system vision system. The possibilities of achieved video stream must be enrichment through other services. Usually the system of gen. 2 has double video stream possibility to gain possibilities to attach www (world wide web) services. In surveillance system dedicated for property protection the second video stream must be lower size. It allow effective transfer without delay through variety link and communication services.
Adapting the gen. 2 functionality to the vision system dedicated to the crane, the two stream functionality must be adopted too. In the gen. 2 video supervision system dedicated to the crane must be distinguished on-line and off-line system. The off-line system is related with the specific transport device while the on-line system is related with whole transport system. In fig. 2 was presented a double stream monitoring system dedicated for overhead double-girder crane. This system was develop on AGH University of Science and Technology and more details were described in scientific paper [13]. The described video system has two independent vision system with double stream functionalities.
The off-line system was dedicated to task related with overhead travelling crane workspace registration and contain commonly use RGB cameras directly connected to the digital card in PC (Personal Computer), which plays a role of the vision server enabling the video stream distribution possibilities. Video images stored by PC server with proper hardware-software architecture allows coding the video files with high efficiency, even in real time mode. In the next step the smallest amount of data can easily transfer via mobile data transmission systems like GSM, UMTS, LTE or others systems without delays. The
on-line acquisition system was built on the base high-resolution megapixel cameras with an
IP (Internet Protocol) communication interface, equipped with lenses with variable focal length and auto-iris function enabling auto-adaptation to the current lighting conditions. The functionality of IP cameras was used for on-line system and allow observing actual situation via the www services and even simply web browser with Active-X protocol enabled. The size of the transmitted data packet to the currently recipient is adapted on the base it internet link bandwidth. The developed monitoring system is flexible and has the features of the full reconfigurability and future upgrades implementation.
2.3. Vision systems generation – generation 3
Today modern semi-automatic vision systems usually is supported by the continually improvement image processing technique. The main result of this change is vision systems efficiency growth. Technical progress deliver new class of vision system so-called machine vision system (MVS). This technology use smart cameras as a primarily vision system unit but not only. Smart camera is a vision system that not only takes images, but can understands them. Smart camera can be defined as a complementary vision system [1] which, in addition to image capture circuitry, is capable of extracting application-specific information from the captured images, along with generating event descriptions or making decisions that are used in an intelligent and automated system. In relation to cranes, all systems in which smart cameras are used can be called the 3-generation vision systems. The smart camera in cranes can be used as a contact less swinging sensor. The process of transport the cargo suspended on the cables by the all type crane devices are always accompanied by swing phenomena that destructively affect the crane equipment and significantly hamper the exact positioning of the moved payload. The described phenomenon of inducing excessive amplitudes of the payload swing is most often the result of excessive accelerations generated by the crane mechanisms in the non-stationary states. In automatic crane control systems prevention of payload swigs requires a feedback signal that returns the amplitude of the steel cable angle at any time. The direct contact measurement systems dedicated for this purpose for measuring the sway deflection angle of the payload constitute a complicated issue. The problem is solvable in the predefined architecture. The smart camera must be suspended under the trolley hoist mechanism (fig. 3.) thereby field of view contain all ropes scrolling through pulley block fixed with the crane hook. The method of measuring the swing angle of the rope or ropes and laser marker was presented in fig. 4. In the diagram the swing angle in the x-y plane is obtained directly
due to the fact that the projection of the plane of the ropes is perpendicular to the object plane directly associated with the camera.
Fig. 3. Camera suspended under the crane hoist mechanism (crane trolley) with laser line emitter in the grip
Fig. 4. The scheme of swing angle extraction with using a single camera and laser marker (direct method)
Indirect measurement method of the sway in x-z axis is more complicated. Measuring of the rope swing in the x-z axis direction has been carried out according to diagrams showed in fig. 5. Moreover, in fig. 5 it was presented a full frame contain view on the crane ropes illuminated by the laser beam recorded by the camera suspended under the crane trolley. The detailed indirect γ angle extraction with math solution was presented in [7].
Fig. 5. Methodology of the γ angle extraction (swing in the x-z axis) with the real photo
2.4. Vision systems generation – generation 4
The progress in automatic control and steering in transport means, especially in the cranes, requires develop new methods and tools supporting them. Supervising the transport means work space get possibility implementation more automatic solution and enable to mark out the optimum grade-separated trajectory of transferring cargo in automatic mode with use a chosen algorithm. The overhead travelling crane workspace modelling is
necessary to minimize delays associated with safe payload trajectory design and by-pass
identified obstacles. The problem of ensuring the safe and efficient cranes operations in automated manufacturing processes involves the needs of the operating workspace identification. The vision system not only in the material handling devices ensure autonomous in this field. Automatic reconstruction of physical objects with their geometrical structure can be subdivided into two kinds of methods: active methods use structured light (e.g. laser scanner) and passive method use 2D image obtained from many points or one camera with changing point of view [12].a) passive method
The architecture of stereo vision system can be based on the single camera suspended under crane's trolley. The acquired stereo pair were taken as a sequence of snapshots during crane usual work. Additionally current location of the camera must be identified through measuring the crane or trolley position this can be achieve thought incremental encoders attached to the crane wheels. The presented approach guarantees identical focal length for each snapshot, and additionally the images baseline is closely parallel oriented to the image plane. In fig. 6 presents the example of the rectified stereo pair of images with epipolar transformation.
Fig. 6. Epipolar transformation of the stereo pair of images
When the stereo pair geometry is perfectly parallel, the disparity in y direction doesn't exist. Thus, disparity map contains only disparities allocated in x direction (fig. 7) determined based on stereo images baselines displacements.
Fig. 7. Disparity scheme with obtained dense disparity map for fig. 6 b) active methods
In fig. 8 was presented a panoramic view of the overhead travelling crane space work taken from their bridge. However a presented figure is not an image. It is a render as a result obtained vision data processing from the 3D scanner. In result of use laser scanner is a large number of points (so-called cloud of point) defined by X, Y and Z in scanning device local coordinates system. A point cloud is a set of vertices in a three-dimensional coordinate system represents the set of points that device has measured. There are many techniques for converting a point cloud to a 3D type surface but the reconstruction especially precise surfaces from unorganized point clouds derived from laser scanner or other methods is a very hard problem, not completely solved and problematic. The best way is always create
a computer model of an object which best fit the reality but contains as small as possible number of point.
Fig. 7. Unfolding 3D structure to cylindrical 2D picture on 3D scan base – render image
Contemporary laser scanners enables a large quantity, dozen millions points number three-dimensional model. Data are collected in a short period of time and each individual point has additionally information about reflection intensity value. This functionality allows generating a point cloud in a local coordinate system with additional color information in RGB type mode.
3. Conclusion - future of vision system in cranes –
anticipation
With a strong probability the near future of vision system in crane devices going into use vision system as a primary sets of solution integrated with autonomic or semi-autonomic systems. The future generation of vision solution can be establish as set of features focused on:
• It is necessary to eliminate the weak link what constitute human factor as cranes operator through the implementation vision solutions detecting and preventing dangerous situations related to the unauthorized device use and detecting operator's fatigue (such solutions are used in the automotive),
• Application of vision systems enabling fully independent crane’s operation possibilities without a human interaction. The operator role should be shifting in to only role transport device (or system) supervisor,
• Realize and implementation more integration between transport means changing the approach to communications of the type Human – Transport Device to Transport Device – Transport Device,
• Detecting payload trajectory schemes for optimizing and proper scheduling any transport tasks realize in the future by implementation more automatic solution and enable to mark out the optimum grade-separated trajectory of transferring cargo in automatic mode with use a chosen algorithm.
Acknowledgement
The work has been financially supported by the Polish Ministry of Science and Higher Education.
4. References
1. Belbachir A.N. (ed.): Smart Cameras. Springer, 2009.
2. Diamant E.: Modeling Visual Information Processing in Brain: A Computer Vision
Point of View and Approach.Lecture Notes in Computer Science, v. 4729, Springer, Berlin/Heidelberg 2007.
3. Dulski R., Życzkowski M., Bieszczad G., Kastek M., Szustakowski M., Ciurapiński W.: Synteza danych w wieloczujnikowym systemie ochrony perymetrycznej obiektów. Biuletyn Wojskowej Akademii Technicznej, Vol. 60, Nr 2, 2011.
4. Hyla P.: Multi camera triggering and synchronization issue - case study. Journal of KONES, Vol. 23, No. 3, 2016.
5. Hyla P.: Night vision applicability in anti-sway vision-based solutions. Proceedings of the 20th International Conference of Methods and Models in Automation and Robotics MMAR 2015, Międzyzdroje, August 24–27, Poland, 2015.
6. Hyla P.: Stereovision system for overhead travelling crane workspace visualization - validation approach. Proceedings of the 18th International Conference of Methods and Models in Automation and Robotics MMAR 2013, Międzyzdroje, August 26–29, Poland, 2013.
7. Hyla P., Szpytko J.: Telematics For The Purpose of Identifying The Conditions of Use of a Bridge Crane. Communications in Computer and Information Science, vol. 897, Springer Nature Switzerland AG, 2018.
8. Hyla P., Szpytko J.: The application of image analysis methods in selected issue solution dedicated for overhead travelling crane. Journal of KONES Powertrain and Transport Means, Vol. 21, No 2, Warsaw 2014.
9. Hyla P., Szpytko J.: The possibility of use a smart camera as a non-contact sway sensor in insufficient light condition – case study. Archives of Transport System Telematics, Vol. 8, Iss. 1, 2015.
10. Roberts D., Bretl T., Golparvar-Fard M.: Detecting and Classifying Cranes Using Camera-Equipped UAVs for Monitoring Crane-Related Safety Hazards, ASCE International Workshop on Computing in Civil Engineering (IWCCE), Univ. Washington, Seattle, 25-27 June 2017.
11. Smoczek J.: Intelligent crane control systems. ITE, Kraków-Radom 2010.
12. Szpytko J., Hyla P.: Material handling devices operation environment 3D-type presentation based on laser scanning systems. Journal of KONES, Vol. 17, No 2, 2010. 13. Szpytko J., Hyla P.: Techniki wizyjne w procesach nadzorowania procesu eksploatacji
suwnic hutniczych. Hutnik, Nr 7, 2012.
14. Szpytko J., Hyla P.: Work space supervising for material handling devices with machine vision assistance. Journal of KONBiN, Safety and reliability systems, No. 3-4, Warsaw 2009.
15. Trinklein E.H., Parker G.G., Zawisza M.S.: Active Load Damping of an Extending Boom Crane using a Low Cost RGB-D Camera. 12th IEEE Sensors Applications Symposium (SAS), Rowan Univ, Glassboro, 13-15 March 2017.
16. Yang J.S., Huang M.L., Chien W.F., et al.: Application of Machine Vision to Collision Avoidance Control of the Overhead Crane. International Conference on Automation, Mechanical and Electrical Engineering (AMEE), Vol. 13, Phuket, Thailand, 26-27 July 2015.
