Difference between revisions of "3D fish tracking system"
Bendikhansen (talk | contribs) m (Text replacement - "{{Note|This technology has been developed in the FIThydro project! See Innovative technologies from FIThydro for a complete list.|reminder}}" to "{{Note|This technology has been developed in the FIThydro project! See Innovative technologies from FIThydro for a complete list.|reminder}} Innovative::developed in FIThydro") |
|||
| (14 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
{{Note|This technology has been developed in the FIThydro project! See [[Innovative technologies from FIThydro]] for a complete list.|reminder}} | {{Note|This technology has been developed in the FIThydro project! See [[Innovative technologies from FIThydro]] for a complete list.|reminder}} | ||
| − | + | ||
=Quick summary= | =Quick summary= | ||
| − | [[file:3d_fish_tracking_installation1.jpg|thumb|250px|Figure 1: 3D fish tracking system | + | [[file:3d_fish_tracking_installation1.jpg|thumb|250px|Figure 1: 3D fish tracking system installed in the etho-hydraulic flume at VAW of ETH Zurich (source: VAW)]] |
[[file:3d_fish_tracking_equipment.jpg|thumb|250px|Figure 2: (a) Camera (acA2000-50gmNIR, Basler) with lens (FE185C086HA-1, Fujifilm), (b) waterproof housing for the camera and lens (Autovimation), (c) high performance computer for camera recording and network switch for camera connection (source: VAW).]] | [[file:3d_fish_tracking_equipment.jpg|thumb|250px|Figure 2: (a) Camera (acA2000-50gmNIR, Basler) with lens (FE185C086HA-1, Fujifilm), (b) waterproof housing for the camera and lens (Autovimation), (c) high performance computer for camera recording and network switch for camera connection (source: VAW).]] | ||
[[file:3d_fish_tracking_3d_ouput.jpg|thumb|250px|Figure 3: (a) Stereo view of a camera pair, (b) three detected fish and noise, (c) 2D tracks of three fish (source: VAW).]] | [[file:3d_fish_tracking_3d_ouput.jpg|thumb|250px|Figure 3: (a) Stereo view of a camera pair, (b) three detected fish and noise, (c) 2D tracks of three fish (source: VAW).]] | ||
| − | [[file:3d_fish_tracking_fish_tracks.jpg|thumb|250px|Figure | + | [[file:3d_fish_tracking_fish_tracks.jpg|thumb|250px|Figure 4: Top view of 3D tracks of three fish from an etho-hydraulic test of fish guidance structure with horizontal bars (source: VAW).]] |
Date: 2018 | Date: 2018 | ||
| Line 21: | Line 21: | ||
=Application= | =Application= | ||
| − | Within the scope of FIThydro, VAW investigates two types of fish guidance structures (FGS), namely with horizontal (Figure | + | Within the scope of FIThydro, VAW investigates two types of fish guidance structures (FGS), namely with horizontal (Figure 1) and vertical curved bars. These FGSs are tested with six different fish species under various hydraulic conditions to evaluate their fish guidance efficiencies and to understand fish behaviour. To this end, the 3D fish tracking system is further developed and tested in these etho-hdyraulic (live-fish) investigations. The present system is similar to that currently used by the German Federal Institute for Hydraulic Engineering (BAW) in Karlsruhe together with the German Federal Institute of Hydrology (BfG, 2018; Detert et al., 2018). |
| − | The present system consists of up to five cameras arranged in a streamwise series facing vertically upwards through the water surface, each with a distance of 1.5 m (Figure | + | The present system consists of up to five cameras arranged in a streamwise series facing vertically upwards through the water surface, each with a distance of 1.5 m (Figure 1). Model acA2000-50gmNIR cameras from Basler are used and equipped with a 185° fisheye lens of FE185C086HA-1 (Fujifilm) (Figure 2a). The camera resolution is 3 MPx. Each camera and lens are waterproofed using a housing from Autovimation (Figure 2b). A GigE Vision 2.0 network with a Precision Time Protocol (PTP) IEEE1588 provided synchronous measurements with frame rates kept constant at 20 fps (Figure 2c). For larger control volume and longer areas, the actual system including the network switch and the high performance PC can theoretically be equipped with up to 48 cameras. However, the frame rate will be lower then. |
| − | An adapted software by Fujifilm Switzerland is used to set-up cameras and record videos. The etho-hydraulic flume is illuminated with 7x1000 W halogen lamps (Figure | + | An adapted software by Fujifilm Switzerland is used to set-up cameras and record videos. The etho-hydraulic flume is illuminated with 7x1000 W halogen lamps (Figure 1). Calibration of the system is essential and made in three steps: finding intrinsic and extrinsic parameters for each of the five cameras using a checkboard, calibrating five stereo cameras according to the overlapping views of camera pairs, and finally performing a rigid transformation of all stereo camera pairs to a global flume coordinate system (Figure 3a, Detert et al., 2018). |
| − | 3D fish tracking is based on the detection of moving fish in each frame and associating the detections corresponding to the same fish over time. These are done by using a background subtraction algorithm and a Karman filter in MATLAB (Detert et al., 2018). The primary results of motion-based tracking are tracks in a distorted and uncalibrated 2D image frame coordinate system for each camera. Figure | + | 3D fish tracking is based on the detection of moving fish in each frame and associating the detections corresponding to the same fish over time. These are done by using a background subtraction algorithm and a Karman filter in MATLAB (Detert et al., 2018). The primary results of motion-based tracking are tracks in a distorted and uncalibrated 2D image frame coordinate system for each camera. Figure 3b and c show the three detected fish and noises caused by reflections from the glass window and their 2D tracks over time. After undistorting such frames and stereo calibrating the cameras, the 2D fish tracks are transferred to a 3D metric-space according to their epipolar geometry based on the camera parameters derived from the calibration (Figure 4). |
The etho-hydraulic tests were done for a flow depth of 90 cm, flume width of 150 cm, distance of 150 cm between the cameras and average flow velocities up to 0.7 m/s. Under such conditions, the 3D fish tracking system provided fish positions in 3D with an accuracy of about ±5 cm and 20 fps. The challenges for a successful implementation of the system are: assignment of individual fish to the tracks, constant illumination of the flow, camera distortion, air bubbles and suspended sediment and humid conditions for the cameras. | The etho-hydraulic tests were done for a flow depth of 90 cm, flume width of 150 cm, distance of 150 cm between the cameras and average flow velocities up to 0.7 m/s. Under such conditions, the 3D fish tracking system provided fish positions in 3D with an accuracy of about ±5 cm and 20 fps. The challenges for a successful implementation of the system are: assignment of individual fish to the tracks, constant illumination of the flow, camera distortion, air bubbles and suspended sediment and humid conditions for the cameras. | ||
| Line 36: | Line 36: | ||
=Other information= | =Other information= | ||
| − | The total costs of the present system is approx. 40’000 USD=35’000 € including camera set-up and | + | The total costs of the present system is approx. 40’000 USD=35’000 € including camera set-up and recording software. For current costs of the equipment, we recommend to ask the corresponding supplier listed below. Note that a cheaper camera and lens set-up can significantly reduce the total cost of the system. The MATLAB-based 3D tracking code developed by VAW will be freely available. |
Latest revision as of 15:44, 2 October 2020
Contents
Quick summary
Date: 2018
Developed by: VAW of ETH Zurich
Introduction
Laboratory investigations with live-fish, i.e. so-called etho-hydraulic tests, serve to understand interactions between the hydraulics of fish protection technologies and fish behaviour and hence to improve the current design of fish passages or develop new technologies. For laboratory application, VAW of ETH Zurich developed a three dimensional (3D) fish tracking system consisting of synchronous vertically submerged cameras and a MATLAB-based 3D tracking software to determine fish locations in the flow from the recorded videos (Figure 1).
Typically, the behaviour of aquatic fauna is documented by manual protocol written down by biologists and (optional) supplementary video recording. The main drawbacks of both techniques are (i) time consuming, (ii) low time resolution, (iii) low spatial resolution, and (iv) providing only qualitative information. The present 3D fish tracking system overcomes such drawbacks by automatically and accurately providing 3D swimming tracks on a larger metric space at a milliseconds time resolution.
The software tracks several fish in 3D. Swimming path-time diagrams give a distinct ‘big picture’ of the fish movement, which helps to identify fish preferred and disliked regions. Furthermore, detailed 3D path analyses of fish interactions and fish velocities are provided as well. The details of the system are documented below. Although the 3D fish tracking system is developed for laboratory use, it may be applied in an adapted version in the field to monitor fish movements or counting, as long as the visual observation is not compromised by turbidity.
Application
Within the scope of FIThydro, VAW investigates two types of fish guidance structures (FGS), namely with horizontal (Figure 1) and vertical curved bars. These FGSs are tested with six different fish species under various hydraulic conditions to evaluate their fish guidance efficiencies and to understand fish behaviour. To this end, the 3D fish tracking system is further developed and tested in these etho-hdyraulic (live-fish) investigations. The present system is similar to that currently used by the German Federal Institute for Hydraulic Engineering (BAW) in Karlsruhe together with the German Federal Institute of Hydrology (BfG, 2018; Detert et al., 2018).
The present system consists of up to five cameras arranged in a streamwise series facing vertically upwards through the water surface, each with a distance of 1.5 m (Figure 1). Model acA2000-50gmNIR cameras from Basler are used and equipped with a 185° fisheye lens of FE185C086HA-1 (Fujifilm) (Figure 2a). The camera resolution is 3 MPx. Each camera and lens are waterproofed using a housing from Autovimation (Figure 2b). A GigE Vision 2.0 network with a Precision Time Protocol (PTP) IEEE1588 provided synchronous measurements with frame rates kept constant at 20 fps (Figure 2c). For larger control volume and longer areas, the actual system including the network switch and the high performance PC can theoretically be equipped with up to 48 cameras. However, the frame rate will be lower then.
An adapted software by Fujifilm Switzerland is used to set-up cameras and record videos. The etho-hydraulic flume is illuminated with 7x1000 W halogen lamps (Figure 1). Calibration of the system is essential and made in three steps: finding intrinsic and extrinsic parameters for each of the five cameras using a checkboard, calibrating five stereo cameras according to the overlapping views of camera pairs, and finally performing a rigid transformation of all stereo camera pairs to a global flume coordinate system (Figure 3a, Detert et al., 2018).
3D fish tracking is based on the detection of moving fish in each frame and associating the detections corresponding to the same fish over time. These are done by using a background subtraction algorithm and a Karman filter in MATLAB (Detert et al., 2018). The primary results of motion-based tracking are tracks in a distorted and uncalibrated 2D image frame coordinate system for each camera. Figure 3b and c show the three detected fish and noises caused by reflections from the glass window and their 2D tracks over time. After undistorting such frames and stereo calibrating the cameras, the 2D fish tracks are transferred to a 3D metric-space according to their epipolar geometry based on the camera parameters derived from the calibration (Figure 4).
The etho-hydraulic tests were done for a flow depth of 90 cm, flume width of 150 cm, distance of 150 cm between the cameras and average flow velocities up to 0.7 m/s. Under such conditions, the 3D fish tracking system provided fish positions in 3D with an accuracy of about ±5 cm and 20 fps. The challenges for a successful implementation of the system are: assignment of individual fish to the tracks, constant illumination of the flow, camera distortion, air bubbles and suspended sediment and humid conditions for the cameras. Overall, despite some shortcomings such as noise due to reflections from the glass windows, the 3D fish tracking system works well, provides important information on fish behaviour affected by fish guidance structures and has the potential for further etho-hydraulic studies.
Relevant mitigation measures and test cases
Other information
The total costs of the present system is approx. 40’000 USD=35’000 € including camera set-up and recording software. For current costs of the equipment, we recommend to ask the corresponding supplier listed below. Note that a cheaper camera and lens set-up can significantly reduce the total cost of the system. The MATLAB-based 3D tracking code developed by VAW will be freely available.
Links to the suppliers of equipment:
Software for 3D fish tracking:
- Available on request.
Relevant literature
- BfG (German Federal Institute of Hydrology), 2018. The behaviour of fish in fishways – BfG and BAW’s ethohydraulic tests. Annual report of 2016/2017, pp.43. https://doi.org/10.5675/bfg-jahresbericht_2016/2017.
- Detert, M., Schütz, C., Czerny, R. (2018). Development and test of a 3D fish-tracking videometry system for an experimenal flume. In Proc. River Flow 2018 - Ninth International Conference on Fluvial Hydraulics, E3S Web of Conferences 40: 03018. https://doi.org/10.1051/e3sconf/20184003018
