With the development of distant-water fisheries, ship fishing and fish catch detection are now vital to modern fishing. Existing manual detection methods are prone to issues such as missed detections and false detections. Deep learning has enabled the deployment of detection models on shipboard devices, offering a new solution. However, many existing models are hindered by large parameters and computational complexity, making them unsuitable for shipboard use due to limited resources and costs onboard ships. To address these challenges, we propose the RCT-YOLOv8 model for tuna catch detection in this paper. Specifically, we adopt YOLOv8 as the base model and replace the network backbone with RepVGG network, which employs re-parameterized convolutions to enhance detection accuracy. Additionally, we incorporate coordinate attention at the end of the backbone to better aggregate channel-wise information. In the neck part, we introduce the contextual transformer (CoT) attention and propose the C2F-CoT model, which combines convolutional neural network with Transformer to capture global features, thereby improving detection accuracy and the effectiveness of feature propagation. We test multiple loss functions and select efficient intersection over union, which is more suitable for our algorithm. Furthermore, to adapt to devices with limited computational resources, we utilize the dependency-graph-based pruning method to compress the network model. Compared to the base network, the pruned model achieves a 9.8% increase in detection accuracy while reducing parameters and computational complexity by 40% and 35.8%, respectively. Compared to various algorithms, the pruned model demonstrates the highest detection accuracy, lowest parameter count, and lowest computational complexity, achieving optimal results at all fronts.