This paper explores the design and development of a class of robust diver detection algorithms for autonomous diver-following applications. By considering the operational challenges for underwater visual tracking in diverse real-world settings, we formulate a set of desired features of a generic diver-following algorithm. We attempt to accommodate these features and maximize general tracking performance by exploiting the state-of-the-art deep object detection models. We fine tune the building blocks of these models with a goal of balancing the tradeoff between robustness and efficiency in an on-board setting under real-time constraints. Subsequently, we design an architecturally simple convolutional neural network based diver detection model that is much faster than the state-of-the-art deep models yet provides comparable detection performances. In addition, we validate the performance and effectiveness of the proposed model through a number of diver-following experiments in closed-water and open-water environments.