Reinforcement Learning

Controlled gliding is one of the most energetically efficient modes of transportation for natural and human powered fliers. Here we demonstrate that gliding and landing strategies with different optimality criteria can be identified through deep-reinforcement-learning without explicit knowledge of the underlying physics. We combine a two-dimensional model of a controlled elliptical body with deep-reinforcement-learning (D-RL) to achieve gliding with either minimum energy expenditure, or fastest time of arrival, at a predetermined location. In both cases the gliding trajectories are smooth, although energy/time optimal strategies are distinguished by small/high frequency actuations. We examine the effects of the ellipse’s shape and weight on the optimal policies for controlled gliding. We find that the model-free reinforcement learning leads to more robust gliding than model-based optimal control strategies with a modest additional computational cost. We also demonstrate that the gliders with D-RL can generalize their strategies to reach the target location from previously unseen starting positions. The model-free character and robustness of D-RL suggests a promising framework for developing robotic devices capable of exploiting complex flow environments.

Proceedings of the 36th International Conference on Machine Learning Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors and can undermine the performance of ER. Many algorithms mitigate this issue by tuning hyper-parameters to slow down policy changes. An alternative is to actively enforce the similarity between policy and the experiences in the replay memory. We introduce Remember and Forget Experience Replay (ReF-ER), a novel method that can enhance RL algorithms with parameterized policies. ReF-ER (1) skips gradients computed from experiences that are too unlikely with the current policy and (2) regulates policy changes within a trust region of the replayed behaviors. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods. We find that ReF-ER consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.

The coordinated motion by multiple swimmers is a fundamental component in fish schooling. The flow field induced by the motion of each self-propelled swimmer implies non-linear hydrodynamic interactions among the members of a group. How do swimmers compensate for such hydrodynamic interactions in coordinated patterns? We provide an answer to this riddle though simulations of two, self-propelled, fish-like bodies that employ a learning algorithm to synchronise their swimming patterns. We distinguish between learned motion patterns and the commonly used a-priori specified movements, that are imposed on the swimmers without feedback from their hydrodynamic interactions. First, we demonstrate that two rigid bodies executing pre-specified motions, with an alternating leader and follower, can result in substantial drag-reduction and intermittent thrust generation. In turn, we study two self-propelled swimmers arranged in a leader-follower configuration, with a-priori specified body-deformations. These two self-propelled swimmers do not sustain their tandem configuration. The follower experiences either an increase or decrease in swimming speed, depending on the initial conditions, while the swimming of the leader remains largely unaffected. This indicates that a-priori specified patterns are not sufficient to sustain synchronised swimming. We then examine a tandem of swimmers where the leader has a steady gait and the follower learns to synchronize its motion, to overcome the forces induced by the leader’s vortex wake. The follower employs reinforcement learning to adapt its swimming-kinematics so as to minimize its lateral deviations from the leader’s path. Swimming in such a sustained synchronised tandem yields up to 30% reduction in energy expenditure for the follower, in addition to a 20% increase in its swimming-efficiency. The present results show that two self-propelled swimmers can be synchronised by adapting their motion patterns to compensate for flow-structure interactions. Moreover, swimmers can exploit the vortical structures of their flow field so that synchronised swimming is energetically beneficial.