Reliability analysis and fair comparison performance of state-of-the-art Visual Inertial Odometry (VIO) methods integrated into the control pipeline of Unmanned Aerial Vehicles (UAVs) in real-world conditions is presented in this paper. Most VIO algorithms achieve excellent localization precision and negligible drift on artificially created datasets, but the aspects of reliability in non-ideal situations, robustness to degraded sensor data, and the effects of external disturbances and feedback control coupling are not well studied. These imperfections, which are inherently present in cases of real-world deployment of UAVs, negatively affect the ability of the selected state-of-the-art VIO approaches to output a sensible pose estimation. We identify the conditions that are critical for a reliable flight under VIO localization and propose workarounds and compensations for situations in which such conditions cannot be achieved. The performance of the studied methods is quantitatively analyzed w.r.t. RTK ground truth and the ability to provide reliable pose estimation for the feedback control is demonstrated onboard a UAV that is tracking dynamic trajectories under challenging illumination.