Don't log exceptions when failing to fetch remote server keys