Saudi diplomat flies back, activists upset