For me, the worry is still relegation this season. Recent results against other poor teams have put us up a few places but it is still a possibility.
If we lose two good players (and no guarantee of adequate replacements - see all the failed bids in summer for midfielders), then my worry would rise.