On the Apple TV Remote on iPhone, you can use buttons instead of swipe gestures.
Go to Settings Accessibility features for mobility on iPhone