You are not ‘forced’ into battling arena if you do not want “prizes or prestige”. There are FB where you can battle all you want and everything is ‘fair’ there.
And if you have been battling opponents who are consistently larger, that is because you are in the trophy range in which they live. If you are not dropping a lot of trophies and losing them them constantly, that is the range, and the levels of opponents, you are capable of battling against. If they are that much larger and stronger, you will naturally fall in trophies. If you stay up there, congratulations.