No strong reason to limit transfers to/from the hardware wallet.
For incoming transfers, you just make sure to never reuse an address. You can generate an unlimited number of addresses for free, and reusing addresses is considered bad practice. I think reuse is only acceptable if the payment comes from the same source, but even then it's not smart. Addresses were never designed to be re-used.︀ All modern wallets consolidate all your address balances for you, so you just see 1︁ combined balance.
For outgoing transfers, you can first coinjoin the coins so nobody even knows︂ how much you had at that one isolated address. Note though that some exchanges don't︃ want to grant you financial privacy and freeze coinjoined funds, so you might want to︄ look at the mechanism employed by Samourai Ricochet. Basically, it emulates a few "normal" transactions︅ after the coinjoin, so the recipient doesn't know if you coinjoined yourself or someone who︆ owned the coins before you got them.
I suggest a wallet like Wasabi Wallet. The︇ "beginner" wallets don't give you enough control. E.g., some wallets don't show you which address︈ you are sending coins from (makes "coin control" impossible).
PS: Some authoritarian governments might consider︉ above practices to lie in the gray area or even be illegal. Please do your︊ research.
