That could work. Removing the Panama company entirely could perhaps make for an even stronger case.
UBO (PA tax resident) owns foreign company. The foreign company engages in trade. The foreign company pays dividends to the UBO. Income is foreign-sourced, so UBO pays no tax.
As I said, you won't get a clear, definite answer to this. If you go by the letter of the law, Panama could maybe try to tax you. But would they? Seems very unlikely.︀ Just don't be too obvious about it and don't stir up any troubles in Panama.︁ Keep a low profile, be a good resident.