Well from the sounds of it you are tax resident in country B then (does not need to be even with 80%)
Just an office is usually not enough to not make country B declare your company as a country B resident / taxable company if you hold the majority of shares. You will need to check your country B CFC / foreign income / management whatever laws.
Lets suppose you pass country Bs substance tests for those and it it is ok with company being a country A company and being taxed in country A then you should probably check the double tax agreement︀ between your two countries. They will have a dividend / passive income clause that will︁ tell you which country is allowed to tax you in which way on those dividends.︂ You might get taxed in both countries but country B might give you a tax︃ credit for the 5% paid in country A. It is very unlikely (unless we are︄ talking Malta, Cyprus etc) that you are not going to pay proper dividend taxes in︅ country B with country B's dividend tax rate.
The double tax agreement will also have︆ clause about what constitutes proper local offices / substance tests in which country and what︇ makes which country be the tax receiver for said company or local office / subsidiary︈ of said company.
Mind you this is all very dangerous private knowledge from having to︉ deal with it myself so just a starter for you to know where to look.︊ I don't understand a lot of those parts correctly myself. Final word should come from︋ a tax lawyer - i would always advise two independent lawyers.
