You could add fertiliser (organic ones such as juices made from stinging nettles, comfrey, seaweed and even banana skins) and change the top soil a bit 10 to 20%; not 50% though. You could also work in bonemeal, fish and blood. Don't do it over the winter though unless you still have active growth in your area. Do this in spring. Potted plants could be repotted every 2 to 3 years, with one size up pots. When the plant is already in a big pot, you could remove old soil from the top and add some enriched one. I have found that plants really like aged wood chips. You could tease out the matted root ball and work in a tiny amount of BBF with well aged bark mulch and good quality enriched compost for the bottom. This certainly won't harm the tree or plant. But use BBF sparingly.