No removal necessary. In my garden I just use black kow composted manure, but to start lay the bags so they touch and cover the whole area. I tried regular sand once. It made the dirt hard so it got moved to bad spots in the lawn and replaced with manure compost. A lot more than I would have thought over a 2 year period, and its is a hillrow for drainage reasons. The problem with clay is root suffocation and poor percolation. While a variety of things work to different degrees, a raised bed rather than a sunken hole is superior when faced with a variety of plants many of which having roots that hate clay wetness and density levels. The winners tend to be plants that grow roots on top of the soil unless you dig in enough amendment and\or create a drainage like a pot with a gravel layer. The hole you dig into clay will also hold water, just deeper. Here and because we have acidic clay with a 5pH, I could see digging a hole at least as deep as a shovel and placing a couple inches of marble chips as a base drainage layer ( calcium carbonate) with perlite on the marble layer. Then on top I would mix the soil and compost in a 1:1. I would have a raised bed border of at least 6 or 8 inches. Even in the wettest winter\spring early summer the roots will be happy. In the garden hillrow system I am using a tiller, so a stone \ glass drainage layer does not make sense, but for a one shot sunken raised bed setup I could see it being worth the effort to dig. Digging is hard work though, so while planning is important my back tells me to never dig twice!
edit to say a low raised bed with sloping pinestraw or mulch edge looks nice actually. It may not require a walled edge, as the first season goes and the bed settles that shrinkage from 8 inches high may well turn into a 4 inch height.