What does the rest of your yard look like? I ask that, because I'm wondering what other possible food source(s) you have for those insects. I'm also wondering if you have ID'ed the insect(s) doing the damage?
I also would not "weed" that bed, rather I'd heavily mulch it with something like leaves and/or whatever you can find.
I had a similar problem during last winter where a bunch of harlequin bugs were demolishing my broccoli plants, but only the ones that were planted in not-so-great soil, all the others they left alone, even only being in very close proximity.
I added kitchen scraps around the base and heavily mulched that plant and not long after they harlequin bugs left the broccoli and jumped on a volunteer Florida Broadleaf mustard plant that was only a foot away and the broccoli plant fully recovered -- I didn't expect such a quick turn around.
I still, as of right now, have tons of harlequin bugs in my yard, but they seem to be concentrating on my 4o'clock plants, which is fine by me, since those plants reseed themselves all over the place.
Working with nature works.