I usually get rid of them when they are brown and dry.
From homegudies
After blooming, the foliage begins to yellow and fade. It might be tempting to cut this unkempt foliage back, but resist the urge. Like many bulbs, lilies get their energy for the next growing season from these leaves. If you cut them back prematurely, next year's growth and flowers might be diminished. Plant other flowering and mounding plants around lilies to hide their ungainly foliage. Cut the foliage back in late fall or early spring after it has completely died back.
http://homeguides.sfgate.com/can-cut-lilies-down-after-bloom-69115.html
So after flowering, just leaves everything on, but get rid of the flowers which require energy to form seeds. Get rid of the dead blooms, but leave the leaves to get energy for next year's growth. When the stems dry and become brown, just chop them off.
