It seems to me the real difficulty with chastity - especially for those who have been sexually abused or have abused the gift of sexuality itself, may come down to a lack of charity. The Catechism defines the virtue of charity this way:
1822 Charity is the theological virtue by which we love God above all things for his own sake, and our neighbor as ourselves for the love of God.
1824 Fruit of the Spirit and fullness of the Law, charity keeps the commandments of God and his Christ: "Abide in my love. If you keep my commandments, you will abide in my love. - CCC
Chastity is difficult when our affections are dissipated, inordinate, disordered, and all over the place.
One may be celibate and remain unchaste. So celibacy isn't the goal.
Chastity frees us, open us, expands all our faculties to love God with our whole being, and our neighbor as our self.
[Lust and attraction are not synonymous either.]
Pier Vittorio Tondelli came to understand chastity as a “a mystic virtue for those who have chosen it and perhaps the most superhuman use of sexuality.”
It seems to me that celibacy is more a condition of single life. All are called to chastity. In contemporary understanding, chaste single life pretty much equals celibacy. Religious life does too - yet ordinarily, religious do not make a vow to be celibate, but to be chaste. To love God with their entire being, otherwise celibacy doesn't mean a great deal.
Just like love, chastity is misunderstood and 'not loved' in our culture. It's not a curse. Tondelli suggests it is chosen - on some level that is true - we can eat of the tree of forbidden fruit, or not. We have free will. More deeply, it is a grace, a gift, a valuable pearl, that one needs to sell everything to obtain. Like love, it requires sacrifice.
"Love is a teacher, but one must know how to acquire it, for it is hard to acquire, it is dearly bought, it is won slowly by long labor. For we must love not occasionally, for a moment, but forever." - Dostoevsky, Fr. Zosima