The first thing for you to do is to meet with a credible singing teacher for professional feedback. A good vocal coach will concentrate on your strengths and once you know what they are your confidence will grow. Also, who are these people that condemn your singing? Are they masters of the voice? Are they professionals? Pro's would never say bad things about your voice. Never!
Start focusing on all your good qualities, all of the nice things about you. Are you patient, a good listener, loving, friendly? Recognize all the good about you which will help with criticism. Above all, don't react to the negative comments. People that are critical have a problem. Love yourself!