If he didn't want to talk with you, he wouldn't leave his friends to talk with you. If you feel uncomfortable about taking him away from his friends, then introduce yourself to them or have him introduce you to them and make some small talk with them. Try to grow the connections with his friendship circle. Encourage him to also initiate contact with you by coming over and talking to you instead of waiting for you to make the first move. All that takes is to wave him on over or text him to come sit with you.