Next time you see him ask if he would like to go get a cup of coffee or something after he gets off work. If he likes you he will go, or he will make some other arrangments with you.
Youre worring way too much
Dont be shy guys like when women hit on them, if he doesnt take you up on it then move on... .His Lose:)