You can use salicylic acid to control your oily skin. This can dry up most of the oil so that it doesn't have time to set and cause blemishes.
Make sure that the kind you're using is oil-free, otherwise you'll be adding more oil to your face.