Multi-objective optimization of video coding and transcoding