Reword the text.
By the time he thought seriously about leaving, it was not a good time of year to travel northwards.
A trip to the North had not been the most convenient time for him at the moment when he began seriously considering his departure.