When to move crate downstairs?

So I’ve been crate training my 3 month old Labrador and he goes down for his naps no problem and sleeps for around 2 hours when downstairs where he isn’t bothered, however when it’s night time I’ll move the crate to my room he’ll sleep for a while but around 4 am he starts barking and won’t stop no matter what I do even if he goes to the toilet, has food and water, we don’t play because I don’t want to reinforce the idea that 4am is play time, he won’t calm back down if he hears the slightest noise from me, should I move the crate downstairs full time? I’m just scared if I do he’ll start barking in the middle of the night and panicking and I won’t hear him as I’m a heavy sleeper usually and I don’t want to ruin crate training

What should I do?