5 month old doesn’t potty outside

hi,

so i have a 5 month old male lab. Up until now he was potty trained with pee pads, because he didn't have all his vaccines. He has been potty trained like this for a month.

Finally, we are able to go outside and we have been doing so for almost a week. He very quickly got over his fear of stairs (we live in an apartment complex on the 4th floor, we are very proud he beat the stairs).

He is also pretty well behaved on walks and he doesn't pull much, but he refuses to go potty outside.

We tried taking him out in the moments he usually goes potty at home, like right after he wakes up or some time after he eats, sleeps or plays. Nothing. We tried bringing a pad outside, he completely ignores it. He goes potty as soon as he comes back home.

We then decided to remove the pads and supervise him very closely at home, take him out every hour or so or take him out whenever he seems to want to go potty. Nothing.

Yesterday he held himself for 11 hours, completely refusing to potty, and i know there's nothing wrong with him, because as soon as we took the leash off of him to go to sleep, he immediately went potty. It seems to me that there's some anxiety that just won't let him potty outside, which is understandable, because even though our neighborhood is not too noisy, occasionally there are cars or birds (birds interest him a lot) and sometimes even dogs.

Last night we were UNFORTUNATE enough to run into an off leash Shitzu that came and tried to aggravate our puppy, with the most unresponsible owner barely mouthing the words "Come back". Our lab was understandably upset and kept barking back at the other dog.

Also yes, we do own a crate, which we put him in after walks to calm down. It is also important to mention that i do occasionally work for home, 2 weeks per month to be specific, but with the amount of work i have it's not possible to take him out every hour. Only in the morning, at 12pm, after work and at night before sleep.

We have absolutely no idea what else to do.