How much time should a puppy have out of his crate?

I work monday through friday, 8:30-5. I walk my puppy before work for around 20 minutes, when I get home for another 20, and again if he still has energy before I put him in his crate to sleep. My mom is able to let him out 2-3 times a day when I’m at work into the backyard to potty, and play for a little bit, but I feel bad since I only have the energy to have him out of his crate for 4-5 hours after work. Any suggestions? He does very well in his crate and doesn’t bark or whine, and is fully potty trained, but I can’t help feeling bad. I would love to have him out later to lay with me, but he just constantly tests boundaries and I can only handle it for so many hours after work. (I wasn’t sure what flair to use)