The problem with QOS is once you're on the public network the QOS is gone. So the best you can do is internally have your network prioritize the traffic. That said, assuming you use cable, upstream is your limited bandwidth and you can make some gains here. I'd suggest one of two approaches, of which I do the later.

1)IF your router supports priority by ports then set all ports to normal, SIP(udp/5060) to one notch above, and RTP(some range) to highest. Now the problem is your softphone needs to let you configure an port range for RTP and it also needs to use the same socket for inbound and outbound. Such that the UDP out has the from port the same as your QOS range.

2)If your softphone/router lets use COS or DSCP then configuring the tagging and priority that way. tag signalling medium and rtp above that. This is ideal esp if you have a more complex network then just a router and switch. Also some Access points will use DSCP or COS to prioritize as well

In the end all you will able to do is prioritize outbound traffic. Inbound traffic is trickier and you'd need a router smart enough to associate traffic because the inbound traffic will be all over the place in terms of ports and sources.