Trouble in Packet City
Latency's ugly head can pop up when it takes awhile for your recipient to hear what you have just said. Combined with a lack of visual cues available in ordinary conversation, one party or the other winds up interrupting the other's words.
Dropped packets (the way the bits and Bytes of your voice are sent over the Internet) are inevitable, but the more dropped packets, the greater the chance of your voice getting garbled, or having entire pieces of conversation chopped out.
So while latency and dropped packets are gremlins, what's the point at which they become a problem?
Engineers say that 1-2% packet loss is somewhat acceptable. Think about it this way: typically, about 50 packets travel with each second of VoIP. So if 49 out of those 50 get through, it would be equivalent to clearly hearing 98% of what I say to you if we were talking in person. When you get into the higher single digits, packet loss becomes more than a bit unacceptable.
As for latency, 20 milliseconds is pretty good, but once you get in the 100 milliseconds (1/10 of a second) range, you may get into the interruption zone.
Another important factor - and one that your ISP doesn't like to talk about - is your relatively puny upstream bandwidth. The typical home Internet connection may let you download files and web surf with a reasonably-sized 1Mbps-or-greater bandwidth. But your upstream (from your computer to the Internet) connection typically is throttled down to as little as 128Mbps or one-tenth the downstream capacity. So you might find your VoIP calls negatively affected by the Internet activity of other users in your household (or vice-versa).