Sounds like you are right, and the problem is lack of bandwidth. Make sure you're using 16BPP, as 32BPP requires twice the bandwidth.
Something else you could try is to use a Router with QOS, there are several inexpensive routers that can be installed with 3rd party firmware that provide QoS (like
http://www.polarcloud.com/tomato). Then , make sure you give priority to VNC (TCP port 5901) over Web traffic.
If possible, you could also try reducing the resolution of your Mac.
Raf.