Linking to an OLPC XO via vnc

This screenshot is taken from my desktop, running the Vinagre vnc client on Ubuntu Intrepid. I can move the XO's mouse cursor from the desktop PC, and hovering over menu items pops the menus open. However that is as much control as I can currently achieve. I cannot send keystrokes nor mouseclicks from the desktop PC to the XO, so I cannot control the XO from the desktop PC. It wold have been especially nice to be able to use teh desktop PCs keyboard. Perhaps I should forget the vnc approach, and just plug a USB keyboard directly into the XO.