When the smilies windows opens, you should have a resizing spot on the lower right hand corner.
It looks like this:
Just grab those angled lines and drag to enlarge the smilies box.
For the screen expanded off to the right, I will have to try it on an XP machine with IE8. Never seen that problem before.












Bookmarks