When the smilies windows opens, you should have a resizing spot on the lower right hand corner.

It looks like this:

Click image for larger version. 

Name:	smilies.jpg 
Views:	5 
Size:	32.8 KB 
ID:	13924

Just grab those angled lines and drag to enlarge the smilies box.

For the screen expanded off to the right, I will have to try it on an XP machine with IE8. Never seen that problem before.