If your monitor has an HDMI connector, then it will display in HD, if the screen is small you won't notice the differences too much.
You can also use a HDMI to DVI adapter: PS3 HDMI to DVI cable: Amazon.co.uk: Electronics I'm not sure what resolutions the PS3 will output, however if your monitor can do 1280x720 then you should be fine for playing in 720p
A 17" screen does around 1024x768 or a little more not sure, it may not be HD but it'll look better than on an SDTV.
The PS3 is displaying a screen resolution your monitor does not support. I dont own a PS3 however this should be how you set the screen resolution on your PS3: PS3™ | Video Output Settings You want to set the PS3 to use HDMI if you have not: Tell it that your TV can only do 480p And finally set your display to 4:3 as it is not widescreen The PS3 should display an image when connected to your monitor then.