1) have you updated the play engine to the latest version?
2) check your system resources utilization (task manager): how much ram is free, and what, if any, are other big processes that you can stop?
3) you may need to freeze tracks and painstakingly work them one at at time, freezing and unfreezing, if you're getting the issue. Try freezing all the tracks except for one of the play VST's, and check if it works then.
4) just for good measure, disable any active antivirus scanning and hard drive indexing while doing work on the project.
Also, are all four sticks of ram installed but only three are showing up? You're on Windows 32-bit or 64-bit?
Edit: oh, and before I forget, unless you're using the 64-bit version of Windows, the 64-bit version of Reaper, and the 64-bit version of Play, I believe Reaper and/or Play is limited to 2GB ram usage total.