This paper proposed a novel projector-camera based human-computer interaction system for touch-pad application of tabletop projection. Taking the advantage of differential shadow in high-frequency projection, the system realized the localization and judgment of touch action without influencing users’ visual experience. A Markov Random Field based shadow segmentation method helped to robustly extract the low-intensity shadow under complex environment. To precisely locate fingertip, the algorithm searched along the contour of the shadow and used a least square method to estimate the curvature of the shadow contour. Furthermore, the binocular constraint and geometrical model based touch judging method helped to raise the touch judging precision in the system. The experiment showed that the proposed system performs better than state-of-art works, with touch recognition rate at 99.2%, false alert rate at 0.6% and touch localization accuracy at 1.77pixels. Meanwhile, the system works at 60 fps on a conventional PC, which satisfies the requirement of most of real-time applications.