RAYD, I don't know your tune settings or boost pressure, but start with the cap turned in one turn from fully out [full out is 6 turns from the full in position and should expose the red (or in some cases black) limit line under the rim of the cap]. Test idle and run the car. If you have any backfires, progressively screw it in until regular backfiring ceases. At low pressures on 50:50, you will hear the BOV open with a quiet Prrrp sound as it sends air to bypass, while lifting of throttle at at higher boost pressures will cause the piston to rise above the bypass port and discharge the excess air to bypass AND to atmosphere, much as you describe. I'm guessing that 3 turns in might do it for you. Be patient. Do it over several days, driving in all types of situations, and increase turns by 1/4 turn each time until it all seems to be working well and there is minimal turbo lag (too little spring pressure will release more air than you need to and this will mean the turbo has to refill the plumbing again, creating a perceptible lag). But don't overtighten the cap as you will not get the bypass or blow-off functioning properly and the BOV will not do what it is designed to do - that is, protect the turbo from stalling on throttle lift-off. Oh, I forgot - ALSO - READ THE INSTRUCTIONS IT COMES WITH!
CP_e Standback & PNP; CP_e 3" SS Downpipe; Corksport FMIC with Top-mount K&N filter & OEM Ram CAI; Turbosmart BOV;Dashhawk;Prosport Boost Guage;Sumitomo HTRZIII's in 225/45 x 18