This phase I trial is studying the side effects and best dose of temsirolimus when given together with hormone therapy in treating patients with relapsed prostate cancer. Androgens can cause the growth of prostate cancer cells. Hormone therapy may fight prostate cancer by lowering the amount of androgens the body makes. Temsirolimus may stop the growth of tumor cells by blocking some of the enzymes needed for cell growth. Giving hormone therapy together with temsirolimus may kill more tumor cells