I visited the shrine many times, but only once I was lucky to visit it on rain... Why lucky? Because, while it is indeed more difficult to take pictures while raining (you need to take care of the camera, etc.), the pictures will look a lot better than anything you can catch on good weather:
EXIF info:
Nikon D90
Lens: VR 18-55mm F/3.5-5.6G
Focal Length: 26mm
Aperture: F/5.6
Shutter Speed: 1/30s
ISO Sensitivity: ISO 500