In the OAV, it was abandoned and quite delapidated. Belldandy performed a singing spell to make the temple remember its former glory and rebuild itself.
One interesting thing about the Temple is that in all of the continuities so far, we've never seen the inside of the temple (other than the individual rooms). We don't know what specific kind of temple it is, what ceremonies are performed, or other things. Also, despite being run by a monk formerly in the manga and TV Series, we don't even know why people don't still visit for their religious rites or offerings! This is a question that may never be answered, but perhaps we need to suspend our disbelief.
The only clue to the Temple's origin is that it is either of a Buddhist or Shinto origin (The Manga, OAV, and TV Series all differ slightly on this), but since the temple appears to have both Shinto and Buddhist traits, one can not be entirely sure. In the TV series and the Movie, we see the main inside of the temple (Which is at the very start of the Manga, right when Belldandy and Keiichi arrive there for the first time), and in the TV series there is a large statue of a Jizo (Protective Statue); but in the movie, the Statue appears to have been removed (Or moved to another location) at some point previously.