About 538 AD, Buddhism was introduced to Japan through China and the Korean Peninsula. With the introduction of Buddhism, temples were built in various parts of Japan.
The technology of manufacturing and constructing stucco was also introduced to Japan. Temples with white walls are typically used lime plaster (sikkui) for their finishing.

 

Following the link, you will find the website maintained by Japanese Plaster Association and it has comprehensive explanation of the stucco plaster used in Japan.  (Japanese only)