In World War II, Moriyama's father was arrested for refusing to go into internment, while Raymond and his mother were held at Slocan, British Columbia. Another illustrious Japanese Canadian, environmentalist David Suzuki, was also interned at Slocan as a boy.
According to The Globe and Mail columnist, Moriyama showed his penchant for building at an early age:
Though it was forbidden at the internment camp, the young Raymond scavenged an axe, some nails and some scrap wood and built a rough, rhomboid-shaped tree house at the edge of the Slocan River. It was a place of quiet refuge and healing. “The view of nature from the tree house was absolutely astonishing,” he recalled in his speech Saturday night. “The mountains, green and silver, around the river; the whisper of the river and the sounds of night; the crisp night sky and the stars so close.”