As well as including Kim Yong Nam, the North's high-level delegation to the South for the Olympics will include Kim Jong Un's sister Kim Yo Jong, who appears to be an increasingly powerful and influential figure.
South Korean tours to the Mt Kumgang resort were closed after a South Korean tourist was shot by a North Korean guard there in 2008.
The ministry said it would discuss reopening the park, "when the North Korean nuclear issue has to some extent entered a phase of a resolution".
"As long as imperialism remains on the Earth and as long as the United States' hostile policy on North Korea continues, the mission of our military, which is to protect our country and people and secure peace, can never change", Kim said.
Pence, who arrived in Japan on Tuesday to meet Japanese Prime Minister, Shinzo Abe, is traveling to South Korea Thursday to attend the Pyeongchang Winter Olympics opening ceremony, just 80 km (50 miles) from the heavily armed border with North Korea.
North Korea has 22 athletes who will compete in the worldwide sports competition, the two Koreas will walk together under a neutral flag at the opening ceremony.
Kim Jong Un and his generals watched the parade from seats above the procession route through the capital.
In this image made from video by North Korea's KRT, a military parade is held in Pyongyang, North Korea Thursday, Feb. 8, 2018.More news: Trump's idea for a military parade draws support, questions
Pence will meet Friday morning with North Korean defectors as he pays respects at the Cheonan Memorial in Seoul, which honors 46 South Korean sailors killed in a 2010 boat sinking attributed to the North. One of Kim's close aides, Choe Ryong Hae, and Kim Yong Nam were also in attendance. "Both Korea and China need to work toward the same goal to encourage the US and North Korea to hold dialogue".
Japan's Kyodo news agency reported earlier that on Thursday, a large amount of military equipment had been detected on Kim Il Sung Square in downtown Pyongyang, where military parades usually take place.
Abe echoed Pence's comments through a translator, urging countries "not to be lured by the charm offensive of North Korea".
The North had insisted on the event, despite criticism from the USA and others about its timing - a day before the opening ceremony of the Winter Olympics, which run from 9 to 25 February in the mountain town of Pyeongchang.
Kim did not address the Olympics in his speech, which, according to the edited footage released by North Korean state television after the parade, lasted about 15 minutes-including applause breaks and a brief appearance by a woman, possibly his sister Kim Yo Jong, from behind a column.
North Korea is under multiple sets of UN Security Council sanctions over its banned nuclear weapons and ballistic missile programmes, which have seen it develop rockets capable of reaching the United States mainland.
Almost 300 North Koreans crossed the Demilitarized Zone between the two countries on Wednesday to participate in the Olympics, including 229 women that were part of a large "cheering squad", The Washington Post reported.