Place potatoes in pot, cover with cold water, and heat over high heat until boiling. Reduce heat to medium and cook about 15 minutes, until tender. Set colander over a bowl and drain the potatoes. Set aside about ½ cup of the water and discard the rest. Rice or mash the potatoes in the now empty bowl, and add salt.
Meanwhile, process the garlic, nuts, and oil into a smooth sauce.
Add sauce to potatoes with lemon juice, oregano, and a couple T of the cooking water. Stir well, adding more water as needed until you have a thick, somewhat stiff sip. Taste and adjust seasonings.
Spread the skordalia into the bowl, drizzle with oil, and garnish with pepper and olives.
Serve warm or at room temperature.