Preheat oven to 400°F.
Pierce sweet potatoes with a fork and bake for 45 minutes, or until soft. Let cool, scoop out the flesh, and mash until smooth.
Lower oven temperature to 350°F.
In a large bowl, cream the softened butter, granulated sugar, and brown sugar until light and fluffy (about 3-5 minutes).
Beat in the eggs one at a time, making sure each egg is fully incorporated before adding the next.
Add the mashed sweet potatoes and mix until well combined.
In a separate bowl, whisk together flour, baking soda, baking powder, salt, cinnamon, nutmeg, and ginger.
Gradually add the dry ingredients to the wet ingredients, alternating with milk, starting and ending with the dry ingredients. Mix until just combined.
Stir in the vanilla extract.
Grease and flour a bundt pan or a 9x5-inch loaf pan. Pour the batter into the prepared pan and smooth the top.
Bake at 350°F for 60-70 minutes, or until a toothpick inserted into the center comes out clean.
Let the cake cool in the pan for 10 minutes, then transfer to a wire rack to cool completely.