Preheat the oven to 350°F (180°C) grease an 8×8-inch baking pan or line with parchment paper. Set aside.
In a medium bowl, whisk together the flour, baking powder, baking soda, cinnamon, ginger, nutmeg, cardamom, allspice, pepper, and salt until combined. Set aside.
In the bowl of a stand mixer fitted with the paddle attachment, or using a handheld mixer, beat together the butter and sugars until light and fluffy. Add the egg, egg yolk, and vanilla and beat until well combined. Then add in the mashed banana and mix until fully incorporated. Don't worry if the mixture looks slightly curdled at this stage!
With the mixer running on low, add the dry ingredients in three additions alternating with the buttermilk, and mix each addition until just incorporated. Be careful not to overmix! The batter will be slightly thick, and a few lumps are okay. Stop the mixer occasionally to scrape down the sides of the bowl.
Pour the batter into the prepared pan, and smooth it out evenly with a spatula. Bake the cake for 28-30 minutes or until a toothpick inserted comes out clean and relatively crumb-free. Remove the cake from the oven and let it cool completely to room temperature.
Frost cooled cake with cream cheese frosting. Cover leftover cake tightly and store in the refrigerator for up to 5 days.