To make the custard, whisk together the sugar, cornstarch, and salt in a medium saucepan. Add the milk slowly, and whisk until combined. Bring the mixture to a simmer over medium heat, stirring constantly to prevent the milk from scorching.
Once the mixture has reached a simmer, simmer for one minute, until thickened. Remove from heat and pour about 1/2 cup of the mixture into the egg yolks, quickly stirring to prevent the eggs from curdling. Pour the egg yolk mixture into the custard mixture, and stir to combine. Return the pot to the heat and bring the custard back to a simmer. Simmer for one minute, stirring constantly. Remove from heat and stir in the butter and vanilla until smooth.
Pour the custard into a bowl, cover tightly with plastic wrap, and refrigerate until cooled to room temperature.
To assemble the pie, slice three of the bananas into the cooled pie crust. Pour the custard over the top of the bananas, spreading evenly. Refrigerate for at least one hour, or up to overnight.
Just before serving, pipe or spread the whipped cream or Cool Whip on top of the pie, and garnish with slices of the remaining banana.