Warm milk and sugar in a small saucepan, stirring until the sugar dissolves. Remove from heat and cool completely. Chill while preparing peaches
Place peaches and lemon juice in the bowl of a food processor and pulse until fairly smooth, but some chunks remain. Transfer mixture to a bowl and stir in cooled milk and buttermilk aside. Cover and refrigerate 30 minutes.
Freeze mixture in an ice cream maker according to the manufacturers’ instructions, until thick and creamy. Transfer to an airtight container and keep frozen for up to 1 week.