In a medium saucepan, melt butter. Stir in sugar, milk and cocoa until combined.
Bring mixture to a rolling boil and hold for 1 minute, then remove from heat.
Working quickly, stir in peanut butter until melted and well-combined. Add vanilla, then mix in oats one cup at a time, followed by the graham cracker cereal. Stir in the marshmallows until just melty (they will continue to melt as you make the cookies).
Drop tablespoons of batter onto the prepared baking sheet. Garnish with additional marshmallows and Golden Grahams if desired.
Let cool entirely before eating. Store in an airtight container for up to 3 days.