The description of the Armor of Agathys spell says (PHB 215, emphasis mine):
A protective magical force surrounds you, manifesting as a spectral frost that covers you and your gear. You gain 5 temporary hit points for the duration. If a creature hits you with a melee attack while you have these hit points, the creature takes 5 cold damage.
The Grappling rule states (PHB 195, emphasis mine):
When you want to grab a creature or wrestle with it, you can use the Attack action to make a special melee attack, a grapple. If you’re able to make multiple attacks with the Attack action, this attack replaces one of them.
The target of your grapple must be no more than one size larger than you and must be within your reach. Using at least one free hand, you try to seize the target by making a grapple check instead of an attack roll: a Strength (Athletics) check contested by the target’s Strength (Athletics) or Dexterity (Acrobatics) check (the target chooses the ability to use). You succeed automatically if the target is incapacitated. If you succeed, you subject the target to the grappled condition. The condition specifies the things that end it, and you can release the target whenever you like (no action required).
I believe that grapple counts as a melee attack that activates Armor of Agathys. Is this assumption correct?
If so, what happens on consecutive rounds of grappling? Assuming the attempt to escape has failed, and Armor of Agathys can still cause damage to opponents who hit it with melee attacks, does Armor of Agathys inflict more damage to the grappler on consecutive rounds of being grappled? Or is it just the initial grapple attempt that activates the spell and causes damage?
Armor of Agathys doesn’t affect grappling
Grappling isn’t ‘hitting with an attack’ which is what triggers Armor of Agathys, it’s using an ability check, and ability checks aren’t “hitting with an attack”. If you’re ever in doubt, the best place to look is the rules for what is an attack:
If there’s ever any question whether something you’re doing counts as an attack, the rule is simple: if you’re making an attack roll, you’re making an attack.
You are not hitting with an attack roll, despite the fact that the section calls it a ‘special melee attack’. The same rule applies for the other ‘special melee attack’, the shove. That too states you make a special melee attack, but that doesn’t actually involve an attack roll either so won’t trigger Armor of Agathys either. Neither involves ‘hitting with a melee attack’.
Grapple and shove are weird ones because they occur during the ‘Attack’ action, but you replace an actual attack with an ability check.
See also the Sage Advice on the matter:
When you make a Strength (Athletics) check to grapple or shove someone, are you making an attack roll? Again, the answer is no. That check is an ability check, so game effects tied to attack rolls don’t apply to it. Going back to an earlier question, the hex spell could be used to diminish a grappler’s effectiveness. And if the grappler’s target is under the effect of the Dodge action, that action doesn’t inhibit the grapple, since Dodge doesn’t affect ability checks.
This is also why Wrath of the Storm and Riposte do not trigger on grapple attempts, because they also do not involve hitting with an attack, as clarified in this sage advice.
Does a grapple or a shove trigger the Tempest cleric’s Wrath of the Storm or a Battle Master’s Riposte? The answer to both questions is no. The grappling and shoving options (PH, 195) don’t result in a hit or a miss.