Solved this by getting the full contact object from the contact/list endpoint, capturing the listid parameter, and adding the p[ID] = ID to the $contact before hitting contact/edit. This post was a hint.
It’s strange as I was not updating any custom fields.
Hopefully this helps someone else.