When there's an expectation that everyone will show up on time, people try their best to avoid being the one who holds up the group. Whether this is to start a meeting or to sit down for dinner, the dynamics are the same.
Unfortunately, once a group drifts away from this norm, people have a strong incentive to show up late. Since no one wants to rush to wait, people try to arrive after a critical mass has formed. But, because no one knows when that will be, people start arriving later and later. In response, organizers start building in buffer time when they send out invitations. But, as people get wise to this, they start to factor in the expected buffer time into when they should show up.
Don't let this spiral out of control. Show up on time.