You are correct, to some degree we are controlled by the margin size. I am not certain how big we could go.
But here are some samples for sake of comparison.
My avatar is 100x88. The current size is 100x100, so your Avatars are going to be somewhere around there.
We could take it up to 120x120 without trouble I think.
I am not sure we could go as big as 150x150
Here are some different sizes (the image will degrade as I make it bigger. That is something to keep in mind in regard to ease of finding an avatar.)
My current avatar 100x88
My current avatar at 110x96
At 120x104
At 130x112
At 150x129
