A few patches ago, before the games population died down, we would have 20+ people online with very few performance issues. For awhile now, it is difficult to have 10+ people on without dropping the server FPS from 15 down to 5-8.
Some theories involved the way buildings were made or the sheer amount of buildings was the cause. Like clockwork, soon as the population dropped, the FPS improved.
I can say with almost 100% confidence that I've figured out the cause. NPC AI. It is fairly easy to test, to which I've done. I started to realize something when the FPS dropped to 7 with only 2 of us online. However, the second guy is in a base with (by his count) 100+ combatant thralls. When they engage an enemy, the FPS goes to hell.
It isn't just thralls. If you get into an area where a lot of NPCs engage, you'll notice a drop as well. When I farm Set City for thralls, I'll be chased by 15 or so and see the FPS drop.
This is why the higher population seemed to be the cause. Those are extra people causing more NPCs to spawn and activate their AI.
As a test to prove to naysayers, I spawned 20-30 archers. Soon as they started attacking me, the FPS went from 14 down to 2. The ping went from 140 to 900+. I killed them and it returned to normal. You can do this over and over and still has the same effect.
When we thought it was the hardware on the server just not keeping up and the location of the server, we had it moved and upgraded. It is located in NYC running 4ghz Xeon SSD. I find it hard to believe this issue is anything other than poor AI coding optimization. Especially since a few patches have "improved" said AI.
We've lost several players due to this lag. Please take a look into it.