Also the fact that Panem was the last human bastion. If there were other areas with civilization, then the whole idea would fail.
Also, the vast differences in technology and general social evolution would not work. Perhaps assigning different universes to these worlds would be more appropriate. Although, according to multiple universe theory, there already are separate universes with those worlds.