You're learning some hard lessons here, but well worthwhile. Game Design Documents are there to keep you in scope and keep the project going in a direction, which is super useful when you're hip deep in the minutiae of doing things.

With that, you can still use agile processes, but you really have to get the "what am I actually doing" part down first. 

Go do the new project. Get some distance, and come back to ComCreatures with fresh eyes, and then look at what you've done, what you need to make it feature complete, and then you can iterate with purpose.

You got this.