According to the book: Practices of an agile developer, Design isn't left up to the user stories in a completely top down approach. Design is done a little bit at a time until the problem itself is well understood. However, they go on to say in the book that design shouldn't drive your code.

Ed was talking to Garrett about a component in the system, one that would act as the router in a complex communications platform. Looking at the code Garrett had written he was confused by the methods of the interface presented to him and so he asked "Why is this method there? what does that support?" to which garrett responded "It's in the design diagram, so i put it there".

Is this happening to you? Don't let your design ideas determine what code gets written.