It's probably normal. The recycle time depends on how much flash power was used in the previous flash. If you're just using the flash for fill work or if your subject is close, or if you're shooting at wide apertures the internal flash may not use up all of its charge when it fires. Since it has some charge left, it takes less time to fully recharge.
On the other hand if you're using a slow lens or if your subject is further from the camera and you're using the flash as the main illumination source, then you may use all the flash charge each time, so it takes longer to recharge.
Recharge time may also increase a bit if the battery charge is low. I think the full recharge time for a 40D with a fully charged bettery is something around 3 seconds, so I wouldn't be surprised at 4 or 5 seconds if the battery is low.
If you want a faster recycle time (and more flash power) the answer is a small external speedlite flash. Something like the
Canon Speedlite 270EX isn't too expensive. While it can still take 3-4 seconds for a full recharge, it's a more powerfull flash and so may not use up all of its charge on each shot like the internal flash would. The
Canon Speedlite 430EX II has even more power and a faster recycle time.
If you do a lot of flash work, I'd go for the 430EX II. It's a very full featured flash and gives you quite a bit more control than the 270EX.