Interesting. Did you reload your website from the "Web" tab after setting up the static file mapping? If not, that might explain the problem: when you add the mappings to the static file table, they don't take effect until the website is restarted, so it's possible that you added the mappings, but didn't reload -- which would mean that Flask would be handling the static files, which would be slow. Then, at a later point, the website was restarted, either because you did it yourself or due to some system load-balancing moving it to another server, at which point the static file mappings kicked in and things sped up.
In terms of caching -- we provide a last-modified header in the response when a browser requests a static file. Most browsers, when they request the file again, will send an "if-modified-since" header set to the last-modified value from the response to the previous request, which means that if the file hasn't been changed, we just send back a "not changed" message instead of the full file -- which should give you all the caching you need.