After spending hours looking deep into the plugin I realized that the links were all broken.
http://www.domain.com/feed became http:www.domain.com. Just fix the links and it should be fine.
It is really hard to see because in the admin page you can still see the link and it works if you click on it (cf. screenshot)
I think some function striped the //, so possibly it could happen again.
The plugin page: http://wordpress.org/plugins/wp-rss-aggregator/