OK so i did a quick extension to change the name - that works:
return filename.replace('(', '_').replace(']', '_').replace('ä', 'ae').replace('ü', 'ue').replace('ö', 'oe').toLowerCase();
-> filename special characters are replaced and file is stored with the correct name. Before I extend this to cover the remaining characters, I ran in the next problem: now the file remains in the tmp-upload and is no longer moved to the proper folder. not even the ones that were moved before. That part I dont understand.
EDIT
I think I tempered at the wrong location. The problem seems to actually be with the \system\libraries\upload.php where the filename should be sanitised further. Is there a config setting to limit the permissable characters?