Towards Architecture-Agnostic Neural Transfer: a Knowledge-Enhanced Approach