Forbidden knowledge in machine learning -- Reflections on the limits of
  research and publication
  Certain research strands can yield "forbidden knowledge". This term refers to
knowledge that is considered too sensitive, dangerous or taboo to be produced
or shared. Discourses about such publication restrictions are already
entrenched in scientific fields like IT security, synthetic biology or nuclear
physics research. This paper makes the case for transferring this discourse to
machine learning research. Some machine learning applications can very easily
be misused and unfold harmful consequences, for instance with regard to
generative video or text synthesis, personality analysis, behavior
manipulation, software vulnerability detection and the like. Up to now, the
machine learning research community embraces the idea of open access. However,
this is opposed to precautionary efforts to prevent the malicious use of
machine learning applications. Information about or from such applications may,
if improperly disclosed, cause harm to people, organizations or whole
societies. Hence, the goal of this work is to outline norms that can help to
decide whether and when the dissemination of such information should be
prevented. It proposes review parameters for the machine learning community to
establish an ethical framework on how to deal with forbidden knowledge and
dual-use applications.
