{'fraction_non_alphanumeric': 0.029117027224635498, 'fraction_numerical': 0.009698507591071352, 'mean_word_length': 5.119621002763521, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation

Automated medical report generation in spine radiology, i.e., given spinal medical images and directly create radiologist-level diagnosis reports to support clinical decision making, is a novel yet fundamental study in the domain of artificial intelligence in healthcare. However, it is incredibly challenging because it is an extremely complicated task that involves visual perception and high-level reasoning processes. In this paper, we propose the neural-symbolic learning (NSL) framework that performs human-like learning by unifying deep neural learning and symbolic logical reasoning for the spinal medical report generation. Generally speaking, the NSL framework firstly employs deep neural learning to imitate human visual perception for detecting abnormalities of target spinal structures. Concretely, we design an adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network through embedding prior domain knowledge, achieving semantic segmentation of spinal structures with high complexity and variability. NSL secondly conducts human-like symbolic logical reasoning that realizes unsupervised causal effect analysis of detected entities of abnormalities through meta-interpretive learning. NSL finally fills these discoveries of target diseases into a unified template, successfully achieving a comprehensive medical report generation. When it employed in a real-world clinical dataset, a series of empirical studies demonstrate its capacity on spinal medical report generation as well as show that our algorithm remarkably exceeds existing methods in the detection of spinal structures. These indicate its potential as a clinical tool that contributes to computer-aided diagnosis.


Introduction
This paper devotes to the task of radiologist-level report generation based on spinal images in the field of spine radiology directly and automatically. Automated spinal medical report generation is a novel yet fundamental task in the domain of artificial intelligence (AI) in healthcare. Nowadays, multiple spinal diseases not only have deteriorated the quality of life but have high morbidity rates worldwide. For instance, Neural Foraminal Stenosis (NFS) has attacked about 80% of the elderly population (Rajaee et al., 2012). In daily radiological practice, radiologists still rely on laborious workloads to prepare tedious medical diagnosis reports through analyzing spinal medical images manually.
Time-consuming medical report generation leads to the problem of the delay of a patients' stay in the hospital and increases the costs of hospital treatment (Vorbeck et al., 2000). In contrast, automatic report generation systems would offer the potential for faster and more efficient delivery of radiological reports and thus would accelerate the diagnostic process (Rosenthal et al., 1997). Therefore, automatic report generation is pivotal to expedite the initiation of many specific therapies and contribute to relevant time savings, such that it could help spinal radiologists from laborious workloads to a certain extent.
To date, Computer-Aided Detection (CADe) and Computer-Aided Diagnosis (CADx) techniques in the medical image analysis community have made significant achievements and even can be on par with human experts (Esteva et al., 2017). However, most of them cannot achieve radiological report gener-ation, let alone the most-related spinal image analysis approaches (He et al., 2017b,c;Yao et al., 2016;Han et al., 2018c,b). Thus, the topic of automated spinal medical report generation based on medical images is still under-explored so far. Besides, magnetic resonance imaging (MRI) is one of the most useful exams in the clinical diagnosis of spinal diseases as it is better to demonstrate spinal anatomy (Kim et al., 2015). Therefore, this paper is devoted to the radiological report generation based on spinal MRI images to support clinical decision making. scales across normal and abnormal structures (Han et al., 2018b). Spinal structures also exist ambiguous spatial correlations that impede predicting consistent detection results. On the other hand, the subproblem of casual effect analysis mainly faces the difficulty of inexact supervision due to the lack of annotated data. This weak supervision pushes us to conduct unsupervised causal effect analysis, however, which con- tributes to the difficulty in discovering the pathogenic factors of target spinal diseases precisely (Han et al., 2018d).
To solve these problems, we formalize the task of spinal medical report generation as a human-like learning process that involves semantic visual perception and high-level symbolic reasoning. More precisely, we propose the Neural-Symbolic Learning (NSL) framework that combines deep neural learning and symbolic logical reasoning in a mutually beneficial way, as shown in Fig. 2. NSL learns to detect complex spinal structures through an adversarial graph network as deep neural learning to imitate human visual perception. Based on these discoveries of neural learning, NSL reasons out the causal effect, and further generate unified spinal medical reports through symbolic logical reasoning. The proposed NSL framework can resolve the facing challenges point-to-point. For handling the structural complexity and ambiguous correlations, we design the adversarial graph network that interpolates a symbolic graph reasoning module into a generative adversarial network to accurately segments complex spinal structures with wide variety and variability. The symbolic graph reasoning module embeds prior knowledge graph into the network to perform reasoning over a group of symbolic nodes, whose outputs explicitly represent different properties of each spine structure. For treating the inexact supervision, we use symbolic logical reasoning approaches that include meta-interpretive learning and firstorder logic programming by bringing in background knowledge to remedy the lake of supervision information.
Combining neural learning and symbolic reasoning for the medical report generation is proper and novel. As we have shown, it is proper because this combination imitates the process of manual spinal report generation in the clinic. Theoretically, it is novel because this combination endows the superiority of the NSL framework that integrates the advantages of neural learning on noisy data processing and the logical reasoning on knowledge representation.
In the history of AI research, neural learning and logical reasoning have almost been separately developed (Zhou, 2019). Neural learning is adept at low-level perceptual tasks but is unable to support secondary reasoning. At the same time, logical reasoning does well in high-level symbol reasoning, but it is hard to handle uncertainty knowledge on noisy data. In other words, modern neural learning adopts a probability and connection mechanism for representing over noisy implicit data. In contrast, classical symbolic AI adopts expressive firstorder logic for reasoning over explicitly represented knowledge (Russell, 2015).
For example, neural learning techniques can recognize target spinal structures, while logical reasoning algorithms can reason out the causal effect by integrating human knowledge. Unifying neural learning and logical reasoning would integrate the low-level perceptual ability and high-level reasoning ability towards robust spinal medical report generation. Accordingly, we formalize the problem of report generation similar to the human decision-making process that bridges perceptual and reasoning strengths in a mutually beneficial way.
In this work, we advance our preliminary attempt (Han et al., 2018a) in the following aspects: 1) propose a new framework that integrates neural learning and logical reasoning in a mutually beneficial way; 2) carry out more extensive experiments on performance analysis, validating the significant advantages of the proposed NSL over existing compared state-of-the-art methods; 3) make a more comprehensive review on medical report generation, providing a technique review on the statistical machine learning and logical reasoning.
The contributions of this paper include: • We propose a novel framework achieving automated spinal medical report generation. The framework provides, for the first time, a reliable solution by integrating deep neural learning and logical reasoning in the medical image analysis community.
• We propose a new graph adversarial network that embeds prior knowledge graph into generative adversarial networks. The proposed network dynamically models the high-level semantic correlations between spinal structures to enhance segmentation accuracy. It can also extend to various medical image segmentation tasks.
• We propose a symbolic logical reasoning model that leverages meta-interpretive learning to induce the causal effects between spinal diseases for discovering valuable pathogenic factors, which are beneficial for the pathogenesisbased diagnosis of spine diseases.
We organize the rest of this paper as follows. In Section 2, we review the related works in terms of medical image analysis and involved methodology.
We introduce the NSL framework in Section 3. We then present the details of validated datasets, experiment settings, and exhaustive results in Section 4.
Finally, we conclude this work in Section 5.

Related Work
In this section, we first review the related works in the medical image analysis community and briefly introduce the related works on methodology. The related works of medical image analysis mainly involve spinal image analysis and medical report generation. The reviewed methodology mainly includes neural learning algorithms and logical reasoning advances.

The Related Works of Medical Image Analysis
To the best of our knowledge, neither the CADe nor CADx technique has achieved spinal report generation. Existing works in spine radiology include but are limited to abnormality localization, semantic segmentation, and disease classification of spinal structures. General speaking, existing detection works of spinal structures include automated localization (Alomari et al., 2011;Corso et al., 2008;Štern et al., 2009;Zhan et al., 2012;Cai et al., 2015), automated segmentation (He et al., 2017b,c;Yao et al., 2016;Xu et al., 2020), and simultaneous localization and segmentation of one or two types of spinal structures (Ghosha et al., 2011;Huang et al., 2009;Klinder et al., 2008;Peng et al., 2006;Shi et al., 2007;Kelm et al., 2013). Although before-mentioned methods achieved accurate detection of spinal structure, they cannot accomplish the radiological classification of spinal structures. After that, a few radiological classification works are proposed, such as lumbar neural foramen grading (He et al., 2016), lumbar disc generation grading (He et al., 2017a;RajaS et al., 2011;Jamaludin et al., 2017), and spondylolisthesis grading (Cai et al., 2017).
Since the before-mentioned works only achieved a simple analysis of few types of respectively, but which do not present detailed workflow and framework. Digital speech recognition is also studied to assist radiologists in report generation for faster delivery of radiological reports (Vorbeck et al., 2000). In this study, Briefly speaking, the advance neural learning algorithms of NSL include dilated convolution, adversarial training, and graph reasoning. The dilated convolution is originally proposed by Holschneider et al. (1990) to compute the wavelet transform. Atrous convolution is then extended into the semantic segmentation Yu & Koltun, 2015;Chen et al., 2017), object recognition (Sermanet et al., 2013), and image scanning (Giusti et al., 2013).
The objective of graph reasoning is to capture the relations between objects.
The node of the graph always represents the specific objects, and the edge rep- GNNs can be applied in various applications, such as chemistry and biology (Duvenaud et al., 2015), knowledge graph (Schlichtkrull et al., 2018), recommend systems (Ying et al., 2018), and computer vision . In this paper, we propose a new graph reasoning module for capturing the relations between spinal structures and improving the high-level semantic representation.

Symbolic Logical Reasoning
At the dawn of AI, logical reasoning was one of the most studied areas of research and has been considered as a fundamental solution of AI (Dai et al., 2018). Representative works of symbolic logical reasoning include expert system (Liao, 2005), decision tree (Safavian & Landgrebe, 1991), and inductive logic programming (ILP) (Lavrac & Dzeroski, 1994). The drawback of symbolic logical reasoning lies in handling uncertainty and noisy data, which limits its application on complex real-world tasks directly, such as visual understanding, speech recognition, natural language processing, etc. With the development of statistical learning, lots of complex real-world tasks can be resolved.
These achievements gradually set off a wave of statistical machine learning, and lots of mainstream algorithms have been proposed, such as support vector machine (Cortes & Vapnik, 1995), Bayes network (Friedman et al., 1997), and neural networks (LeCun et al., 2015). However, statistical machine learning still faces several drawbacks (Russell, 2015). Firstly, it has weak generalization ability, i.e., it cannot understand the intrinsic subconcepts and the high-order semantic feature of concept classes. As mentioned in the AI community, convolutional neural networks would recognize a dog image to be a panda by applying a certain hardly perceptible perturbation (Szegedy et al., 2013). Secondly, lots of statistical machine learning algorithms require a large number of annotated datasets, according to PAC-learning (Valiant, 1984). Finally, lots of statistical machine learning algorithms are black-box without comprehensibility (Murdoch et al., 2019).
Since logical reasoning and machine learning have almost been separately developed in the history of AI research, a fundamental idea to overcame beforementioned limitations is to unify them in a mutually beneficial way. However, developing a unified framework has been deemed as the holy grail challenge for the AI community (Zhou, 2019). The primary difficulty lies in the fact that modern machine learning cannot provide first-order representation that are necessary inputs for classical symbolic AI (Russell, 2015). In recent years, few works have made efficient attempts to overcome this difficulty. Probabilistic Logic Program (PLP) (De Raedt & Kimmig, 2015) and Statistical Relational Learning (SRL) (Koller et al., 2007) are aiming at integrating probabilistic inference and logical reasoning. However, they usually require semantic-level inputs.
Neural logic machine (Dong et al., 2019) and PrediNet (Shanahan et al., 2019) are aiming at instead of traditional logic programming by using pure neural networks but still has the drawbacks of statistical machine learning. Lately, abductive learning achieves a breakthrough and can recognize numbers and resolve unknown mathematical operations simultaneously from images of simple handwritten equations (Zhou, 2019;Dai et al., 2018). This paper proposes a new neural-symbolic learning framework that combines deep neural learning and logical reasoning in a mutually beneficial way, and the results have demonstrated that it can generate robust medical reports in spine radiology.

Methodology
In this section, we give the problem setting of spinal report generation in Section 3.1 and then present the details of the neural symbolic learning framework in Section 3.2.

Learning Set-up
In the real-world scenario, we can observe the weak information about spinal medical reports; that is, there are existing object-level annotations (i.e., semantic segmentation annotations) rather than causal effect annotations in the learning period. Formally, a sample of n inexact-supervised labeled training is independently and identically drawn according to an underlying distribution D defined on X × Y, where X is a set of MRI images and Y is a set of semantic segmentation ground-truth maps that can be observed for each instance x i . Each pixel in a segmentation map y i has the possibility of M classes comprised of M − 1 types of normal/abnormal spinal structures and background, that is, Y = {c 1 , . . . , c M } and c M is the M -th class. Given one spine MRI image x i , the objective is to generate a medical report R i . Note that the learner has no access to ground-truth reports.
We conduct this setting because the weakly-supervised learning way is supposedly the only resolution for the spinal report generation. One may wonder the alternative resolution that directly trains end-to-end medical image captioning models using the ground truth of medical reports. However, this resolution is impractical so far. On the one hand, we argue that conventional natural image captioning technologies like (Kulkarni et al., 2011) do not meet clinical demands because they cannot achieve accurate prediction of keywords, such as disease types, locations, and causal effect analyses. As expected, these keywords among spinal medical reports are exactly significant clinical concerns, which are undoubtedly unlearnable for end-to-end image captioning models. Since clinical concerns inside a few keywords decide to the correctness of a radiological report, it is also improper to evaluate the performance of the end-to-end models on computer-made reports compared with radiologist-made reports using NLP evaluation metrics.
On the other hand, the amount of ground truth medical reports do not meet the requirement of end-to-end image captioning models. In daily practice, radiologists always write radiological reports with various styles, which leads to a lack of useful annotated data of the medical report, like the image captioning dataset, Visual Genome (Krishna et al., 2017). We all know that it is impossible to generate medical reports end-to-end using image captioning techniques with a small amount of dataset. As we have shown, the problem setting implies two critical subproblems comprised of supervised semantic segmentation and unsupervised causal effect analysis. More specifically, as shown in Fig. 3, it is proper to decompose the task into multiple procedures, i.e., detect learnable concerns by object segmentation and radiological classification (i.e., semantic segmentation) using pixel-level annotations first, and then discover the latent unlearnable concerns, causal effect, without any annotations. After these two procedures, we finally fill these discovered concerns in a standard template to generate unified radiological reports.  Generative Adversarial Network. Unlike image generation-oriented traditional generative adversarial networks, our network is designed specifically for semantic segmentation of complex spinal structures. It includes a generative network and a discriminative network with mutual promotion. More specifically, the objective of the generative network is to predicate pixel-level semantic segmentation maps, while the adversarial network is to supervise and promote the generative network. In the training period, the generative network targets generating vivid maps to trick the discriminate network. In contrast, the discriminate network  target discriminating inputs maps into fake maps generated by the generative network or true maps from the ground-truth. When an apparent confrontation occurs, the discriminative network actively assists the generative network to look out mismatches in a wide range of higher-order statistics.

Neural Symbolic Learning
Generative network. We construct the generative network according to the characteristic of spinal structures. Generally speaking, we set the amount and kernel size of layers according to the receptive fields, to ensure the receptive field of every layer to coincide with target spinal structures in MRI images. The layers of the generative network are organized in an autoencoder. As shown in Fig. 5, the encoder comprises of two standard convolutional layers, and four dilated convolutional layers. The decoder comprises of two deconvolution layers and one standard convolutional layers. Formally, we denote by y the output feature map, and by k the kernel with weight w and bias b. For each point i in the input feature map x, dilated convolution computes its output as y where σ denotes the active function. The r · f is equivalent to convolve the input feature map x with up-sampled kernels, which is produced by inserting zeros between two consecutive values of each kernel along each spatial dimension. Incremental dilated rates r of {2, 4, 8, 16} is adaptively used on four dilated convolutional layers, respectively, according to kernel's receptive fields. With the help of dilated convolution, the generative network can produce semantic task-aware representation using fewer parameters only. In summary, the generative network enables the NSL framework to address the challenges from high variability and complexity of spinal structures in MRI images.
Discriminative network. As shown in Fig. 5, the discriminative network is a simple classification network comprised of three convolutional layers with large kernels, three batch normalization layers, three average pooling layers, and two fully connected layers with dropout. The inputs of the discriminative network include the ground-truth maps or the generated segmentation maps from the generative network. The output is a single scalar that presents the probability of predicting the inputs from weather or not ground-truth. Note that the discriminate network can enable the generative adversarial network to correct predicted errors and breakthrough small dataset limitations. The discriminate network can avoid over-fitting as well as achieve continued gains on global-level contiguity, which make the generative adversarial network obtain reliable generalization.
Symbolic Graph Reasoning. To leverage the structural correlations of the lumbar spine, we design the symbolic graph reasoning module. The function of symbolic graph reasoning is to improve the segmentation consistency by embedding useful prior knowledge into neural networks. The symbolic graph reasoning performs reasoning over a group of symbolic nodes whose outputs explicitly represent different properties of each semantic in a prior knowledge graph. As illustrated in Fig. 5, we interpolate this module into the center between encoder and decoder of the generative network. The symbolic graph reasoning module firstly constructs a symbolic graph that represents the prior semantic knowledge of spinal structures. It then receives the latent code from the output encoder of the generative network. It finally performs reasoning over the latent code within a symbolic graph. As such, the symbolic graph reasoning module mainly has two processes: a symbolic graph construction process and a symbolic graph embedding process, which are introduced below, respectively.
Symbolic graph construction. The symbolic graph is formulated as G = (V, E), where V represents graph node-set and E represents graph edge-set. In this task, the symbolic nodes of the symbolic graph represent the normal and abnormal spinal structures, and the edges represent the spatial relationships between them. The symbolic graph construction conducts the construction of a group of symbolic nodes and edges, which explicitly represent different properties of prior knowledge. More specifically, assume the target normal/abnormal spinal structures have entities (classes) of M , the V is a sparse non-symmetric matrix with the shape of M × N , where N denotes the dimension of the value of entities. Graph edges shoulder the responsibility of concept belongings between entities. The symbolic graph adopts soft edges that shoulder the occurrence probabilities. In other words, each node represents one class of normal/abnormal spinal structure; as such, each edge between two nodes represent the relationship between two classes.
As for the value of i-th node V i , we use one common feature descriptor to extract the feature of i-th class. More specifically, we averagely extract the semantic feature of i-th class' image patches from the training dataset. For edges, we calculate the occurrence probabilities between two nodes as the value of the connected edge for generating the overall E formalized in a matrix with the size of M × N . After creating the symbolic graph, we embed it into the neural networks. Incorporating such high-level prior knowledge can facilitate networks to prune spurious explanations after knowing the relationship of each entity pair, resulting in good semantic coherency . Symbolic graph embedding. As we have shown, the objective of symbolic graph embedding is to embed the constructed symbolic graph into the autoencoder to enhance local features with prior domain knowledge. Generally speaking, we first use an attention-based mechanism to summarize the local features encoded in the feature map of the encoder into global semantic information. This process is called local semantic attention for shouldering the representations of symbolic nodes. Based on relationship evidence of symbolic nodes, we then integrate the global semantic information with graph representation. This process is called global graph reasoning that leverages semantic constraints from prior knowledge in the spine image to evolve global observations. We finally use the evolved global representations to boost the capability of each local feature representation by a global-local mapping process.
Local semantic attention process. Formally, the local semantic attention process first receives the hidden feature tensor X l ∈ R H L ×W L ×D L from decoder outputs, where H L , W L and D L represent the high, width, and depth of feature maps from the final decoder layer, respectively. We uses two convolutional layers with 1 × 1 kernels to convert X l into A l ∈ R H L ×W L ×M and B l ∈ R H L ×W L ×D N , respectively. Next, the A l tensor is reshaped to R M ×HM and is applied by a softmax in the M dimension to formalize the attention mechanism to the importance of distinct symbolic nodes. The B l tensor is reshaped to R HM ×N and multiply A l into final output H lsa ∈ R M ×N with the same size of graph entities V. The unify process can be presented by a function φ, where W lsa ∈ R D L ×D N is the trainable transformation matrix for converting each local feature into the same dimension with entities representation. The function φ is computed as: Here W a is a trainable weight matrix for calculating voting weights. H lsa m ∈ H lsa . x i ∈ X encoder . The attention weight a xi→m ∈ A l represents the attention importance of assigning local feature x i to the node m .
Global graph reasoning process. The global graph reasoning process performs graph propagation over representations H lsa of all symbolic nodes via the matrix multiplication form, resulting in the evolved features H g : where , which integrates the prior information by concatenating node representation V. W g ∈ R 2N ×N is a trainable weight matrix. The node adjacency weight e m→m ∈ E g is hard weight (i.e. {0,1}) if adjacent m and m occur simultaneously in the spinal MRI image. To avoid the feature scale shift problem from large magnitude, E g is normalized into Q − 1 2 E g Q − 1 2 in which all rows sum to one, such that whereQ ii = jÊ g ii , andÊ g = E g + I added self-connections and I is the identity matrix.
Global local mapping process. Similar to the local semantic attention process, this final process is to map the integrated tensors H g to the input of the decoder: where W glm ∈ R N ×N is also a learnable transform matrix. In contrast with A l , the A g is the attention weight matrix that maps by evaluating the compatibility of each symbolic node h g ∈ H g with each local feature x i : In summary, combining the high-level constructed symbolic graph and the symbolic graph embedding process leads to hybrid reasoning behaviors, which is also beneficial for merging the prior knowledge in the middle of the autoencoder.
The symbolic graph reasoning model is capable of representing the inherent feature of target structures, measuring the connection weight between normal and abnormal structures, and constructing the spine graph into the generative network. The graph reasoning thus enables the generative network to model the latent yet crucial correlations between normal and abnormal structures dynamically.
Learning Strategy. The learning strategy of the adversarial graph network has two stages: 1) construct the symbolic graph, and 2) optimize the network. Since we have present the previous first stages in the corresponding section, we only introduce the second stage. We denote by G(·; θ g ) the generative network while by D(·; θ d ) the discriminative network, where θ g and θ d are their learnable variables, respectively. As such,ŷ i = G(x i ; θ g ) represent the predicted segmentation map, and D(ŷ i ) denotes the probability thatŷ i is ground truth segmentation map. The objective of the adversarial graph network is to generate optimal segmentation maps where the value of each pixel represents a radiological classification result. Inspired by Goodfellow et al. (2014), we minimize a hybrid loss function denoted by L(θ g , θ d ), which is defined as: ] . (7) λ controls the equilibrate of adversarial training, and is set to one without loss of generality. The generative loss function (L mcl ) is a weighted multi-class cross-entropy loss function. The weight balance the prediction of generative network by computing the pixel amount of target classes. The discriminative loss function L bcl is a binary cross-entropy loss function with stable convegence.

Symbolic Logical Reasoning
Based on the results of the adversarial graph network, the symbolic logical reasoning conducts human-like reasoning that achieves unsupervised causal effect analysis of detected entities of spinal diseases. Symbolic logical reasoning utilizes meta-interpretive learning and first-order logic programming by bringing in background knowledge to remedy the lake of supervision information.
Causal effect for spinal report generation refers to the pathological relations between detected spinal diseases, which are value keywords among diagnostic reports in spine radiology. As such, causal effect analysis is inevitable in automated spinal report generation. It is noteworthy that causal effect analysis can significantly 1) promote clinical pathogenesis-based diagnosis, and 2) help early diagnosis when the pathogenic factor is solely occurring. Fig. 6 represents the workflow of symbolic logical reasoning. Generally speaking, we split the task of unsupervised causal effect analysis into two steps: 1) induce the hypothesis of the causal effect between target spinal diseases using meta-interpretive learning; and 2) conduct unsupervised labeling of segmented spinal structures using firstorder logic programming. Based on the induced hypothesis, we finally obtain the causal effect between labeled spinal structures. We present the two steps, respectively.

Meta-Interpretive Learning for Hypothesis Induction of Causal Effect between
Target Spinal Diseases. The objective of hypothesis induction is to summarize the pathological relations between target spinal diseases. We use metainterpretive learning, a novel inductive logic programming framework, to induce the causal effect hypothesis of pathological relations formalized by firstorder logic clauses. Meta-interpretive learning is proposed by Muggleton et al. (2015), and it supports predicate invention and efficient learning of logic hy- where A is an atom representing a specific target spinal structure. Atoms are the first-order logic formulas without conjunctions (∨, ∧), such as q(t 1 , . . . , t n ) where q is a predicate and t i are terms. Terms are constant z, variable V , or structured term in the form of f (u 1 , . . . , u n ) where f is a functor. B i is literal, which is an atom q(x) or its negation ¬q(x). A clause that does not contain any variable is grounded, and grounded atoms are ground facts.
The workflow of a MIL is continuous to prove a set of logic facts according to background knowledge by fetching higher-order meta-rules. The proving process is a predicate substitution process, and a predicate is invented if the substituted Here the symbol comma (;) represents disjunction (∨).

First-Order Logic Programming for Unsupervised Labeling of Segmented Spinal
Structures.. After the hypothesis induction process, we find the index of segmented spinal structures, i.e., labeling the order of segmented spinal structures.
Since segmentation ground-truth does not have the order information, it desires to do unsupervised labeling. It is worth note that unsupervised labeling is the base for further causal analysis and report generation. We leverage first-order logic programming to achieve this process.
As shown in Fig. 6, the inputs of the unsupervised labeling process are the generated segmentation maps from adversarial graph network, and the outputs are several dictionaries comprised of orders and normalities of spinal structures.
The keys of each dictionary are the order of one type structure, while the values of the dictionary are the normality conditions at the sites of one type structure in a lumbar spine. The first step is to discover patterns for location assignment  Because the segmented structures in segmentation maps always contain a few spots, the second step is a post-processing procedure to eliminate these spots and to estimate the correct label of isolated structures. Clinical concerns among medical reports are the situation of lumbar vertebrae, discs, and neural foramen from L1 to L5. Let lumbar discs as an example, we first calculate out the minimal height of vertebral in the training set and then let the height divided by four be the margin between pixels of intervertebral discs. The order can be determined according to the above logical clauses. We then compare the pixel amounts between normal and abnormal labels and then choose the one that has the most amount pixels as the final label. We finally collect the labeling results formed in a standard dictionary for the next process. After obtaining the order of segmented spinal structures, we input them into the hypothesis of causal effect to analyze the pathological relations between target spinal diseases.
Spinal report generation. In the end, we summarize the discoveries from segmentation and casual effect analysis, then fill these discoveries into a unified template. We use If-Then logical operations to create a unified template. For instance, if the neural foramen, disc, and vertebra are abnormal at L3-L4, the captioning process can output "At L3-L4, the intervertebral disc has obvious degenerative changes. The above vertebra also has deformation changes. They lead to the neural foraminal stenosis to a certain extent.". If the neural foramen is normal, and disc or vertebra is abnormal, one can predict the neural foramen has a large possibility to be stenosis.

Data and Configuration
The NSL is evaluated on a real-world clinical dataset. This dataset is collected from multi-center and various models of vendors. It includes 253 clinical patients. The average year of patient age is 53±38, with 147 females and 106 males. Among sequential T1/T2-weighted MRI scans of each patient, one middle lumbar MRI image was selected to better present neural foramina, discs, and vertebra simultaneously in the sagittal direction. In each MRI image, we can observe three types of spinal structures: neural foramen, intervertebral disc, and lumbar vertebrae. These three types of spinal structures are associated with three types of spinal diseases: lumbar neural foraminal stenosis (NFS), intervertebral disc deformation (IDD), and lumbar vertebral deformation (LVD).
The ground-truth was annotated by extracting from clinical reports, which are double-checked by board-certified radiologists.
The framework directly handles clinical MRI images without any pre/postprocessing and data augmentation. The feature descriptor for graph construction is the Histogram of Oriented Gradient (HOG). The generative network uses the RMSProp algorithm to optimize the weights θ g , while the discriminative network uses the Adam optimization algorithm to optimize the weights θ d .
The weights of both θ g and θ d are initialized with Xavier initialization. Considering the task of the generative network is harder than the adversarial network, the initial learning rate η 1 of RMSProp is set to 0.01, while the learning rate η 2 of Adam is 0.001. In terms of RMSProp optimizer, decay is 0.9, momentum is 0.9, and is 1e-10. In terms of Adam optimizer, β 1 is 0.9, β 2 is 0.999, and is 1e-08. The adversarial graph network is implemented in Python and Tensorflow library (Abadi et al., 2016). The logical reasoning model is implemented in Prolog. We use mini-batch size is 4, and training epochs is 300 using one Nvidia GPU Titan X with cuDNN v5.1 and Intel CPU Xeon(R) E5-2620@2.5GHz.
We split the whole dataset into a training set (80%) and a testing set (20%).
Standard five-fold cross-validation on the training set is employed for the model selection.

Experimental Design
The evaluation metrics include pixel-level accuracy, Dice coefficient, specificity, and sensitivity. The semantic segmentation of one spinal structure is correct if this structure is pixel-wisely segmented and classified correctly.
We compare the semantic segmentation ability of our neural symbolic learning framework (NSL) with several state-of-the-art semantic segmentation networks as follows.
• Fully Convolutional Network (FCN) (Shelhamer et al., 2017). FCN is a pixels-to-pixels semantic segmentation network. It transforms fully connected layers into convolutional layers with multi-resolution layers. The FCN-VGG16 is used for comparison, and the deconvolution layers of FCN-VGG16 are initialized by bi-linear up-sampling.
• SegNet (Badrinarayanan et al., 2015). SegNet is an encoder-decoder manner semantic segmentation network, in which the decoder up-samples its lower resolution input feature map. The used backbone network of SegNet is VGG16, with 13 convolutional layers.
• DeepLabV3+ . It extends DeepLabV3  by adding a decoder module to refine the segmentation results.
• U-Net (Ronneberger et al., 2015). U-Net is a very popular semantic segmentation network that is primarily designed for medical image segmentation. The core of U-Net is that appends skip connections between encoder and decoder layers.
• Spine-GAN (Han et al., 2018c). It is the state-of-the-art network for the semantic segmentation of multiple spinal structures. Spine-GAN is a different adversarial network and uses a local long-short term memory module (Local-LSTM) for modeling the spatial relationships of spinal structures.
• Generative Network without the symbolic graph reasoning module (GN-SGR). GN-SGR is an ablated version of the adversarial graph network with the autoencoder only.
• Adversarial Graph Network without the symbolic graph reasoning module (AGN-SGR). It is an ablated version combining the generative and discriminative networks.
• Adversarial Graph Network without the discriminative network (AGN-DN). It is an ablated version by combining the generative network and symbolic graph reasoning as well as removing the discriminative network.

Medical Report Generation
The representative radiological reports generated by the proposed NSL framework are illustrated in Fig. 7. Empirical results prove that NSL can directly generate radiologist-level diagnosis reports with weakly-supervised information.

Radiological Report
At T12-L1, the intervertebral disc has obvious degenerative changes. The neural foramen does not have stenosis. At L1-L2, the above vertebra does not have deformative changes. Disc degenerative changes are associated with neural foraminal stenosis. At L2-L3, the neural foramen has obvious stenosis. The intervertebral disc does not have obvious degenerative changes. The above vertebra does not have deformative changes. At L3-L4, the intervertebral disc has obvious degenerative changes. The above vertebra also has deformative changes. They may cause the neural foraminal stenosis. At L4-L5, the intervertebral disc has obvious degenerative changes. The above vertebra also has deformative changes. They may cause the neural foraminal stenosis. At L5-S1, intervertebral disc has obvious degenerative changes. The above vertebra also has deformative changes. They may cause neural foraminal stenosis to a certain extent. At L4-L5, the intervertebral disc has obvious degenerative changes. The above vertebra also has deformative changes. They may cause the neural foraminal stenosis. At L5-S1, the above vertebra does not have deformative changes. Disc degenerative changes are associated with neural foraminal stenosis. These results justify the significance of unifying deep neural learning and symbolic logical reasoning. These also verify the validity that NSL integrates the advantages of neural learning on noisy data processing and the logical reasoning on the knowledge representation.

L5
As shown in the first report in Fig. 7, the learnable and unlearnable concerns of spinal structures are predicated accurately and reliably, thanks to the powerful segmentation ability from the adversarial graph network. The labeling information in the generated reports are also robust that demonstrates the correctness of the first-order logical programming based on clinical background knowledge. These once justify the value of embedding domain knowledge into the learning process.
As the purple color text presented in Fig. 7, NSL achieves reliable causal effect analysis thanks to the symbolic logical reasoning. NSL can also produce pathological correlations between spinal diseases of NFS, LVD, and IDD, which demonstrate the feasibility and effectiveness of meta-interpretive learning. In the first report shown in Fig. 7, NSL automatically presents that the pathogenic factors of the NFS between L4-L5 are its surrounding L5 vertebra (LVD), L4-L5 intervertebral disc (IDD). Also, in the second report presented in Fig. 7, NSL rightly discovers that the abnormal L5 disc is the pathogenic factor of the L4-L5 NFS.
Generated unified reports justify that the weakly-supervised way is robust, and endows our framework a potential as a clinical tool to relieve radiologists from laborious workloads to a certain extent. Since it is impossible to judge the correctness of computer-made medical reports compared with radiologist-made reports using NLP metrics, it is possible to evaluate the accuracy of keywords about critical concerns in the generated spinal medical reports, by computing the metrics about semantic segmentation performance and labeling accuracy.

Semantic Segmentation Performance
As illustrated in Table 1, we achieve higher performance than the compared state-of-the-art methods in the semantic segmentation of three types of spinal structures. NSL significantly outperforms the FCN network by 4.8% pixel accuracy and 12.5% average Dice coefficient. NSL outperforms the U-Net network by 4.5% pixel accuracy and 8.2% average Dice coefficient. As illustrated  in Table 2 and Table 3, the effectiveness and advantages of NSL has also been demonstrated. NSL has simultaneously achieved accurate segmentation, precise radiological classification of neural foramen, intervertebral discs, and vertebrae, as shown in Fig. 8. Even both the structural complexity and ambiguous correlations between various spine structures lead to unusual difficulties, NSL obtains robust performance, which demonstrates its strengths in addressing the spatial relationships and high structures variability. Fig. 11 presents detailed charts that show visible improvement achieve by our algorithm. The representative bad cases are shown in Fig. 9, we can see that bad cases may be caused by the specialized structures of MRI images that have more spinal structures than general MRI images, which seldom impact the report generation performance.  The reason for semantic segmentation rather than object detection is that segmentation is better to present more spatial details than object detection.
Regard to the ablation study, various experimental results are shown in Fig. 11 (e, f) and Table 1 combination of the three modules (NSL) achieves better performance than its ablation version. Regarding radiological classification, NSL also achieves higher specificity and sensitivity than its ablated versions. Therefore, the combination of these sub-modules makes NSL an efficient and reliable resolution for the semantic segmentation of multiple spinal structures.

Unsupervised Labeling Accuracy
As illustrated in Fig. 12, the symbolic logical reasoning model achieves stable unsupervised labeling and produces a highly accurate labeling accuracy rate.
Under the condition of accurate semantic segmentation, the labeling accuracy rate is up to 100%, which proves the robustness of first-order logic programming.
That once demonstrates the importance of logical reasoning on knowledge representation. The combination of logical reasoning and neural learning can well handle noisy data and knowledge representation towards machine learning with the abilities of generalization, robustness, and interpretability.

Conclusion
In this paper, we proposed the Neural-Symbolic Learning (NSL) framework for the automated generation of medical diagnosis reports in spine radiology.
NSL combines neural learning and symbolic reasoning in a mutually beneficial way. As such, NSL is a human-like learning framework with visual perception ability and high-level logical reasoning strength. This combination can boost the generalization and interpretability of neural learning, also give a robust solution naturally. Extensive results have demonstrated its effectiveness and potential as a clinical tool to relieve spinal radiologists from laborious workloads. This framework has scalability and sustainability such that it can be easily extended to other diseases with the need of radiological report generation.