{'fraction_non_alphanumeric': 0.027295959980425208, 'fraction_numerical': 0.00581806318307868, 'mean_word_length': 5.313765877102643, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Web-based Multi-dimensional Medical Image Collaborative Annotation System

Medical image annotation is playing an increasingly important role in clinical diagnosis and medical research. Existing medical image annotation is faced with many demands and challenges. 1) The emergence and sharp increasing speed of multi-dimensional medical images. 2) Image annotation includes not only text annotation, but also graphical annotation, clinical diagnostic information and image content features information. 3) Uneven distribution of medical resources, which makes difficult to aggregate group intelligence from a much larger scale of distributed experts. Most of the present study is texted based within hospitals on single images annotation. It is difficult to organize and manage unstructured medical image annotation and collaborative sharing information. This paper dedicated to the research on collaborative web-based multi-dimensional medical image annotation and retrieval in order to address these problems, overcome the shortcoming of traditional thin client and facilitate medical experts in different locations to exchange views and comments,. It proposed 1) a system architecture that provides authoring, storing, querying, and exchanging of annotations, and supports webbased collaboration. 2) 2D multi-frame and 3D medical image collaborative annotation data model. 3) Collaborative annotation mechanisms.


Introduction
In-depth development of modern imaging and image processing technology makes medical images rapidly expanding in recent years. Images can be used for clinical, teaching and research. For example, early in 2002, only University Hospital of Geneva radiation department generated 12,000 pieces of images every day [1].
The total amount of image data produced in United States and European Union gets thousands of TB annually. In China a medium-sized hospital produced about 1~6TB image data annually and hospitals across the country produce about 2PB each year. Bill Ziegler, IBM's global vice president, forecasts that in 5-10 years 30% of the global storage will be used to store medical image data.
Medical image annotation and mark is the core of medical research whether for clinical trials or scientific researches. Traditional image annotation of PACS systems is text-based, which is difficult for organizing, managing and sharing unstructured medical image annotation information. The standard storage format of medical images is DICOM, but there is no standard data model for medical images annotation. Using appropriate method to represent the image annotation and markup, share image data and knowledge has been the bottleneck for researchers. Image annotation includes not only the graphical annotation, such as rectangles, polylines, irregular graphics, as well as clinical diagnostic information, feature information of image content. Therefore, establishment of an effective model for storage and management of annotation data is critical. So above analysis, this paper gives a new multi-dimensional medical image annotation model which integrated graphics, text, and the underlying characteristics. We only need to extract features of the label the region where we are interested in the disease, which can greatly reduce the time and improve the space efficiency.
Some of complex image annotation data often requires experts and doctors to synchronously diagnose and collaborative operation on the interest annotation image data sets. So they can get better analysis through sharing knowledge and experience. In order to facilitate diagnosis and research, exchanging and sharing medical experts' diagnosis opinions and comments in different locations is very meaningful. But now diagnostic systems are installed within hospitals, which is difficult to concentrate the wisdom of experts around between the hospitals. Collaborative systems technology is relatively mature, such as based on message delivery, and service-based transfer etc. This paper gives an effective collaborative annotation mechanism.

Related Work
Present paper and related work about medical image label model is very enough. Professor Wang Fu sheng proposed a single-frame medical images marked model CAM [2] when he did research in Siemens in 2008.Stanford University Daniel L. and others gave an ontology-based annotation model, AIM model [3].
In order to obtain an accurate marked regional characteristic, an effective feature extraction algorithm is very important. Recent feature extraction algorithm research contains follows. Harris corner detection algorithm [4] is a classic image feature extraction algorithm. Corner is robust for image translation, image rotation and image noise, but cannot adapt to changes in image scale. In recent years, some scale-invariant feature points are extracted from feature point extraction algorithm, such as Harris-Laplacian, Patch-Duplets algorithm, Laplacian algorithm and SIFT algorithm. And the first four algorithms are based on Harris corner point and extended to scale space. Such algorithms have robustness, but high computational complexity and poor performance in real-time, not suitable for Web-based operation to extract the features. SIFT (Scale-invariant feature transform, SIFT) algorithm is published in 1999 by David Lowe [5], perfectly summarized in 2004 [6]. It is a computer vision algorithm which is used to detect and describe the localized features in image .It finds the extreme points in the spatial scale, and extracted its location, scale, rotation invariant. This algorithm imports the image pyramid structure into scale space in order to reduce the amount of computation. At the same time it used BBF algorithm to speed up the search process, achieved good results on 128-dimensional feature vector space. This article uses ROI-based SIFT feature extraction.
In this paper, after we label the interest region, we can extract the ROI features. SIFT features extracted the following four steps [7][8]:Construct the scale space, detect the extreme points, gain the scale invariance; Feature point filter and precise positioning, excluding unstable feature points; Extract feature descriptor at feature points to determine the main direction of the key points; Generate feature descriptors and find match points by using feature descriptors. Through the above four steps, finally we get SIFT local feature descriptors which not sensitive to illumination changes, scale changes and rotation.
At present, there appears some medical image collaborative visualization system, the DIAGNOSIS systems [ 9 ], Disney system [ 10 ] and CORBA-based (Common Object Request Broker Architecture) medical image collaborative visualization system [11]. These system are adopted a collaborative model of sharing event which: 1) need to backup an image data set in each collaborative client. 2) Each collaborative client needs operating the event to achieve the same state. This collaborative mechanism is inadequate for it needs to transfer large image data sets to each collaborative client before the collaborative session. 1) On the one hand, it requires high network bandwidth to transmit image data to each collaborative client. 2) On the other hand, each collaborative client needs to install a professional collaborative visualization customer. SOA technology is mature, we can build SOA-based collaborative mechanisms, which all operating service are called and executed on the server, and the server pushes result status to each collaborative client. This greatly increases the speed of collaborative and save collaborative cost. MICAS is divided into server and client side. System client provides users a Web-based GUI, including user register module, image web annotation module, features extract module, collaborative annotation, and annotation image retrieval. The server contains two layers: data layer and service layer. The data layer contains three databases: source DICOM image database, annotation user database and annotation data database which using MIAM data model. The service layer includes DICOM image rendering service, annotation operation service, feature extract service, annotation recommend service, collaborative framework service and retrieval service.
Its main functions are as follows: 1) DICOM image rendering service: It contains Muti-2D frame images rendering and 3D medical image rendering. We can directly open ".dcm" files through the internet.
2) Web-based Image Annotation: we provide kinds of graphical annotation shapes including rectangle, circle, ellipse, line, polyline, etc. Users can add sematic annotation that contains diagnostic information, study date and doctor name. And we can extract the features of ROI using the SIFT algorithm. These data is very important for annotation image retrieval and collaboration. 4) Annotation retrieval module: In order to improve the precision and resolve describe difficulties, the retrieval module include three patterns. First, a simple search method just uses keywords and sematic information such as the time marked by the user. Second, retrieval based on SIFT features which can find a similar images by using ROI features. Third, integrate the keywords and image features for retrieval. 5) Collaborative service: experts in different locations can access the system at the same time, diagnose the same cases. They can see the others' opinions and can audio and video collaborative so as to help diagnosing. 6) Annotation data management service: the source DICOM data is organized by AUDR and the annotation image data is organized by MIAM annotation data model, which contains four parts of annotation data, graphical data, sematic data, features data and collaborative data. And we need a user database to manage user data for safety consideration.

Collaborative Image Annotation Data Model
Now in hospitals, medical image data are stored in DICOM format, including image data and image header files. The image data contain single-frame data, 3D image data and 2D multi-frame data (Figure2). The header file information called "head Info" contains basic information of the patient and image.
We proposed a Medical Imaging collaborative Annotation data Model (MIAM) as Figure 3 depicts. In the model, each root node, as DICOM, represents an annotation data and the meaning of each child nodes are explained respectively. Each DICOM node contains image header file information, as headInfo, and one or more User Annotation Space, as UAS. For 3D images, we first need to label the interested view of 3D image called VOI (View of Interest). And each view has different interested frames called SOI (Slice of Interest) .We can organize different SOI into groups ,as GOI(Group of Interest),which can be used to identify a disease. And for each single-frame image, we just need to mark the different regions of interest ROI. Each ROI contains graphical annotations, semantic annotations, feature extractions and collaborative comments. (1) H, headInfo, denotes the basic information collection of medical image. This information is contained in the head files of DICOM data and can be obtained by parsing the DICOM data. The patients' information include patient number, name, ID number, gender, age, date of birth. The images information includes check date, hospital, hospital Code, Image, position, clinical, sharer, image type (CT, MR, DR etc.), shooting site etc. U represents the collection of UAS .The system allows more than one user visit it at the same time, and a user can have more than one annotation space, so we allow more than one UAS node. Each UAS is determined by annotation state of the current. U is defined as Definition 2.

Definition 2：
(2) In definition (2), U represents the current UAS. D is basic information of the user and can be obtained from UserInfo. It contains username, password, realname, role, company, address, mobile, telephone, fax, email, title, description, collaborative group of the annotation user. V represents VOI. G is the group of interested frames GOI, R represents ROI and can be obtained by the user operation and feature extraction, collaborative diagnosis. , an element of R, is defined as definition (3):

Definition 3：
(3) According definition (3), each r represents a ROI, where M is the graphical annotation information called markup, a graphical notation information, such as lines, rectangles, polylines. A contains the diagnosis of semantic information, such as diagnostic information "pleurisy". S means the regional SIFT feature characteristics of ROI. C represents the collaborative diagnosis /reviews /comments and visibility settings etc.

Collaborative Annotation Mechanisms
In this section, we will present the collaborative annotation model, framework, and collaborative operation flow and design.

Collaborative annotation model
Collaborative annotation model is as Figure 4. This collaborative annotation model uses server push technology and mechanism of sharing the event result. In this model, the primary client, as in Figure 4 Client A, who execute annotation on the medical image sent this operation event to the server. On the server side, server calls the corresponding medical image annotation service according to the operation event type. The service returns medical image annotation results. Collaboration servers will distribute the results to every collaborative client, Figure 4 Client B and Client C. The client can be Notebook computers or mobile phones, just need a browser and internet. Various collaborative clients receive the results and achieve the same label status through front show.

Collaborative Annotation Framework
Collaborative annotation framework [12][13], as Figure 5 shows, is divided into three levels: collaboration client layer, collaboration server layer, and annotation image server layer.

Fig.5 Collaborative Annotation Frameworks
With web 2.0 [14][15] technology development, rich client-side technologies are increasingly being used to build Web applications. In order to provide better interactivity and user interface, we adopt Flex technology to build a collaborative client. Collaborative client layer contains a Flex-based user interface to demonstrate the collaborative annotation results and user interaction. And users can do interoperability of medical imaging through the user interface. Logic modules and data modules are used to maintain the logic operation and annotation data of the client. The message listener and sender modules are used for receiving collaborative notification messages and sending operating messages.
Collaborative server layer consists of three core services: collaborative arbitration service, message notification service, and collaborative session service. We manage collaborative session by using collaborative session service, and client can join or leave a collaborative session under the management of this service. Collaborative arbitration services are very important. It determines the acceptance of collaborative client interaction events, and calls the corresponding image annotation service. Message notification service distribute message to the collaborative client.
The image server layer includes medical image annotation services and annotation data store service. The image annotation service can perform the image visualization operations and generate visualization results. And collaborative client can annotate medical image by the image services. The Annotation data store service provide access interface for image data storage and management.

Collaborative operation flow and design
In the system, we create a group for each user which they can choose the user they want to communicate with. And in the collaborative conversation group, all clients operating processes show in Figure 6. The main client first initiates the collaborative operation, and then the image annotation operation passed to the collaborative arbitration service. Collaborative arbitration services analysis annotation operation message, check the legality of the operation, then map the operating event to the corresponding Web Service [16] [17]. The Web Service returns the collaboration state resources which including the current label shape, size, location, and the feature values of the marked region, as well as the label image state and view state. All collaborative clients in the collaborative session get status updates, through message notification service and finally update the local state and collaboration servers consistent.

System Results and GUI
This part simply gives the system GUI, the feature of ROI results and collaborative results.
In the System, as shown in Figure 7, we can chose the annotation shape, line style, line color etc. Then can add sematic annotation as "diagnose information" window shows, which including diagnostic results, doctor, and date. On the right side, we can see extract feature button, see the current annotation button, annotation recommend button. And we can see the collaborative operation zone.   Figure 8 shows the SIFT feature of ROI. Figure 8-a shows the intercepted part feature extraction results which the end of the arrow represents the location of the feature points, the direction of the arrow represents the direction of the feature points, the segment length represents the scale of the feature points. And in Figure  8-b shows the result is saved as a text file in order to store into the database. Figure 9 shows collaborative annotation, which including audio and text communication.

Conclusions and Future Work
The most import of the research is that gives a new online medical application diagnostic form and architecture. This paper completes the following aspects: 1. We propose a new scenarios and system architecture about medical image collaborative annotation. 2. In order to solve the problem of effective organization and management of unstructured medical image annotation data, we propose a collaborative medical image annotation data model (MIAM). 3. In order to facilitate experts in different locations to access the system at the same time, diagnose the same cases, we give a collaborative mechanism to improve the speed, saving the cost of collaboration.
In future, in order to improve the retrieval accuracy of the region of interest, we use SIFT features and sematic annotation together.