← Back to All

TL;DR version

I helped create a computer-aided device system that streams in video from an endoscope (camera used in a colonoscopy) and using AI and computer vision algorithms highlights possible characteristics of diverticular disease and presents them to the doctor for verification.

I worked diretly with the doctors and observed and asked questions when they watched colonocopy videos to understood the useful features needed to assist them during a colonoscopy. Then translated those into a working prototype.

This was developed in windows specifically for Mayo Clinic so the interface looks like windows.

A sample of the video a doctor would see with side-by-side frames

Long wordy version

Introduction

DIVERTICULOSIS or colonic diverticular disease (CDD) is a devastating disease that affects hundreds of thousands of people every day – and will eventually affect every person in the western world. Diverticulosis is endemic in societies in which consume highly processed food – it is thought to be related to the low stool bulk and colonic dysmotility, which leads to increased pressure in the colon as it is essentially the formation of numerous tiny pockets in the bowel lining on weakened spots. The pathologic features of Diverticulosis include thickening of the colonic walls, therefore narrowing of the lumen and decreased distensibility of the colon. Diverticulosis occurs in ten percent of people over age 40, fifty percent of people over age 60, and almost everyone over age 70-75 [5].

Currently, Irritable Bowel Syndrome (IBS) and stomach ulcers are the leading symptoms of Diverticulosis coupled with little else. Doctors will use your medical history with analysis on one or two of the following methods; X-rays, CT scanning, ultrasound testing, a sigmoidoscopy, colonoscopy, barium enema, and blood tests. CT scan the standard de facto methodology as it is non invasive and relatively quick. The main drawback being the resolution and the more ease of confusing symptoms with anomalous readings. Right now they work with four different ‘key symptoms’ which are (1) pericolic fat infiltration, (2) thickened fascia (about 78.9 percent correct), (3) muscular hypertrophy (26.3 percent) and (4) "arrowhead sign" (23.7 percent) [6]. These are observed when looking at CT scans and specifically looking for CDD.

Colonoscopy is currently the preferred screening modality for prevention of colorectal diseases including diverticular. During a colonoscopy procedure, a flexible endoscope (with a tiny video camera at the tip) is gradually advanced under direct vision via the anus and rectum into the most proximal part of the colon (signified by the appearance of the appendiceal orifice or the terminal ileum). Next, the endoscope is gradually withdrawn. The video camera generates a sequence of images (frames) of the internal mucosa of the colon. These images are displayed on a monitor for real-time manual analysis by the endoscopist. Biopsy and therapeutic operations such as polyp removal can be performed during the procedure.

Colonoscopy has resulted in a decline in the number of colorectal disease related deaths such as colon cancer. Nevertheless, recent data from clinical trials suggest that there is a significant miss-rate for detection of both large polyps and cancers [4, 8]. The miss-rate may be related to the experience of the endoscopist and the location of the lesion in the colon, but no large prospective studies related to this have been done thus far., In order to reduce the missing detection rate, there is needed to develop a computer-aided system to help the endoscopists to reduce the missing detection of all kinds of diseases in colon of patients; the diverticular disease is one of the targets which needs to be detected in this computer-aided system.

The proposed computer-aided system for the detection of Diverticulosis in this project will be designed for detecting the colon wall (or lumen folders) surface that have properties characteristic of Diverticular disease. For our literature survey, current technique focus on the diverticular detection using CT image, there is no literature addressing the diverticular detection in colonoscopy video images so far. From our observation, there are a lot of challenges issues need to be handled for the detection technique in our system.

Challenges

There are some challenges problems from our observation in this project which are needed to be considerate for the handling before the proposed system is addressed:

(1). Strong light reflection: Due to the reflection from the light generated by camera lighting against the colon wall, there are a lot of images with strong light reflections which could cause noisy edges in the binary edge images. The reflection area could also appear inside of diverticulars which are filled with some water, so that it could lead the difficulty to extract the good features from diverticulars.

(2). Limited texture or color information: Unlike the regular images in the real word, people can use some information such as color or texture from the surface of object as feature to tell from the background objects. There is monotone color information or even hard to find any texture information for the images captured inside the colon of human body.

(3). Large variance of shape and appearance: The shape of diverticular has large variance from different angles of view as the camera moves through the passages. Also, the diverticular appearances have large variance for different videos or patients.

(4). Large scale and rotation variance: The distance between camera and diverticular, and the rotation of camera during inspection procedure by doctor could cause the rotation and the large variance in the scale of diverticular region as well.

There are still some other challenges such as blurry frames which is caused by the motion of camera, we will create some robust techniques in our system to solve most of challenges issues we have addressed for all clear seen diverticulars except the blurry images since there is too hard to extract the edge on such images

Proposed System

The figure 2 depicts the overview of the computer aided system that will be developed in our project. We will give the explanation for the function of each component in system as following:

Frame Capturer: The component is designed to generate the JPEG image frame from a MPEG movie by setting the start capture frame, end capture frame, and frame rate. We have existing code finished for this component; the code was done using C++ and will be integrated into our system by revoking the exe file.

Blurry Image Filter: We will use the video captured from a Fujinon endoscope, from our observation, the blurry image generated by this brand of Fujinon endoscope can be up to half size number of images of the whole video. Blurry frames can not be used to do the recognition since they are even hard to be recognized by human eyes. We will apply the blurry image filter proposed in [1] for the remove of the blurry frame in the image preprocessing step.

Light Reflection Detector: The image can have some light reflection areas which can generate noisy edges in our edge detection step. We will apply the light reflection detection and remove technique. The method proposed in [1] is to do the segmentation to margin the relative light area and then do the light reflection detection based on the Saturation (S) and Value (V) in HSV color space.

We will not apply the segmentation step to detect relative light area discussed in [1] because the relative light area will not affect our edge detection results. Instead, we will consider only the strong light reflection areas which would cause to generate edges in binary image and implement the light reflection remove method to break apart the edges caused by light reflection area in this component.

Edge Detector: We will extract the edges to detection the contour of diverticulars using the Canny edge detector or Laplacian of Gaussian (LoG) filter. Before the edge detector, a Gaussian filter will be applied to smooth the grayscale image. To remove noisy edges and redundant branches, we plan to apply a skeleton algorithm [2] to remove redundant pixels on the edge boundaries but do not allow edges to break apart and then apply a spur operation with a pruning algorithm [2] to remove small branches on edges, so that we can get clear edges without branches on them. We will consider an edge as a noisy edge if the number of edge pixels of this edge is smaller than a lower threshold or larger than an upper threshold since they are not likely the diverticular edges, and they will be removed also.

Ellipse Fitting Model: One of the initial steps in the computational detection performed on colons is the segmentation of the colon lumen as it has an elliptical shape as well. Traditional techniques in CT imaging, such as threshold region growing, can be used to easily segment the lumen because of the large difference in CT attenuation values. In a colonoscopy video we plan to employ an idea that would to make the difference between the lumen and a region of interest like a diverticular, by shading. In one of our proposed approach, we will fit an ellipse to most the ‘elliptical’ objects in the frame or picture and then segment by the shading regions. We have tried the least square ellipse fitting method based on some binary images with only edges. (fig 5) We will remove the edges which will not fit well in the ellipse model, since the diverticular contours are most elliptical shape liked edges. We obtain the most of clear diverticulars from this step, however, most of lumen folder contour are also ellipse shape liked edges. Some of lumen folder (the big nested boundary edges on lumen) can be filtered out because they are generally consisted by large size of edge. We will also remove the edges which are fitted by the ellipse with large curvature so that some lumen folders can be filtered out. However, there would be still several lumen folder edges left. From our next component, we will focus on classifying the diverticular edges with lumen folder edges left after this step.

Pattern/Feature Extractor: This component will be designed to extract the pattern or feature which can be used to classify the diverticular edges with lumen folder edges. We proposed several models in figure 6 based on ellipse obtained by the ellipse fitting method. We are going to reject the edges with lumen folder patterns. We separate the ellipses into different kinds of patterns, depending on the appearance of ellipse. We try to use the grayscale value and Saturation (S) in HSV color space to determine whether the detected contour in the testing data are accepted or not by classifying it into the patterns defined in the training template.

Edge Cross-Section Profile Generator: Consider the intensity crossing the edge along the normal direction pointing to the center of the ellipse. It will be getting dark inside the ellipse and getting light outside of the ellipse edge. The watershed method has been widely used to do the segmentation based on the intensity difference. Consider the cave structure in the diverticular compared with on the flat area of surrounding the diverticular edges, the watershed technique probably can be used to find the cave from the information of intensity; we will try to extract the profile of edge cross-section along the edge to get the watershed structure as another pattern.

Recognition Engine: The recognition engine has the rules defined or the patterns extracted from the training samples. We will try some statistical learning method such as Naive Bayesian Gaussian model, Decision Tree, Support Vector Machine and so on in Weka [3] to build a robust model based on the evaluation using 10-folder cross validation method, or we will probably define the rules to do the classification based the feature or pattern extracted.

Temporal Tracker: This component is considered as the possible future improvement from the single image detection results to enhance the performance in the video. The component is to do the tracking based on the temporal information on the sequential frames of video. In a time window, if the detection results of some frames are labeled as diverticular frames, then it is a high probability that other non-diverticular frames in that window are wrong detected. And they will be changed to diverticular frames by this temporal information. (See fig 7)

Feedback Demo Generator: The component will be designed to generate the output image frames by using the detected frames which have diverticular inside and the positions of diverticular will be labeled by highlighted color. The generated frames will be saved into a new video. The demo will be the combined video with the original video and detected video with each of them taking the space of half size of video screen.

Evaluation Method

We will evaluate our method using at least three videos, and calculate the confusion matrices based on the experiment results. The diverticular detection performance with sensitivity and specificity of all non-blurry images in each video will be presented as final results.

The expected outcome will be the sensitivity and specify rates are larger than 80% for the clear frames. The expected good demo will highlight the position in the frames of the video of the diverticular frames with the reasonable detection rate so that it could help the doctor for the review.

We will present some future improvements such as how to use the video information to enhance the detection rate from sequential images, such as the tracking based on the temporal information we have mentioned. And we will present a better idea if we come out one at the end but we have not more time to implement in this project.

Reference

[1]. JungHwan Oh, Sae Hwang, JeongKyu Lee, Wallapak Tavanapong, Johnny Wong, and Piet C. de Groen, Informative frame classification for endoscopy video, Medical Image Analysis, vol. 11, Issue 2, pp. 110-127, Apr. 2007.

[2]. William K. Pratt, “Digital Image Processing”, John Wiley & Sons, Inc., 1991.

[3]. Weka 3: Data Mining Software in Java, http://www.cs.waikato.ac.nz/ml/weka/ .

[4] American Cancer Society, "Colorectal Cancer Facts & Figures," American Cancer Society, 2006.

[5] Joshua Katz, M.D., George Washington University School of Medicine - The National Digestive Diseases Information, 2007. http://digestive.niddk.nih.gov/ddiseases/pubs/diverticular/

[6] Shen SH, Chen JD, Tiu CM, Chou YH, Chang CY, Yu C. Colonic diverticulitis diagnosed by computed tomography in the ED. Am J Emerg Med 2002;20:552.

[7] http://hcd2.bupa.co.uk/fact_sheets/html/diverticular_disease.html

[8] D. Lieberman (Editorial), "Quality and Colonoscopy: a New Imperative," Gastrointestinal Endoscopy, vol. 61, no. 3, pp. 392-394, 2005.

[9] Computer-Aided Detection of Colonic Diverticular Disease, Van Uitert, R.L.; Jiang Li; Summers, R.M. Biomedical Imaging: ISBI 2007. 4th IEEE International Symposium on Volume , Issue , 12-15 April 2007 Page(s):1244 - 1247