title: Image Recognition on Browser using Python
body: <p>I want to implement a software for counting cards in blackjack, using some image recognition to automate the process. But i don't know where to start. 
  I think the problem could be divided into these steps:</p>
  
  <p>1- Take an image from the browser in-game (it's basically an Adobe Flash Game)</p>
  
  <p>2- Process the image, with some image recognition, that recognize all the cards.</p>
  
  <p>3- Updating the counter using the strategy Hi-Lo</p>
  
  <p>4- Showing the result on screen</p>
  
  <p>How can i do this using python? What are the libraries that could help me? This is a totally new field for me. I'll try to implement the question based on your suggestions. </p>
  
  <p>EDIT 1:</p>
  
  <p><a href="https://www.seleniumhq.org/docs/03_webdriver.jsp" rel="nofollow noreferrer">Selenium Webdriver</a> works fine, so far i've used this peace of code to get the screenshot of the main page, but i can't get into the game, because i don't have the money to play lol: </p>
  
  <pre><code>from selenium import webdriver
  
  browser = webdriver.Chrome()
  browser.get('https://www.888casino.it/giochi-da-casino/')
  browser.save_screenshot('screenie.png')
  browser.quit()
  </code></pre>
  
  <p>But basically i need to substitute the <code>browser.get()</code> with something that hooks the browser, rather then something that opens a new page.
  Then i need to implement a for loop that takes screenshots every second while i'm playing the game, and after that i can start processing those images.</p>
  
  <p>EDIT 2:</p>
  
  <p>I will try <a href="https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API" rel="nofollow noreferrer">TensorFlow API</a> for the image processing, but i didn't find any training model for recognizing cards. So i have to create a brand new model, i found this <a href="https://github.com/EdjeElectronics/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10" rel="nofollow noreferrer">tutorial</a> that helps me training my own object recognition model. Please, if you know an existing training model, link that.</p>
  
  <p>EDIT 3:</p>
  
  <p>Using Tensorflow i was able to create my own model for object recognition, now i need to use that model in a python script. For now i've used this sample script, that opens an image and draw rectangles around the cards. </p>
  
  <pre><code>import os
  import cv2
  import numpy as np
  import tensorflow as tf
  import sys
  
  # This is needed since the notebook is stored in the object_detection folder.
  sys.path.append("..")
  
  # Import utilites
  from utils import label_map_util
  from utils import visualization_utils as vis_util
  
  # Name of the directory containing the object detection module we're using
  MODEL_NAME = 'inference_graph'
  IMAGE_NAME = 'test1.jpg'
  
  # Grab path to current working directory
  CWD_PATH = os.getcwd()
  
  # Path to frozen detection graph .pb file, which contains the model that is used
  # for object detection.
  PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,'frozen_inference_graph.pb')
  
  # Path to label map file
  PATH_TO_LABELS = os.path.join(CWD_PATH,'training','labelmap.pbtxt')
  
  # Path to image
  PATH_TO_IMAGE = os.path.join(CWD_PATH,IMAGE_NAME)
  
  # Number of classes the object detector can identify
  NUM_CLASSES = 13
  
  # Load the label map.
  # Label maps map indices to category names, so that when our convolution
  # network predicts `5`, we know that this corresponds to `king`.
  # Here we use internal utility functions, but anything that returns a
  # dictionary mapping integers to appropriate string labels would be fine
  label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
  categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
  category_index = label_map_util.create_category_index(categories)
  
  # Load the Tensorflow model into memory.
  detection_graph = tf.Graph()
  with detection_graph.as_default():
      od_graph_def = tf.GraphDef()
      with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
          serialized_graph = fid.read()
          od_graph_def.ParseFromString(serialized_graph)
          tf.import_graph_def(od_graph_def, name='')
  
      sess = tf.Session(graph=detection_graph)
  
  # Define input and output tensors (i.e. data) for the object detection classifier
  
  # Input tensor is the image
  image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
  
  # Output tensors are the detection boxes, scores, and classes
  # Each box represents a part of the image where a particular object was detected
  detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
  
  # Each score represents level of confidence for each of the objects.
  # The score is shown on the result image, together with the class label.
  detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
  detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')
  
  # Number of objects detected
  num_detections = detection_graph.get_tensor_by_name('num_detections:0')
  
  # Load image using OpenCV and
  # expand image dimensions to have shape: [1, None, None, 3]
  # i.e. a single-column array, where each item in the column has the pixel RGB value
  image = cv2.imread(PATH_TO_IMAGE)
  image_expanded = np.expand_dims(image, axis=0)
  
  # Perform the actual detection by running the model with the image as input
  (boxes, scores, classes, num) = sess.run(
      [detection_boxes, detection_scores, detection_classes, num_detections],
      feed_dict={image_tensor: image_expanded})
  
  # Draw the results of the detection (aka 'visulaize the results')
  
  vis_util.visualize_boxes_and_labels_on_image_array(
      image,
      np.squeeze(boxes),
      np.squeeze(classes).astype(np.int32),
      np.squeeze(scores),
      category_index,
      use_normalized_coordinates=True,
      line_thickness=8,
      min_score_thresh=0.80)
  
  # All the results have been drawn on image. Now display the image.
  cv2.imshow('Object detector', image)
  
  # Press any key to close the image
  cv2.waitKey(0)
  
  # Clean up
  cv2.destroyAllWindows()
  </code></pre>
  
  <p>Now i need to create my own script that recognise the cards, and for every card updates a counter that must be shown on screen. This is the most tricky part because i don't know where to start. I have several problems with this step, first the script must be able to distinguish the cards that left the deck from the new one, this way it doesn't mess up the counter every time a screenshot is taken. Second, the counter should be updated with -1 for high cards(ten - ace), +1 for low cards(two-six), and 0 for neutral cards(7-8-9), and must be visible on screen.</p>
  
  <p>EDIT 4:
  I've build the first version of the software, but there are some issues, the counter doesn't update correctly. Here is the code:</p>
  
  <pre><code>import pyscreenshot as ImageGrab
  from win32api import GetSystemMetrics
  import os
  import cv2
  import numpy as np
  import tensorflow as tf
  import sys
  
  import warnings
  import h5py
  
  def UpdateCounter(labels, c):
      for i in labels:
          if labels['ace'] &gt; 0:
              c = c - 1
          if labels['king'] &gt; 0:
              c = c - 1
          if labels['queen'] &gt; 0:
              c = c - 1
          if labels['jack'] &gt; 0:
              c = c - 1
          if labels['ten'] &gt; 0:
              c = c - 1
          if labels['six'] &gt; 0:
              c = c + 1
          if labels['five'] &gt; 0:
              c = c + 1
          if labels['four'] &gt; 0:
              c = c + 1
          if labels['three'] &gt; 0:
              c = c + 1
          if labels['two'] &gt; 0:
              c = c + 1
          return c
  
  if __name__ == '__main__':
  
  
      sys.path.append("..")
      from utils import label_map_util
      from utils import visualization_utils as vis_util
  
      MODEL_NAME = 'inference_graph'
      IMAGE_NAME = 'test1.jpg'
      CWD_PATH = os.getcwd()
      PATH_TO_CKPT = os.path.join(CWD_PATH,MODEL_NAME,'frozen_inference_graph.pb')
  
      PATH_TO_LABELS = os.path.join(CWD_PATH,'training','labelmap.pbtxt')
      PATH_TO_IMAGE = os.path.join(CWD_PATH,IMAGE_NAME)
      os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
      NUM_CLASSES = 13
  
      label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
  
      categories = label_map_util.convert_label_map_to_categories(label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
  
      category_index = label_map_util.create_category_index(categories)
  
      detection_graph = tf.Graph()
  
      with detection_graph.as_default():
  
          od_graph_def = tf.GraphDef()
  
          with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
              serialized_graph = fid.read()
              od_graph_def.ParseFromString(serialized_graph)
              tf.import_graph_def(od_graph_def, name='')
              sess = tf.Session(graph=detection_graph)
  
      image_tensor = detection_graph.get_tensor_by_name('image_tensor:0')
  
      detection_boxes = detection_graph.get_tensor_by_name('detection_boxes:0')
  
      detection_scores = detection_graph.get_tensor_by_name('detection_scores:0')
      detection_classes = detection_graph.get_tensor_by_name('detection_classes:0')
  
      num_detections = detection_graph.get_tensor_by_name('num_detections:0')
  
      c = 0
  
  
      while True:
          labels = {"ace" : 0, "king": 0, "queen": 0, "jack": 0, "ten": 0, "nine": 0, "eight": 0,"seven": 0, "six": 0, "five": 0, "four":0, "three": 0, "two": 0}
  
          with warnings.catch_warnings():
              warnings.filterwarnings("ignore",category=FutureWarning)
              screenshot=ImageGrab.grab(bbox=(42,42, GetSystemMetrics(0),GetSystemMetrics(1)))
              screenshot.save(IMAGE_NAME)
  
  
          image = cv2.imread(PATH_TO_IMAGE)
          image_expanded = np.expand_dims(image, axis=0)
  
          (boxes, scores, classes, num) = sess.run(
              [detection_boxes, detection_scores, detection_classes, num_detections],
              feed_dict={image_tensor: image_expanded})
  
          data = [category_index.get(value) for index,value in enumerate(classes[0]) if scores[0,index] &gt; 0.9]
  
  
  
          for ch in data:
                          if ch['name'] == "ace":
                              labels["ace"] += 1
                          elif ch['name'] == "king":
                              labels["king"] += 1
                          elif ch['name'] == "queen":
                              labels["queen"] += 1
                          elif ch['name'] == "jack":
                              labels["jack"] += 1
                          elif ch['name'] == "ten":
                              labels["ten"] += 1
                          elif ch['name'] == "nine":
                              labels["nine"] += 1
                          elif ch['name'] == "eight":
                              labels["eight"] += 1
                          elif ch['name'] == "seven":
                              labels["seven"] += 1
                          elif ch['name'] == "six":
                              labels["six"] += 1
                          elif ch['name'] == "five":
                              labels["five"] += 1
                          elif ch['name'] == "four":
                              labels["four"] += 1
                          elif ch['name'] == "three":
                              labels["three"] += 1
                          elif ch['name'] == "two":
                              labels["two"] += 1
  
          print(UpdateCounter(labels, c))
  </code></pre>
  
  <p>Pls how can i fix this? I need to show the counter only when new cards get recognised, and i need to fix also the bad matchup that the program gets.</p>