title: How to ignore duplicate lines when processing ip addresses from multiple files?
body: <p>I am writing a python script to do ping tests.</p>
  <ol>
  <li>Based on circuit_id.txt, I will search from testfile1.txt and testfile2.txt</li>
  <li>If matched, extract IP addresses from testfile1.txt and testfile2.txt</li>
  <li>Use the extracted IPs to do ping tests</li>
  <li>In reality, there are hundreds of lines in each file.</li>
  </ol>
  <p>Problems:<br />
  It duplicates pinging the same IP address. I want each IP address to be pinged once.</p>
  <p>circuit_id.txt:</p>
  <pre><code>11111
  22222
  </code></pre>
  <p>testfile1.txt:</p>
  <pre><code>1.1.1.1 circuit ID: 11111 description: ping test
  1.1.1.1 circuit ID: 11111 description: public ip
  1.1.1.1 circuit ID: 11111 description: secure dns
  </code></pre>
  <p>testfile2.txt:</p>
  <pre><code>8.8.8.8 circuit ID: 22222 description: ping test
  8.8.8.8 circuit ID: 22222 description: public ip
  8.8.8.8 circuit ID: 22222 description: google dns
  </code></pre>
  <p>...
  ...</p>
  <p>Results:</p>
  <pre><code>11111
  PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
  64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=5.15 ms
  
  --- 1.1.1.1 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 5.151/5.151/5.151/0.000 ms
  
  =============================================================
  
  11111
  PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
  64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=4.91 ms
  
  --- 1.1.1.1 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 4.913/4.913/4.913/0.000 ms
  
  =============================================================
  
  11111
  PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
  64 bytes from 1.1.1.1: icmp_seq=1 ttl=58 time=5.08 ms
  
  --- 1.1.1.1 ping statistics ---
  1 packets transmitted, 1 received, 0% packet loss, time 0ms
  rtt min/avg/max/mdev = 5.078/5.078/5.078/0.000 ms
  </code></pre>
  <p>And my code:</p>
  <pre><code>import glob
  import os
  import re
  
  os.system(&quot;clear&quot;)
  
  def getping(ip):
      response = os.system(&quot;ping -c 1 &quot; + ip)
  
  with open('circuit_id.txt','r') as f1:      
      cid = f1.read().splitlines()
  
  for cidnum in cid:
      for filename in glob.glob('testfile[1-2].txt'):
          with open(filename) as f2:
              for linef2 in f2:
                  for part in linef2.split():
                      if cidnum in part:
                          pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})') 
                          lst=[] 
                          lst.append(pattern.search(linef2)[0]) 
                          print(cidnum) #print the line number                
                          for ip in lst:
                   
                              getping(ip)
  
                              print(&quot;\n=============================================================\n&quot;)
  </code></pre>
  <p>I am trying to use Gino's suggestion for using <code>set</code> and after googling on how to use set, it still produces duplicate ping results. Pls help point out my mistakes. Here are the codes with <code>set</code>:</p>
  <pre><code>import glob
  import os
  import re
  
  os.system(&quot;clear&quot;)
  
  def getping(ip):
      response = os.system(&quot;ping -c 1 &quot; + ip)
  
  
  with open('circuit_id.txt','r') as f1:      
      cid = f1.read().splitlines()
  
  
  for cidnum in cid:
      for filename in glob.glob('testfile[1-2].txt'):
          with open(filename) as f2:
              for linef2 in f2:
                  for part in linef2.split():
                      if cidnum in part:
                          pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})') 
                     
                          set1= set() #using set here
                          set1.add(pattern.search(linef2)[0])
  
                          print(cidnum) #print the line number                
                          for ip in set1:
                   
                              getping(ip)
  
                              print(&quot;\n=============================================================\n&quot;)
  </code></pre>