title: Apache HttpClient, User-Agent, and corporate proxy
body: <p>This is somewhat of a speculative question in that the answer may not be apparent in the info I have available, but I am hoping that someone with sufficient experience will recognize a likely answer based on common practices for corporate proxies.</p>
  
  <p>I work (not as a software developer) behind a corporate proxy.  In my spare time I was messing around with a Java program I'm developing.  This program needs to make a few very simple HTTP GET requests, and I'm using Apache HttpClient for that.  I was concerned at first about whether or not I'd make it through the proxy server.  In our web browsers, the proxy server is simple entered into the network settings... no authentication needed.  So, I added the following to my Java program:</p>
  
  <pre><code>myClient.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, MY_PROXY);
  </code></pre>
  
  <p>Sure enough, it worked!  However, I had another concern.  The HTTP requests coming from my program probably had some strange User-Agent specified (I've since confirmed this is the case), and I did not want them to ever trigger any sort of suspicion in automated or manual packet inspections.  So I said to myself, "why not just set the User-Agent header to be the same as the browser on this machine?"</p>
  
  <pre><code>myClient.getParams().setParameter(CoreProtocolPNames.USER_AGENT, BROWSER_AGENT);
  </code></pre>
  
  <p>Here is where it gets weird.  If the <code>BROWSER_AGENT</code> string above is set to <strong>exactly</strong> the same value as the corporate supplied browser on my machine (either IE or FF), I get an "authentication failed, missing credentials" type error message returned from the corporate proxy server.  But, if I set the User-Agent header to something generic, like say <code>Mozilla 5.0</code> or even a totally bogus string, or even an <strong>empty</strong> string, it all works fine!  The parts that confuse me are:</p>
  
  <ul>
  <li>When User-Agent is set to the same as my browser (a long complex string), I "fail authentication" somehow, which makes no sense since in the real browser I provide no authentication information (unless it comes from some pre-installed certificate maybe?)</li>
  <li>If the corporation requires authentication for any requests sent to the proxy server on port 80, then how come they let random User-Agent strings get through?  Oversight?  Some other reason I can't comprehend?</li>
  </ul>
  
  <p>Hopefully this question is not too speculative to be deemed constructive.  I'd love to hear from people with experience in this area.  Thanks.</p>