title: How can you correlate logs with APM?
body: <p>I have an application that will write logs to elasticsearch using serilog. I configured the APM server using docker-compose. Once I start the application up and perform an operation (navigate through pages in the browser), then close the application. Those logs are then recorded to elasticsearch. I came across this <a href="https://www.elastic.co/blog/how-to-easily-correlate-logs-apm-traces-for-better-observability-elastic-stack" rel="nofollow noreferrer">article</a> that talks about correlating logs with APM. I selected a few steps to follow since I am not using python in this application, and noticed that there are transactions that are inside of APM.</p>
  <p><strong>With these transactions, how I would I be able to correlate the logs to each other. In other words, how can I tie these logs together is there a unique variable/id/key that will tie all the logs that were recorded in one single transaction (when I started the application, performed operations, then closed the application)?</strong></p>
  <p>When I looked into each of the transactions, I noticed that they have a transcation_id and a trace_id. But, they are changing per each operation that I am performing. <strong>I am wanting to know if it is possible and if it is, how can I gather all the logs that pertain to that single transaction?</strong> For instance, if I query by a single id, then all of those logs will be returned.</p>
  <p><a href="https://i.stack.imgur.com/DdyiY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DdyiY.png" alt="enter image description here" /></a></p>
  <p><code>docker-compose.yml</code></p>
  <pre><code>version: '2.2'
  services:
    apm-server:
      image: docker.elastic.co/apm/apm-server:7.13.0
      depends_on:
        elasticsearch:
          condition: service_healthy
        kibana:
          condition: service_healthy
      cap_add: [&quot;CHOWN&quot;, &quot;DAC_OVERRIDE&quot;, &quot;SETGID&quot;, &quot;SETUID&quot;]
      cap_drop: [&quot;ALL&quot;]
      ports:
      - 8200:8200
      networks:
      - elastic
      command: &gt;
         apm-server -e
           -E apm-server.rum.enabled=true
           -E setup.kibana.host=kibana:5601
           -E setup.template.settings.index.number_of_replicas=0
           -E apm-server.kibana.enabled=true
           -E apm-server.kibana.host=kibana:5601
           -E output.elasticsearch.hosts=[&quot;elasticsearch:9200&quot;]
      healthcheck:
        interval: 10s
        retries: 12
        test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/
  
    elasticsearch:
      image: docker.elastic.co/elasticsearch/elasticsearch:7.13.0
      environment:
      - bootstrap.memory_lock=true
      - cluster.name=docker-cluster
      - cluster.routing.allocation.disk.threshold_enabled=false
      - discovery.type=single-node
      - ES_JAVA_OPTS=-XX:UseAVX=2 -Xms1g -Xmx1g
      ulimits:
        memlock:
          hard: -1
          soft: -1
      volumes:
      - esdata:/usr/share/elasticsearch/data
      ports:
      - 9200:9200
      networks:
      - elastic
      healthcheck:
        interval: 20s
        retries: 10
        test: curl -s http://localhost:9200/_cluster/health | grep -vq '&quot;status&quot;:&quot;red&quot;'
  
    kibana:
      image: docker.elastic.co/kibana/kibana:7.13.0
      depends_on:
        elasticsearch:
          condition: service_healthy
      environment:
        ELASTICSEARCH_URL: http://elasticsearch:9200
        ELASTICSEARCH_HOSTS: http://elasticsearch:9200
      ports:
      - 5601:5601
      networks:
      - elastic
      healthcheck:
        interval: 10s
        retries: 20
        test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:5601/api/status
  
  volumes:
    esdata:
      driver: local
  
  networks:
    elastic:
      driver: bridge
  </code></pre>
  <hr />
  <p><strong>UPDATED</strong></p>
  <p>After looking into the <a href="https://www.elastic.co/guide/en/apm/agent/dotnet/current/serilog.html" rel="nofollow noreferrer">documentation</a> for <code>Elastic.Apm.SerilogEnricher</code>, I went ahead and included it to my <code>Startup.cs</code> file and my <code>Program.cs</code> file. Just wanted to double check that I am incorporating it correctly.</p>
  <p><code>Startup.cs</code>:</p>
  <pre><code>namespace CustomerSimulatorApp
  {
      public class Startup
      {
          public Startup(IConfiguration configuration)
          {
              Configuration = configuration;
  
              var logger = new LoggerConfiguration()
                  .Enrich.WithElasticApmCorrelationInfo()
                  .WriteTo.Console(outputTemplate: &quot;[{ElasticApmTraceId} {ElasticApmTransactionId} {Message:lj} {NewLine}{Exception}&quot;)
                  .CreateLogger();
          }
  
          public IConfiguration Configuration { get; }
  
          // This method gets called by the runtime. Use this method to add services to the container.
          public void ConfigureServices(IServiceCollection services)
          {
              services.AddControllersWithViews();
  
              // create a new node instance
              var node = new Uri(&quot;http://localhost:9200&quot;);
              // settings instance for the node
              var settings = new ConnectionSettings(node);
              settings.DefaultFieldNameInferrer(p =&gt; p);
  
              services.AddSingleton&lt;IElasticClient&gt;(new ElasticClient(settings));
          }
  
          // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
          public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
          {
              app.UseAllElasticApm(Configuration);
  
              if (env.IsDevelopment())
              {
                  app.UseDeveloperExceptionPage();
              }
              else
              {
                  app.UseExceptionHandler(&quot;/Home/Error&quot;);
                  // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                  app.UseHsts();
              }
  
              app.UseHttpsRedirection();
              app.UseStaticFiles();
  
              app.UseRouting();
  
              app.UseAuthorization();
  
              app.UseEndpoints(endpoints =&gt;
              {
                  endpoints.MapControllerRoute(
                      name: &quot;default&quot;,
                      pattern: &quot;{controller=Home}/{action=Index}/{id?}&quot;);
              });
          }
      }
  }
  </code></pre>
  <p><code>Program.cs</code></p>
  <pre><code>namespace CustomerSimulatorApp
  {
      public class Program
      {
          public static void Main(string[] args)
          {
              CreateHostBuilder(args).Build().Run();
          }
  
  
          public static IHostBuilder CreateHostBuilder(string[] args) =&gt;
              Host.CreateDefaultBuilder(args)
                  .UseSerilog((context, configuration) =&gt;
                  {
                      configuration.Enrich.FromLogContext()
                          .Enrich.WithElasticApmCorrelationInfo()
                          .Enrich.WithMachineName()
                          .WriteTo.Console()
                          .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(context.Configuration[&quot;ElasticConfiguration:Uri&quot;]))
                          {
                              IndexFormat = $&quot;{context.Configuration[&quot;ApplicationName&quot;]}-logs-{context.HostingEnvironment.EnvironmentName?.ToLower().Replace(&quot;.&quot;, &quot;-&quot;)}-{DateTime.UtcNow:yyyy-MM}&quot;,
                              AutoRegisterTemplate = true
                          })
                          .Enrich.WithProperty(&quot;Environment&quot;, context.HostingEnvironment.EnvironmentName)
                          .ReadFrom.Configuration(context.Configuration);
                  })
                  .ConfigureWebHostDefaults(webBuilder =&gt;
                  {
                      webBuilder.UseStartup&lt;Startup&gt;();
                  })
                  .UseAllElasticApm();
      }
  }
  </code></pre>
  <p>I noticed when I ran the program and performed operations on the browser then checking APM that the trace.id &amp; transaction.id are still changing so I am not able to correlate this single transaction that I performed on the browser with the logs. Did I implement the <code>Elastic.Apm.SerilogEnricher</code> incorrectly above?</p>
  <p><a href="https://i.stack.imgur.com/arWmT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/arWmT.png" alt="enter image description here" /></a></p>
  <p>Different ID's (there are more diff. ones but do not want to expand it with screenshots)</p>
  <p><a href="https://i.stack.imgur.com/RXaAe.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/RXaAe.png" alt="enter image description here" /></a></p>
  <p>They all change per page redirect so I am unable to gather the logs from a single ID.</p>
  <p>This is what I see on the console as well with the updated startup.cs and program.cs files:</p>
  <p><a href="https://i.stack.imgur.com/ji6DX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ji6DX.png" alt="enter image description here" /></a></p>
  <p>I eventually shut down the program:</p>
  <p><a href="https://i.stack.imgur.com/FwABI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/FwABI.png" alt="enter image description here" /></a></p>