{"action":"created","issue":{"url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/519","repository_url":"https://api.github.com/repos/Azure/service-fabric-issues","labels_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/519/labels{/name}","comments_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/519/comments","events_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/519/events","html_url":"https://github.com/Azure/service-fabric-issues/issues/519","id":264336173,"number":519,"title":"FabricDCA Fails to create container in blob storage, Exception occured while creating an object of type FabricDCA.AzureBlobEtwCsvUploader","user":{"login":"kensykora","id":64756,"avatar_url":"https://avatars3.githubusercontent.com/u/64756?v=4","gravatar_id":"","url":"https://api.github.com/users/kensykora","html_url":"https://github.com/kensykora","followers_url":"https://api.github.com/users/kensykora/followers","following_url":"https://api.github.com/users/kensykora/following{/other_user}","gists_url":"https://api.github.com/users/kensykora/gists{/gist_id}","starred_url":"https://api.github.com/users/kensykora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kensykora/subscriptions","organizations_url":"https://api.github.com/users/kensykora/orgs","repos_url":"https://api.github.com/users/kensykora/repos","events_url":"https://api.github.com/users/kensykora/events{/privacy}","received_events_url":"https://api.github.com/users/kensykora/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"mattrowmsft","id":11323503,"avatar_url":"https://avatars2.githubusercontent.com/u/11323503?v=4","gravatar_id":"","url":"https://api.github.com/users/mattrowmsft","html_url":"https://github.com/mattrowmsft","followers_url":"https://api.github.com/users/mattrowmsft/followers","following_url":"https://api.github.com/users/mattrowmsft/following{/other_user}","gists_url":"https://api.github.com/users/mattrowmsft/gists{/gist_id}","starred_url":"https://api.github.com/users/mattrowmsft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattrowmsft/subscriptions","organizations_url":"https://api.github.com/users/mattrowmsft/orgs","repos_url":"https://api.github.com/users/mattrowmsft/repos","events_url":"https://api.github.com/users/mattrowmsft/events{/privacy}","received_events_url":"https://api.github.com/users/mattrowmsft/received_events","type":"User","site_admin":false},"assignees":[{"login":"mattrowmsft","id":11323503,"avatar_url":"https://avatars2.githubusercontent.com/u/11323503?v=4","gravatar_id":"","url":"https://api.github.com/users/mattrowmsft","html_url":"https://github.com/mattrowmsft","followers_url":"https://api.github.com/users/mattrowmsft/followers","following_url":"https://api.github.com/users/mattrowmsft/following{/other_user}","gists_url":"https://api.github.com/users/mattrowmsft/gists{/gist_id}","starred_url":"https://api.github.com/users/mattrowmsft/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattrowmsft/subscriptions","organizations_url":"https://api.github.com/users/mattrowmsft/orgs","repos_url":"https://api.github.com/users/mattrowmsft/repos","events_url":"https://api.github.com/users/mattrowmsft/events{/privacy}","received_events_url":"https://api.github.com/users/mattrowmsft/received_events","type":"User","site_admin":false},{"login":"anmolah","id":11351400,"avatar_url":"https://avatars2.githubusercontent.com/u/11351400?v=4","gravatar_id":"","url":"https://api.github.com/users/anmolah","html_url":"https://github.com/anmolah","followers_url":"https://api.github.com/users/anmolah/followers","following_url":"https://api.github.com/users/anmolah/following{/other_user}","gists_url":"https://api.github.com/users/anmolah/gists{/gist_id}","starred_url":"https://api.github.com/users/anmolah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anmolah/subscriptions","organizations_url":"https://api.github.com/users/anmolah/orgs","repos_url":"https://api.github.com/users/anmolah/repos","events_url":"https://api.github.com/users/anmolah/events{/privacy}","received_events_url":"https://api.github.com/users/anmolah/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2017-10-10T18:48:24Z","updated_at":"2017-10-13T14:02:15Z","closed_at":null,"author_association":"NONE","body":"As of the last few days I've started running into an issue when creating new service fabric cluster deployments to Azure. The nodes when they start up report unhealthy due to the fact that they cannot send logs to Azure storage. I'm fairly certain as early as last week I was not encountering this issue.\r\n\r\nService Fabric Version: `6.0.211.9494`\r\n\r\nPlease let me know if this is an inappropriate place for this, or if I should work with Azure support to remedy this instead.\r\n\r\nHere is the unhealthy evaluation of the node message:\r\n\r\n```\r\n\tError event: SourceId='FabricDCA', Property='DataCollectionAgent'.\r\nException occured while creating an object of type FabricDCA.AzureBlobEtwCsvUploader (assembly AzureFileUploader, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) for creating consumer AzureBlobServiceFabricEtw.\r\nException occured while creating an object of type FabricDCA.AzureTableQueryableEventUploader (assembly AzureTableUploader, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) for creating consumer AzureTableServiceFabricEtwQueryable.\r\n```\r\n\r\nWhen i remote into the VM and go to the event viewer, I can find more detailed logs that seem to indicate it's a pretty deep exeception that's being thrown. I'm not confident if it's related to the data I'm configuring or not -- I've validated that the values I'm passing it are correct, so I don't think that is it.\r\n\r\nThe error messages indicate it can't convert base64 string to a byte array, but I'm unclear what the source of the data it's trying to convert is. I've checked the D:\\SvcFab\\clusterManifest.xml and D:\\SvcFab\\FabricHostSettings.xml -- both have proper StoreConnectionString parameters:\r\n\r\n```\r\n<Parameter Name=\"StoreConnectionString\" Value=\"xstore:BlobEndpoint=https://skmskstglog.blob.core.windows.net/;TableEndpoint=https://skmskstglog.table.core.windows.net/;AccountName=skmskstglog;ProtectedAccountKeyName=StorageAccountKey1\"/>\r\n```\r\n\r\nThe protected settings for my VMSS have the StorageAccountKey1 set and as far as I can tell it's populated just fine (I put the same value into my ARM template output to verify the key is correct).\r\n\r\nI'm not sure how to keep troubleshooting this, so hopefully you can help me figure out what's going on.\r\n\r\nEvent Viewer Logs from Application and Services Logs -> Microsoft-Service Fabric -> Admin:\r\n\r\nThese events all happen within 3 seconds of each other\r\n\r\n## 1st Error:\r\n\r\n```\r\nError creating container fabriclogs-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, account skmskstglog.\r\nSystem.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureFileUploader.CreateContainer(Object context)\r\n   at System.Fabric.Dca.Utility.PerformWithRetries[T](Action`1 worker, T context, RetriableOperationExceptionHandler exceptionHandler, Int32 initialRetryIntervalMs, Int32 maxRetryCount, Int32 maxRetryIntervalMs)\r\n   at FabricDCA.AzureFileUploader..ctor(ExtensionsEvents traceSource, String logSourceId, String folderName, String destinationPath, String workFolder, StorageConnection storageConnection, String containerName, TimeSpan uploadIntervalMinutes, TimeSpan fileSyncIntervalInMinutes, TimeSpan blobDeletionAgeMinutes, String fabricNodeInstanceName, String deploymentId)\r\n```\r\n\r\n## 2nd Error:\r\n\r\n```\r\nException occured while creating an object of type FabricDCA.AzureBlobEtwCsvUploader (assembly AzureFileUploader, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) for creating consumer AzureBlobServiceFabricEtw. Exception information: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: AzureFileUploader could not be constructed. ---> System.InvalidOperationException: Error creating container fabriclogs-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, account skmskstglog. ---> System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureFileUploader.CreateContainer(Object context)\r\n   at System.Fabric.Dca.Utility.PerformWithRetries[T](Action`1 worker, T context, RetriableOperationExceptionHandler exceptionHandler, Int32 initialRetryIntervalMs, Int32 maxRetryCount, Int32 maxRetryIntervalMs)\r\n   at FabricDCA.AzureFileUploader..ctor(ExtensionsEvents traceSource, String logSourceId, String folderName, String destinationPath, String workFolder, StorageConnection storageConnection, String containerName, TimeSpan uploadIntervalMinutes, TimeSpan fileSyncIntervalInMinutes, TimeSpan blobDeletionAgeMinutes, String fabricNodeInstanceName, String deploymentId)\r\n   --- End of inner exception stack trace ---\r\n   at FabricDCA.AzureFileUploader..ctor(ExtensionsEvents traceSource, String logSourceId, String folderName, String destinationPath, String workFolder, StorageConnection storageConnection, String containerName, TimeSpan uploadIntervalMinutes, TimeSpan fileSyncIntervalInMinutes, TimeSpan blobDeletionAgeMinutes, String fabricNodeInstanceName, String deploymentId)\r\n   at FabricDCA.EtwCsvUploadWorker..ctor(EtwCsvUploadWorkerParameters initParam)\r\n   --- End of inner exception stack trace ---\r\n   at FabricDCA.EtwCsvUploadWorker..ctor(EtwCsvUploadWorkerParameters initParam)\r\n   at FabricDCA.AzureBlobEtwCsvUploader..ctor(ConsumerInitializationParameters initParam)\r\n   --- End of inner exception stack trace ---\r\n   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)\r\n   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)\r\n   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)\r\n   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)\r\n   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)\r\n   at FabricDCA.ConsumerFactory.CreateConsumer(String consumerInstance, ConsumerInitializationParameters initParam, String moduleName, String typeName).\r\n```\r\n\r\n3rd Error:\r\n\r\n```\r\nDue to an error in table creation ETW traces will not be uploaded to Azure table storage.\r\nSystem.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureTableQueryableEventUploader.CreateTables()\r\n   at FabricDCA.AzureTableQueryableEventUploader..ctor(ConsumerInitializationParameters initParam)\r\n```\r\n\r\n4th Error:\r\n```\r\nException occured while creating an object of type FabricDCA.AzureTableQueryableEventUploader (assembly AzureTableUploader, Version=6.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35) for creating consumer AzureTableServiceFabricEtwQueryable. Exception information: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Due to an error in table creation ETW traces will not be uploaded to Azure table storage. ---> System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureTableQueryableEventUploader.CreateTables()\r\n   at FabricDCA.AzureTableQueryableEventUploader..ctor(ConsumerInitializationParameters initParam)\r\n   --- End of inner exception stack trace ---\r\n   at FabricDCA.AzureTableQueryableEventUploader..ctor(ConsumerInitializationParameters initParam)\r\n   --- End of inner exception stack trace ---\r\n   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)\r\n   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)\r\n   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)\r\n   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)\r\n   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)\r\n   at FabricDCA.ConsumerFactory.CreateConsumer(String consumerInstance, ConsumerInitializationParameters initParam, String moduleName, String typeName).\r\n```\r\n\r\n5th Error:\r\n```\r\nKernel crash upload is configured but failed to get kernel crash dump folder.\r\n```\r\n\r\n6th Error:\r\n```\r\nError creating container fabriccrashdumps-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, account skmskstglog.\r\nSystem.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureFileUploader.CreateContainer(Object context)\r\n   at System.Fabric.Dca.Utility.PerformWithRetries[T](Action`1 worker, T context, RetriableOperationExceptionHandler exceptionHandler, Int32 initialRetryIntervalMs, Int32 maxRetryCount, Int32 maxRetryIntervalMs)\r\n   at FabricDCA.AzureFileUploader..ctor(ExtensionsEvents traceSource, String logSourceId, String folderName, String destinationPath, String workFolder, StorageConnection storageConnection, String containerName, TimeSpan uploadIntervalMinutes, TimeSpan fileSyncIntervalInMinutes, TimeSpan blobDeletionAgeMinutes, String fabricNodeInstanceName, String deploymentId)\r\n```\r\n\r\n7th Error:\r\n```\r\nFailed to configure upload to Azure blob.  Storage account: skmskstglog, container: fabriccrashdumps-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, Local folder path: D:\\SvcFab\\Log\\CrashDumps.\r\n```\r\n\r\n8th Error:\r\n```\r\nError creating container fabriccounters-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, account skmskstglog.\r\nSystem.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. \r\n   at System.Convert.FromBase64_Decode(Char* startInputPtr, Int32 inputLength, Byte* startDestPtr, Int32 destLength)\r\n   at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)\r\n   at System.Convert.FromBase64CharArray(Char[] inArray, Int32 offset, Int32 length)\r\n   at System.Fabric.Dca.Utility.SecureStringToByteArray(SecureString secureString)\r\n   at FabricDCA.AzureUtility.GetCloudStorageAccount(StorageConnection storageConnection, CloudStorageAccount& storageAccount)\r\n   at FabricDCA.AzureFileUploader.CreateContainer(Object context)\r\n   at System.Fabric.Dca.Utility.PerformWithRetries[T](Action`1 worker, T context, RetriableOperationExceptionHandler exceptionHandler, Int32 initialRetryIntervalMs, Int32 maxRetryCount, Int32 maxRetryIntervalMs)\r\n   at FabricDCA.AzureFileUploader..ctor(ExtensionsEvents traceSource, String logSourceId, String folderName, String destinationPath, String workFolder, StorageConnection storageConnection, String containerName, TimeSpan uploadIntervalMinutes, TimeSpan fileSyncIntervalInMinutes, TimeSpan blobDeletionAgeMinutes, String fabricNodeInstanceName, String deploymentId)\r\n```\r\n\r\n9th Error:\r\n```\r\nFailed to configure upload to Azure blob.  Storage account: skmskstglog, container: fabriccounters-cc636ce4-d115-41b4-bfd1-42c3fb0d00f2, Local folder path: D:\\SvcFab\\Log\\PerformanceCounters_ServiceFabricPerfCounter.\r\n```"},"comment":{"url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/comments/336461089","html_url":"https://github.com/Azure/service-fabric-issues/issues/519#issuecomment-336461089","issue_url":"https://api.github.com/repos/Azure/service-fabric-issues/issues/519","id":336461089,"user":{"login":"kensykora","id":64756,"avatar_url":"https://avatars3.githubusercontent.com/u/64756?v=4","gravatar_id":"","url":"https://api.github.com/users/kensykora","html_url":"https://github.com/kensykora","followers_url":"https://api.github.com/users/kensykora/followers","following_url":"https://api.github.com/users/kensykora/following{/other_user}","gists_url":"https://api.github.com/users/kensykora/gists{/gist_id}","starred_url":"https://api.github.com/users/kensykora/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kensykora/subscriptions","organizations_url":"https://api.github.com/users/kensykora/orgs","repos_url":"https://api.github.com/users/kensykora/repos","events_url":"https://api.github.com/users/kensykora/events{/privacy}","received_events_url":"https://api.github.com/users/kensykora/received_events","type":"User","site_admin":false},"created_at":"2017-10-13T14:02:14Z","updated_at":"2017-10-13T14:02:14Z","author_association":"NONE","body":"Hey guys,\r\n\r\nAny idea on this? I think I am going to have to disable diagnostics which is really unfortunate. Should I open an Azure support ticket? How can I troubleshoot further (e.g., can I try to somehow get the SecureString that it's using which is causing the error to investigate the data itself?)"}}
{"actor":{"display_login":"kensykora"}}