text:
#==============================================================================#
# Firehose.jl
#
# This file is generated from:
# https://github.com/aws/aws-sdk-js/blob/master/apis/firehose-2015-08-04.normal.json
#==============================================================================#

__precompile__()

module Firehose

using AWSCore

"""
    using AWSSDK.Firehose.create_delivery_stream
    create_delivery_stream([::AWSConfig], arguments::Dict)
    create_delivery_stream([::AWSConfig]; DeliveryStreamName=, <keyword arguments>)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "CreateDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "CreateDeliveryStream", DeliveryStreamName=, <keyword arguments>)

# CreateDeliveryStream Operation

Creates a Kinesis Data Firehose delivery stream.

By default, you can create up to 50 delivery streams per AWS Region.

This is an asynchronous operation that immediately returns. The initial status of the delivery stream is `CREATING`. After the delivery stream is created, its status is `ACTIVE` and it now accepts data. Attempts to send data to a delivery stream that is not in the `ACTIVE` state cause an exception. To check the state of a delivery stream, use [DescribeDeliveryStream](@ref).

A Kinesis Data Firehose delivery stream can be configured to receive records directly from providers using [PutRecord](@ref) or [PutRecordBatch](@ref), or it can be configured to use an existing Kinesis stream as its source. To specify a Kinesis data stream as input, set the `DeliveryStreamType` parameter to `KinesisStreamAsSource`, and provide the Kinesis stream Amazon Resource Name (ARN) and role ARN in the `KinesisStreamSourceConfiguration` parameter.

A delivery stream is configured with a single destination: Amazon S3, Amazon ES, Amazon Redshift, or Splunk. You must specify only one of the following destination configuration parameters: **ExtendedS3DestinationConfiguration**, **S3DestinationConfiguration**, **ElasticsearchDestinationConfiguration**, **RedshiftDestinationConfiguration**, or **SplunkDestinationConfiguration**.

When you specify **S3DestinationConfiguration**, you can also provide the following optional values: **BufferingHints**, **EncryptionConfiguration**, and **CompressionFormat**. By default, if no **BufferingHints** value is provided, Kinesis Data Firehose buffers data up to 5 MB or for 5 minutes, whichever condition is satisfied first. **BufferingHints** is a hint, so there are some cases where the service cannot adhere to these conditions strictly. For example, record boundaries might be such that the size is a little over or under the configured buffering size. By default, no encryption is performed. We strongly recommend that you enable encryption to ensure secure data storage in Amazon S3.

A few notes about Amazon Redshift as a destination:

*   An Amazon Redshift destination requires an S3 bucket as intermediate location. Kinesis Data Firehose first delivers data to Amazon S3 and then uses `COPY` syntax to load data into an Amazon Redshift table. This is specified in the **RedshiftDestinationConfiguration.S3Configuration** parameter.

*   The compression formats `SNAPPY` or `ZIP` cannot be specified in `RedshiftDestinationConfiguration.S3Configuration` because the Amazon Redshift `COPY` operation that reads from the S3 bucket doesn't support these compression formats.

*   We strongly recommend that you use the user name and password you provide exclusively with Kinesis Data Firehose, and that the permissions for the account are restricted for Amazon Redshift `INSERT` permissions.

Kinesis Data Firehose assumes the IAM role that is configured as part of the destination. The role should allow the Kinesis Data Firehose principal to assume the role, and the role should have permissions that allow the service to deliver the data. For more information, see [Grant Kinesis Data Firehose Access to an Amazon S3 Destination](http://docs.aws.amazon.com/firehose/latest/dev/controlling-access.html#using-iam-s3) in the *Amazon Kinesis Data Firehose Developer Guide*.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream. This name must be unique per AWS account in the same AWS Region. If the delivery streams are in different accounts or different Regions, you can have multiple delivery streams with the same name.

## `DeliveryStreamType = "DirectPut" or "KinesisStreamAsSource"`
The delivery stream type. This parameter can be one of the following values:

*   `DirectPut`: Provider applications access the delivery stream directly.

*   `KinesisStreamAsSource`: The delivery stream uses a Kinesis data stream as a source.

## `KinesisStreamSourceConfiguration = [ ... ]`
When a Kinesis data stream is used as the source for the delivery stream, a [KinesisStreamSourceConfiguration](@ref) containing the Kinesis data stream Amazon Resource Name (ARN) and the role ARN for the source stream.
```
 KinesisStreamSourceConfiguration = [
        "KinesisStreamARN" => <required> ::String,
        "RoleARN" => <required> ::String
    ]
```

## `S3DestinationConfiguration = [ ... ]`
[Deprecated] The destination in Amazon S3. You can specify only one destination.
```
 S3DestinationConfiguration = [
        "RoleARN" => <required> ::String,
        "BucketARN" => <required> ::String,
        "Prefix" =>  ::String,
        "BufferingHints" =>  [
            "SizeInMBs" =>  ::Int,
            "IntervalInSeconds" =>  ::Int
        ],
        "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
        "EncryptionConfiguration" =>  [
            "NoEncryptionConfig" =>  "NoEncryption",
            "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `ExtendedS3DestinationConfiguration = [ ... ]`
The destination in Amazon S3. You can specify only one destination.
```
 ExtendedS3DestinationConfiguration = [
        "RoleARN" => <required> ::String,
        "BucketARN" => <required> ::String,
        "Prefix" =>  ::String,
        "BufferingHints" =>  [
            "SizeInMBs" =>  ::Int,
            "IntervalInSeconds" =>  ::Int
        ],
        "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
        "EncryptionConfiguration" =>  [
            "NoEncryptionConfig" =>  "NoEncryption",
            "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "S3BackupMode" =>  "Disabled" or "Enabled",
        "S3BackupConfiguration" =>  [
            "RoleARN" => <required> ::String,
            "BucketARN" => <required> ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "DataFormatConversionConfiguration" =>  [
            "SchemaConfiguration" =>  [
                "RoleARN" =>  ::String,
                "CatalogId" =>  ::String,
                "DatabaseName" =>  ::String,
                "TableName" =>  ::String,
                "Region" =>  ::String,
                "VersionId" =>  ::String
            ],
            "InputFormatConfiguration" =>  ["Deserializer" =>  [
                    "OpenXJsonSerDe" =>  [
                        "ConvertDotsInJsonKeysToUnderscores" =>  ::Bool,
                        "CaseInsensitive" =>  ::Bool,
                        "ColumnToJsonKeyMappings" =>  ::Dict{String,String}
                    ],
                    "HiveJsonSerDe" =>  ["TimestampFormats" =>  [::String, ...]]
                ]],
            "OutputFormatConfiguration" =>  ["Serializer" =>  [
                    "ParquetSerDe" =>  [
                        "BlockSizeBytes" =>  ::Int,
                        "PageSizeBytes" =>  ::Int,
                        "Compression" =>  "UNCOMPRESSED", "GZIP" or "SNAPPY",
                        "EnableDictionaryCompression" =>  ::Bool,
                        "MaxPaddingBytes" =>  ::Int,
                        "WriterVersion" =>  "V1" or "V2"
                    ],
                    "OrcSerDe" =>  [
                        "StripeSizeBytes" =>  ::Int,
                        "BlockSizeBytes" =>  ::Int,
                        "RowIndexStride" =>  ::Int,
                        "EnablePadding" =>  ::Bool,
                        "PaddingTolerance" =>  double,
                        "Compression" =>  "NONE", "ZLIB" or "SNAPPY",
                        "BloomFilterColumns" =>  [::String, ...],
                        "BloomFilterFalsePositiveProbability" =>  double,
                        "DictionaryKeyThreshold" =>  double,
                        "FormatVersion" =>  "V0_11" or "V0_12"
                    ]
                ]],
            "Enabled" =>  ::Bool
        ]
    ]
```

## `RedshiftDestinationConfiguration = [ ... ]`
The destination in Amazon Redshift. You can specify only one destination.
```
 RedshiftDestinationConfiguration = [
        "RoleARN" => <required> ::String,
        "ClusterJDBCURL" => <required> ::String,
        "CopyCommand" => <required> [
            "DataTableName" => <required> ::String,
            "DataTableColumns" =>  ::String,
            "CopyOptions" =>  ::String
        ],
        "Username" => <required> ::String,
        "Password" => <required> ::String,
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3Configuration" => <required> [
            "RoleARN" => <required> ::String,
            "BucketARN" => <required> ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "S3BackupMode" =>  "Disabled" or "Enabled",
        "S3BackupConfiguration" =>  [
            "RoleARN" => <required> ::String,
            "BucketARN" => <required> ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `ElasticsearchDestinationConfiguration = [ ... ]`
The destination in Amazon ES. You can specify only one destination.
```
 ElasticsearchDestinationConfiguration = [
        "RoleARN" => <required> ::String,
        "DomainARN" => <required> ::String,
        "IndexName" => <required> ::String,
        "TypeName" => <required> ::String,
        "IndexRotationPeriod" =>  "NoRotation", "OneHour", "OneDay", "OneWeek" or "OneMonth",
        "BufferingHints" =>  [
            "IntervalInSeconds" =>  ::Int,
            "SizeInMBs" =>  ::Int
        ],
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3BackupMode" =>  "FailedDocumentsOnly" or "AllDocuments",
        "S3Configuration" => <required> [
            "RoleARN" => <required> ::String,
            "BucketARN" => <required> ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `SplunkDestinationConfiguration = [ ... ]`
The destination in Splunk. You can specify only one destination.
```
 SplunkDestinationConfiguration = [
        "HECEndpoint" => <required> ::String,
        "HECEndpointType" => <required> "Raw" or "Event",
        "HECToken" => <required> ::String,
        "HECAcknowledgmentTimeoutInSeconds" =>  ::Int,
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3BackupMode" =>  "FailedEventsOnly" or "AllEvents",
        "S3Configuration" => <required> [
            "RoleARN" => <required> ::String,
            "BucketARN" => <required> ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

# Returns

`CreateDeliveryStreamOutput`

# Exceptions

`InvalidArgumentException`, `LimitExceededException` or `ResourceInUseException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/CreateDeliveryStream)
"""
@inline create_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = create_delivery_stream(aws, args)

@inline create_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "CreateDeliveryStream", args)

@inline create_delivery_stream(args) = create_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.delete_delivery_stream
    delete_delivery_stream([::AWSConfig], arguments::Dict)
    delete_delivery_stream([::AWSConfig]; DeliveryStreamName=)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "DeleteDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "DeleteDeliveryStream", DeliveryStreamName=)

# DeleteDeliveryStream Operation

Deletes a delivery stream and its data.

You can delete a delivery stream only if it is in `ACTIVE` or `DELETING` state, and not in the `CREATING` state. While the deletion request is in process, the delivery stream is in the `DELETING` state.

To check the state of a delivery stream, use [DescribeDeliveryStream](@ref).

While the delivery stream is `DELETING` state, the service might continue to accept the records, but it doesn't make any guarantees with respect to delivering the data. Therefore, as a best practice, you should first stop any applications that are sending records before deleting a delivery stream.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

# Returns

`DeleteDeliveryStreamOutput`

# Exceptions

`ResourceInUseException` or `ResourceNotFoundException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/DeleteDeliveryStream)
"""
@inline delete_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = delete_delivery_stream(aws, args)

@inline delete_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "DeleteDeliveryStream", args)

@inline delete_delivery_stream(args) = delete_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.describe_delivery_stream
    describe_delivery_stream([::AWSConfig], arguments::Dict)
    describe_delivery_stream([::AWSConfig]; DeliveryStreamName=, <keyword arguments>)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "DescribeDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "DescribeDeliveryStream", DeliveryStreamName=, <keyword arguments>)

# DescribeDeliveryStream Operation

Describes the specified delivery stream and gets the status. For example, after your delivery stream is created, call `DescribeDeliveryStream` to see whether the delivery stream is `ACTIVE` and therefore ready for data to be sent to it.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

## `Limit = ::Int`
The limit on the number of destinations to return. You can have one destination per delivery stream.

## `ExclusiveStartDestinationId = ::String`
The ID of the destination to start returning the destination information. Kinesis Data Firehose supports one destination per delivery stream.

# Returns

`DescribeDeliveryStreamOutput`

# Exceptions

`ResourceNotFoundException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/DescribeDeliveryStream)
"""
@inline describe_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = describe_delivery_stream(aws, args)

@inline describe_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "DescribeDeliveryStream", args)

@inline describe_delivery_stream(args) = describe_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.list_delivery_streams
    list_delivery_streams([::AWSConfig], arguments::Dict)
    list_delivery_streams([::AWSConfig]; <keyword arguments>)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "ListDeliveryStreams", arguments::Dict)
    firehose([::AWSConfig], "ListDeliveryStreams", <keyword arguments>)

# ListDeliveryStreams Operation

Lists your delivery streams.

The number of delivery streams might be too large to return using a single call to `ListDeliveryStreams`. You can limit the number of delivery streams returned, using the **Limit** parameter. To determine whether there are more delivery streams to list, check the value of `HasMoreDeliveryStreams` in the output. If there are more delivery streams to list, you can request them by specifying the name of the last delivery stream returned in the call in the `ExclusiveStartDeliveryStreamName` parameter of a subsequent call.

# Arguments

## `Limit = ::Int`
The maximum number of delivery streams to list. The default value is 10.

## `DeliveryStreamType = "DirectPut" or "KinesisStreamAsSource"`
The delivery stream type. This can be one of the following values:

*   `DirectPut`: Provider applications access the delivery stream directly.

*   `KinesisStreamAsSource`: The delivery stream uses a Kinesis data stream as a source.

This parameter is optional. If this parameter is omitted, delivery streams of all types are returned.

## `ExclusiveStartDeliveryStreamName = ::String`
The name of the delivery stream to start the list with.

# Returns

`ListDeliveryStreamsOutput`

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/ListDeliveryStreams)
"""
@inline list_delivery_streams(aws::AWSConfig=default_aws_config(); args...) = list_delivery_streams(aws, args)

@inline list_delivery_streams(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "ListDeliveryStreams", args)

@inline list_delivery_streams(args) = list_delivery_streams(default_aws_config(), args)

"""
    using AWSSDK.Firehose.list_tags_for_delivery_stream
    list_tags_for_delivery_stream([::AWSConfig], arguments::Dict)
    list_tags_for_delivery_stream([::AWSConfig]; DeliveryStreamName=, <keyword arguments>)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "ListTagsForDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "ListTagsForDeliveryStream", DeliveryStreamName=, <keyword arguments>)

# ListTagsForDeliveryStream Operation

Lists the tags for the specified delivery stream. This operation has a limit of five transactions per second per account.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream whose tags you want to list.

## `ExclusiveStartTagKey = ::String`
The key to use as the starting point for the list of tags. If you set this parameter, `ListTagsForDeliveryStream` gets all tags that occur after `ExclusiveStartTagKey`.

## `Limit = ::Int`
The number of tags to return. If this number is less than the total number of tags associated with the delivery stream, `HasMoreTags` is set to `true` in the response. To list additional tags, set `ExclusiveStartTagKey` to the last key in the response.

# Returns

`ListTagsForDeliveryStreamOutput`

# Exceptions

`ResourceNotFoundException`, `InvalidArgumentException` or `LimitExceededException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/ListTagsForDeliveryStream)
"""
@inline list_tags_for_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = list_tags_for_delivery_stream(aws, args)

@inline list_tags_for_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "ListTagsForDeliveryStream", args)

@inline list_tags_for_delivery_stream(args) = list_tags_for_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.put_record
    put_record([::AWSConfig], arguments::Dict)
    put_record([::AWSConfig]; DeliveryStreamName=, Record=)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "PutRecord", arguments::Dict)
    firehose([::AWSConfig], "PutRecord", DeliveryStreamName=, Record=)

# PutRecord Operation

Writes a single data record into an Amazon Kinesis Data Firehose delivery stream. To write multiple data records into a delivery stream, use [PutRecordBatch](@ref). Applications using these operations are referred to as producers.

By default, each delivery stream can take in up to 2,000 transactions per second, 5,000 records per second, or 5 MB per second. If you use [PutRecord](@ref) and [PutRecordBatch](@ref), the limits are an aggregate across these two operations for each delivery stream. For more information about limits and how to request an increase, see [Amazon Kinesis Data Firehose Limits](http://docs.aws.amazon.com/firehose/latest/dev/limits.html).

You must specify the name of the delivery stream and the data record when using [PutRecord](@ref). The data record consists of a data blob that can be up to 1,000 KB in size, and any kind of data. For example, it can be a segment from a log file, geographic location data, website clickstream data, and so on.

Kinesis Data Firehose buffers records before delivering them to the destination. To disambiguate the data blobs at the destination, a common solution is to use delimiters in the data, such as a newline (`\\n`) or some other character unique within the data. This allows the consumer application to parse individual data items when reading the data from the destination.

The `PutRecord` operation returns a `RecordId`, which is a unique string assigned to each record. Producer applications can use this ID for purposes such as auditability and investigation.

If the `PutRecord` operation throws a `ServiceUnavailableException`, back off and retry. If the exception persists, it is possible that the throughput limits have been exceeded for the delivery stream.

Data records sent to Kinesis Data Firehose are stored for 24 hours from the time they are added to a delivery stream as it tries to send the records to the destination. If the destination is unreachable for more than 24 hours, the data is no longer available.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

## `Record = ["Data" => <required> blob]` -- *Required*
The record.

# Returns

`PutRecordOutput`

# Exceptions

`ResourceNotFoundException`, `InvalidArgumentException` or `ServiceUnavailableException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/PutRecord)
"""
@inline put_record(aws::AWSConfig=default_aws_config(); args...) = put_record(aws, args)

@inline put_record(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "PutRecord", args)

@inline put_record(args) = put_record(default_aws_config(), args)

"""
    using AWSSDK.Firehose.put_record_batch
    put_record_batch([::AWSConfig], arguments::Dict)
    put_record_batch([::AWSConfig]; DeliveryStreamName=, Records=)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "PutRecordBatch", arguments::Dict)
    firehose([::AWSConfig], "PutRecordBatch", DeliveryStreamName=, Records=)

# PutRecordBatch Operation

Writes multiple data records into a delivery stream in a single call, which can achieve higher throughput per producer than when writing single records. To write single data records into a delivery stream, use [PutRecord](@ref). Applications using these operations are referred to as producers.

By default, each delivery stream can take in up to 2,000 transactions per second, 5,000 records per second, or 5 MB per second. If you use [PutRecord](@ref) and [PutRecordBatch](@ref), the limits are an aggregate across these two operations for each delivery stream. For more information about limits, see [Amazon Kinesis Data Firehose Limits](http://docs.aws.amazon.com/firehose/latest/dev/limits.html).

Each [PutRecordBatch](@ref) request supports up to 500 records. Each record in the request can be as large as 1,000 KB (before 64-bit encoding), up to a limit of 4 MB for the entire request. These limits cannot be changed.

You must specify the name of the delivery stream and the data record when using [PutRecord](@ref). The data record consists of a data blob that can be up to 1,000 KB in size, and any kind of data. For example, it could be a segment from a log file, geographic location data, website clickstream data, and so on.

Kinesis Data Firehose buffers records before delivering them to the destination. To disambiguate the data blobs at the destination, a common solution is to use delimiters in the data, such as a newline (`\\n`) or some other character unique within the data. This allows the consumer application to parse individual data items when reading the data from the destination.

The [PutRecordBatch](@ref) response includes a count of failed records, **FailedPutCount**, and an array of responses, **RequestResponses**. Each entry in the **RequestResponses** array provides additional information about the processed record. It directly correlates with a record in the request array using the same ordering, from the top to the bottom. The response array always includes the same number of records as the request array. **RequestResponses** includes both successfully and unsuccessfully processed records. Kinesis Data Firehose tries to process all records in each [PutRecordBatch](@ref) request. A single record failure does not stop the processing of subsequent records.

A successfully processed record includes a **RecordId** value, which is unique for the record. An unsuccessfully processed record includes **ErrorCode** and **ErrorMessage** values. **ErrorCode** reflects the type of error, and is one of the following values: `ServiceUnavailable` or `InternalFailure`. **ErrorMessage** provides more detailed information about the error.

If there is an internal server error or a timeout, the write might have completed or it might have failed. If **FailedPutCount** is greater than 0, retry the request, resending only those records that might have failed processing. This minimizes the possible duplicate records and also reduces the total bytes sent (and corresponding charges). We recommend that you handle any duplicates at the destination.

If [PutRecordBatch](@ref) throws **ServiceUnavailableException**, back off and retry. If the exception persists, it is possible that the throughput limits have been exceeded for the delivery stream.

Data records sent to Kinesis Data Firehose are stored for 24 hours from the time they are added to a delivery stream as it attempts to send the records to the destination. If the destination is unreachable for more than 24 hours, the data is no longer available.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

## `Records = [["Data" => <required> blob], ...]` -- *Required*
One or more records.

# Returns

`PutRecordBatchOutput`

# Exceptions

`ResourceNotFoundException`, `InvalidArgumentException` or `ServiceUnavailableException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/PutRecordBatch)
"""
@inline put_record_batch(aws::AWSConfig=default_aws_config(); args...) = put_record_batch(aws, args)

@inline put_record_batch(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "PutRecordBatch", args)

@inline put_record_batch(args) = put_record_batch(default_aws_config(), args)

"""
    using AWSSDK.Firehose.tag_delivery_stream
    tag_delivery_stream([::AWSConfig], arguments::Dict)
    tag_delivery_stream([::AWSConfig]; DeliveryStreamName=, Tags=)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "TagDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "TagDeliveryStream", DeliveryStreamName=, Tags=)

# TagDeliveryStream Operation

Adds or updates tags for the specified delivery stream. A tag is a key-value pair (the value is optional) that you can define and assign to AWS resources. If you specify a tag that already exists, the tag value is replaced with the value that you specify in the request. Tags are metadata. For example, you can add friendly names and descriptions or other types of information that can help you distinguish the delivery stream. For more information about tags, see [Using Cost Allocation Tags](https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html) in the *AWS Billing and Cost Management User Guide*.

Each delivery stream can have up to 50 tags.

This operation has a limit of five transactions per second per account.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream to which you want to add the tags.

## `Tags = [[ ... ], ...]` -- *Required*
A set of key-value pairs to use to create the tags.
```
 Tags = [[
        "Key" => <required> ::String,
        "Value" =>  ::String
    ], ...]
```

# Returns

`TagDeliveryStreamOutput`

# Exceptions

`ResourceNotFoundException`, `ResourceInUseException`, `InvalidArgumentException` or `LimitExceededException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/TagDeliveryStream)
"""
@inline tag_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = tag_delivery_stream(aws, args)

@inline tag_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "TagDeliveryStream", args)

@inline tag_delivery_stream(args) = tag_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.untag_delivery_stream
    untag_delivery_stream([::AWSConfig], arguments::Dict)
    untag_delivery_stream([::AWSConfig]; DeliveryStreamName=, TagKeys=)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "UntagDeliveryStream", arguments::Dict)
    firehose([::AWSConfig], "UntagDeliveryStream", DeliveryStreamName=, TagKeys=)

# UntagDeliveryStream Operation

Removes tags from the specified delivery stream. Removed tags are deleted, and you can't recover them after this operation successfully completes.

If you specify a tag that doesn't exist, the operation ignores it.

This operation has a limit of five transactions per second per account.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

## `TagKeys = [::String, ...]` -- *Required*
A list of tag keys. Each corresponding tag is removed from the delivery stream.

# Returns

`UntagDeliveryStreamOutput`

# Exceptions

`ResourceNotFoundException`, `ResourceInUseException`, `InvalidArgumentException` or `LimitExceededException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/UntagDeliveryStream)
"""
@inline untag_delivery_stream(aws::AWSConfig=default_aws_config(); args...) = untag_delivery_stream(aws, args)

@inline untag_delivery_stream(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "UntagDeliveryStream", args)

@inline untag_delivery_stream(args) = untag_delivery_stream(default_aws_config(), args)

"""
    using AWSSDK.Firehose.update_destination
    update_destination([::AWSConfig], arguments::Dict)
    update_destination([::AWSConfig]; DeliveryStreamName=, CurrentDeliveryStreamVersionId=, DestinationId=, <keyword arguments>)

    using AWSCore.Services.firehose
    firehose([::AWSConfig], "UpdateDestination", arguments::Dict)
    firehose([::AWSConfig], "UpdateDestination", DeliveryStreamName=, CurrentDeliveryStreamVersionId=, DestinationId=, <keyword arguments>)

# UpdateDestination Operation

Updates the specified destination of the specified delivery stream.

Use this operation to change the destination type (for example, to replace the Amazon S3 destination with Amazon Redshift) or change the parameters associated with a destination (for example, to change the bucket name of the Amazon S3 destination). The update might not occur immediately. The target delivery stream remains active while the configurations are updated, so data writes to the delivery stream can continue during this process. The updated configurations are usually effective within a few minutes.

Switching between Amazon ES and other services is not supported. For an Amazon ES destination, you can only update to another Amazon ES destination.

If the destination type is the same, Kinesis Data Firehose merges the configuration parameters specified with the destination configuration that already exists on the delivery stream. If any of the parameters are not specified in the call, the existing values are retained. For example, in the Amazon S3 destination, if [EncryptionConfiguration](@ref) is not specified, then the existing `EncryptionConfiguration` is maintained on the destination.

If the destination type is not the same, for example, changing the destination from Amazon S3 to Amazon Redshift, Kinesis Data Firehose does not merge any parameters. In this case, all parameters must be specified.

Kinesis Data Firehose uses **CurrentDeliveryStreamVersionId** to avoid race conditions and conflicting merges. This is a required field, and the service updates the configuration only if the existing configuration has a version ID that matches. After the update is applied successfully, the version ID is updated, and can be retrieved using [DescribeDeliveryStream](@ref). Use the new version ID to set **CurrentDeliveryStreamVersionId** in the next call.

# Arguments

## `DeliveryStreamName = ::String` -- *Required*
The name of the delivery stream.

## `CurrentDeliveryStreamVersionId = ::String` -- *Required*
Obtain this value from the **VersionId** result of [DeliveryStreamDescription](@ref). This value is required, and helps the service perform conditional operations. For example, if there is an interleaving update and this value is null, then the update destination fails. After the update is successful, the `VersionId` value is updated. The service then performs a merge of the old configuration with the new configuration.

## `DestinationId = ::String` -- *Required*
The ID of the destination.

## `S3DestinationUpdate = [ ... ]`
[Deprecated] Describes an update for a destination in Amazon S3.
```
 S3DestinationUpdate = [
        "RoleARN" =>  ::String,
        "BucketARN" =>  ::String,
        "Prefix" =>  ::String,
        "BufferingHints" =>  [
            "SizeInMBs" =>  ::Int,
            "IntervalInSeconds" =>  ::Int
        ],
        "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
        "EncryptionConfiguration" =>  [
            "NoEncryptionConfig" =>  "NoEncryption",
            "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `ExtendedS3DestinationUpdate = [ ... ]`
Describes an update for a destination in Amazon S3.
```
 ExtendedS3DestinationUpdate = [
        "RoleARN" =>  ::String,
        "BucketARN" =>  ::String,
        "Prefix" =>  ::String,
        "BufferingHints" =>  [
            "SizeInMBs" =>  ::Int,
            "IntervalInSeconds" =>  ::Int
        ],
        "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
        "EncryptionConfiguration" =>  [
            "NoEncryptionConfig" =>  "NoEncryption",
            "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "S3BackupMode" =>  "Disabled" or "Enabled",
        "S3BackupUpdate" =>  [
            "RoleARN" =>  ::String,
            "BucketARN" =>  ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "DataFormatConversionConfiguration" =>  [
            "SchemaConfiguration" =>  [
                "RoleARN" =>  ::String,
                "CatalogId" =>  ::String,
                "DatabaseName" =>  ::String,
                "TableName" =>  ::String,
                "Region" =>  ::String,
                "VersionId" =>  ::String
            ],
            "InputFormatConfiguration" =>  ["Deserializer" =>  [
                    "OpenXJsonSerDe" =>  [
                        "ConvertDotsInJsonKeysToUnderscores" =>  ::Bool,
                        "CaseInsensitive" =>  ::Bool,
                        "ColumnToJsonKeyMappings" =>  ::Dict{String,String}
                    ],
                    "HiveJsonSerDe" =>  ["TimestampFormats" =>  [::String, ...]]
                ]],
            "OutputFormatConfiguration" =>  ["Serializer" =>  [
                    "ParquetSerDe" =>  [
                        "BlockSizeBytes" =>  ::Int,
                        "PageSizeBytes" =>  ::Int,
                        "Compression" =>  "UNCOMPRESSED", "GZIP" or "SNAPPY",
                        "EnableDictionaryCompression" =>  ::Bool,
                        "MaxPaddingBytes" =>  ::Int,
                        "WriterVersion" =>  "V1" or "V2"
                    ],
                    "OrcSerDe" =>  [
                        "StripeSizeBytes" =>  ::Int,
                        "BlockSizeBytes" =>  ::Int,
                        "RowIndexStride" =>  ::Int,
                        "EnablePadding" =>  ::Bool,
                        "PaddingTolerance" =>  double,
                        "Compression" =>  "NONE", "ZLIB" or "SNAPPY",
                        "BloomFilterColumns" =>  [::String, ...],
                        "BloomFilterFalsePositiveProbability" =>  double,
                        "DictionaryKeyThreshold" =>  double,
                        "FormatVersion" =>  "V0_11" or "V0_12"
                    ]
                ]],
            "Enabled" =>  ::Bool
        ]
    ]
```

## `RedshiftDestinationUpdate = [ ... ]`
Describes an update for a destination in Amazon Redshift.
```
 RedshiftDestinationUpdate = [
        "RoleARN" =>  ::String,
        "ClusterJDBCURL" =>  ::String,
        "CopyCommand" =>  [
            "DataTableName" => <required> ::String,
            "DataTableColumns" =>  ::String,
            "CopyOptions" =>  ::String
        ],
        "Username" =>  ::String,
        "Password" =>  ::String,
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3Update" =>  [
            "RoleARN" =>  ::String,
            "BucketARN" =>  ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "S3BackupMode" =>  "Disabled" or "Enabled",
        "S3BackupUpdate" =>  [
            "RoleARN" =>  ::String,
            "BucketARN" =>  ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `ElasticsearchDestinationUpdate = [ ... ]`
Describes an update for a destination in Amazon ES.
```
 ElasticsearchDestinationUpdate = [
        "RoleARN" =>  ::String,
        "DomainARN" =>  ::String,
        "IndexName" =>  ::String,
        "TypeName" =>  ::String,
        "IndexRotationPeriod" =>  "NoRotation", "OneHour", "OneDay", "OneWeek" or "OneMonth",
        "BufferingHints" =>  [
            "IntervalInSeconds" =>  ::Int,
            "SizeInMBs" =>  ::Int
        ],
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3Update" =>  [
            "RoleARN" =>  ::String,
            "BucketARN" =>  ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

## `SplunkDestinationUpdate = [ ... ]`
Describes an update for a destination in Splunk.
```
 SplunkDestinationUpdate = [
        "HECEndpoint" =>  ::String,
        "HECEndpointType" =>  "Raw" or "Event",
        "HECToken" =>  ::String,
        "HECAcknowledgmentTimeoutInSeconds" =>  ::Int,
        "RetryOptions" =>  ["DurationInSeconds" =>  ::Int],
        "S3BackupMode" =>  "FailedEventsOnly" or "AllEvents",
        "S3Update" =>  [
            "RoleARN" =>  ::String,
            "BucketARN" =>  ::String,
            "Prefix" =>  ::String,
            "BufferingHints" =>  [
                "SizeInMBs" =>  ::Int,
                "IntervalInSeconds" =>  ::Int
            ],
            "CompressionFormat" =>  "UNCOMPRESSED", "GZIP", "ZIP" or "Snappy",
            "EncryptionConfiguration" =>  [
                "NoEncryptionConfig" =>  "NoEncryption",
                "KMSEncryptionConfig" =>  ["AWSKMSKeyARN" => <required> ::String]
            ],
            "CloudWatchLoggingOptions" =>  [
                "Enabled" =>  ::Bool,
                "LogGroupName" =>  ::String,
                "LogStreamName" =>  ::String
            ]
        ],
        "ProcessingConfiguration" =>  [
            "Enabled" =>  ::Bool,
            "Processors" =>  [[
                "Type" => <required> "Lambda",
                "Parameters" =>  [[
                    "ParameterName" => <required> "LambdaArn", "NumberOfRetries", "RoleArn", "BufferSizeInMBs" or "BufferIntervalInSeconds",
                    "ParameterValue" => <required> ::String
                ], ...]
            ], ...]
        ],
        "CloudWatchLoggingOptions" =>  [
            "Enabled" =>  ::Bool,
            "LogGroupName" =>  ::String,
            "LogStreamName" =>  ::String
        ]
    ]
```

# Returns

`UpdateDestinationOutput`

# Exceptions

`InvalidArgumentException`, `ResourceInUseException`, `ResourceNotFoundException` or `ConcurrentModificationException`.

See also: [AWS API Documentation](https://docs.aws.amazon.com/goto/WebAPI/firehose-2015-08-04/UpdateDestination)
"""
@inline update_destination(aws::AWSConfig=default_aws_config(); args...) = update_destination(aws, args)

@inline update_destination(aws::AWSConfig, args) = AWSCore.Services.firehose(aws, "UpdateDestination", args)

@inline update_destination(args) = update_destination(default_aws_config(), args)

end # module Firehose

#==============================================================================#
# End of file
#==============================================================================#


text:
@testset "expr conversions" begin
    @test K(:(a + b)) == q("(+;`a;`b)")
    @test K(:(a + b + c)) == q("(+;`a;(+;`b;`c))")
    @test K(:(a + b + c + d)) == q("(+;`a;(+;`b;(+;`c;`d)))")
    @test K(:(a + b * c)) == q("(+;`a;(*;`b;`c))")
    @test K(:(-a)) == K(:(neg(a))) == q("(neg;`a)")
    @test K(:(enlist(x))) == q("(enlist;`x)")
    @test K(:(enlist(a,b,c,d,e))) == q("(enlist;`a;`b;`c;`d;`e)")
end

@testset "expr eval" begin
    @test q("eval", :(til(5))) == Array(0:4)
    @test (q("eval", :(a = 42)); q("a") == 42)
    @test (r = q("eval", :(a=2;b=3)); r == q("b") == 3 && q("a") == 2)
end

ext: jl


text:
using GeoPhyInv
using Statistics

medium=Medium(:acou_homo1); # load a simple homogeneous acoustic medium from the gallery
update!(medium, [:vp,:rho], randn_perc=5); # add some random noise to the medium

ageom=AGeom(medium.mgrid,:xwell, SSrcs(2)); # load a simple acquisition using `mgrid` of the medium

tgrid = range(0.0,stop=2.0,length=2000); # generate a time grid
wav = ricker(10.0, tgrid, tpeak=0.25,); # ricker wavelet
srcwav = SrcWav(tgrid, ageom, [:P]);
update!(srcwav, [:P], wav);

pa=SeisForwExpt(Fdtd(),medium=medium, ageom=ageom, srcwav=srcwav,
	snaps_flag=true, tsnaps=[0.3, 0.4, 0.5], tgrid=tgrid, verbose=true);

@time update!(pa);

snaps=pa[:snaps,1]; # extracting snaps of the first supersource
snaps=pa[:snaps,2]; # second supersource


text:
import JUDI: lsrtm_objective, time_modeling, fwi_objective
import JUDI: judiJacobian, adjbornop, bornop, judiAbstractJacobian, subsample

# Model smoother
function smooth(m::Model; sigma=3)
    new_model = deepcopy(m)
    new_model.m.data[:] = imfilter(m.m.data, Kernel.gaussian(sigma))
    return new_model
end

SincInterpolation(Y, S, Up) = sinc.( (Up .- S') ./ (S[2] - S[1]) ) * Y

function time_resample_data(d::judiVector, dt_new)
    return time_resample_data(d.data[1], dt_new, d.geometry.dt[1], d.geometry.t[1])
end

time_resample_data(d::SeisCon, dt_new, dt_old, t) = time_resample_data(Float32.(d[1].data), dt_new, dt_old, t)

function time_resample_data(d::Array, dt_new, dt_old, t)
    if dt_new == dt_old
        return d
    else
        timeAxis = 0:dt_old:t
        timeInterp = 0:dt_new:t
        dataInterp = Float32.(SincInterpolation(d, timeAxis, timeInterp))

        return dataInterp
    end
end

Base.getindex(g::GeometryIC, i::Int64) = subsample(g, i)

function get_pad(model::Model)
    pm = load_pymodel()
    modelPy = pm."Model"(origin=model.o, spacing=model.d, shape=model.n,
                         m=.44, nbl=model.nb)
    return modelPy.padsizes
end

function get_model(src_geom::Geometry, rec_geom::Geometry, model::Model, options::Options)
    ~options.limit_m && return model
    return limit_model_to_receiver_area(src_geom, rec_geom, deepcopy(model), options.buffer_size)[1]
end

function get_model(src_geom::Geometry, rec_geom::Geometry, model::Model, options::Options, dm)
    ~options.limit_m && return (model, dm)
    return limit_model_to_receiver_area(src_geom, rec_geom, deepcopy(model), options.buffer_size; pert=dm)
end

# Time modeling
function time_modeling(model::Model, q::judiVector, residual::judiVector, dobs::judiVector, options::Options, ps::Integer)
    q.geometry = Geometry(q.geometry)
    residual.geometry = Geometry(residual.geometry)
    dobs.geometry = Geometry(dobs.geometry)

    model_loc = get_model(q.geometry, residual.geometry, model, options)
    modelPy = devito_model(model_loc, options)
    d0, Q, eu = forward(model_loc, q, dobs; ps=ps, options=options, modelPy=modelPy)
    ge = backprop(model_loc, residual, Q, eu; options=options, modelPy=modelPy)
    options.limit_m && (ge = extend_gradient(model, model_loc, ge))
    return PhysicalParameter(ge, model.d, model.o)
end

# FWI and lsrtm function

function fwi_objective(model::Model, q::judiVector, dobs::judiVector, options::Options, ps::Integer)
    dobs = get_data(dobs)
    normalize!(dobs)
    q.geometry = Geometry(q.geometry)
    dobs.geometry = Geometry(dobs.geometry)

    model_loc = get_model(q.geometry, dobs.geometry, model, options)
    modelPy = devito_model(model_loc, options)
    d0, Q, eu = forward(model_loc, q, dobs; ps=ps, options=options, modelPy=modelPy)
    normalize!(d0)
    residual = d0 - dobs
    ge = backprop(model_loc, residual, Q, eu; options=options, modelPy=modelPy)
    options.limit_m && (ge = extend_gradient(model, model_loc, ge))

    obj = loss(d0, residual)
    return obj, PhysicalParameter(ge, model.d, model.o)
end

# LSRTM

function lsrtm_objective(model::Model, q::judiVector, dobs::judiVector, dm::Union{Array, PhysicalParameter}, options::Options, nlind::Bool, ps::Integer)
    q.geometry = Geometry(q.geometry)
    dobs.geometry = Geometry(dobs.geometry)

    model_loc, dm = get_model(q.geometry, dobs.geometry, model, options, dm)
    modelPy = devito_model(model_loc, options; dm=dm)
    dnl, dl, Q, eu = born(model_loc, q, dobs, dm; ps=ps, options=options, modelPy=modelPy)
    residual = nlind ? dl - (dobs - dnl) : dl - dobs
    ge = backprop(model_loc, residual, Q, eu; options=options, modelPy=modelPy)
    options.limit_m && (ge = extend_gradient(model, model_loc, ge))
    return .5f0*norm(residual)^2, PhysicalParameter(ge, model.d, model.o)
end

#########################

mutable struct judiJacobianP{DDT, RDT} <: judiAbstractJacobian{DDT, RDT}
    name::String
    m::Integer
    n::Integer
    info::Info
    model::Model
    source::judiVector
    dobs::judiVector
    ps::Integer
    options::Options
    fop::Function              # forward
    fop_T::Union{Function, Nothing}  # transpose
end

Base.getproperty(J::judiJacobianP, sym::Symbol) = sym == :recGeometry ? J.dobs.geometry : getfield(J, sym)

set_r!(J::judiJacobianP, ps::Integer) = (J.ps = ps)
set_ps!(J::judiJacobianP, ps::Integer) = (J.ps = ps)

# Jacobian with probing
function JUDI.judiJacobian(F::judiPDEfull, source::judiVector, ps::Int64, dobs::judiVector; DDT::DataType=Float32, RDT::DataType=DDT, options=nothing)
    # JOLI wrapper for nonlinear forward modeling
    compareGeometry(F.srcGeometry, source.geometry) == true || judiJacobianException("Source geometry mismatch")
    (DDT == Float32 && RDT == Float32) || throw(judiJacobianException("Domain and range types not supported"))
    m = typeof(F.recGeometry) <: GeometryOOC ? sum(F.recGeometry.nsamples) : sum([length(F.recGeometry.xloc[j])*F.recGeometry.nt[j] for j=1:source.nsrc])
    n = F.info.n

    isnothing(options) && (options = F.options)
    return J = judiJacobianP{Float32,Float32}("linearized wave equation", m, n,
        F.info, F.model, source, dobs, ps, options, bornop, adjbornop)
end

judiJacobian(J::judiJacobianP{DDT,RDT}; name=J.name, m=J.m, n=J.n, info=J.info, model=J.model, source=J.source,
     dobs=J.dobs, ps=J.ps, opt=J.options, fop=J.fop, fop_T=J.fop_T) where {DDT, RDT} =
            judiJacobianP{DDT,RDT}(name, m, n, info, model, source, dobs, ps, opt, fop, fop_T)

# Subsample Jacobian
function subsample(J::judiJacobianP{ADDT,ARDT}, srcnum) where {ADDT,ARDT}
    nsrc = typeof(srcnum) <: Int ? 1 : length(srcnum)
    info = Info(J.info.n, nsrc, J.info.nt[srcnum])
    dobs_loc = J.dobs[srcnum]
    m = dobs_loc.m

    return judiJacobian(J; m=m, info=info, source=J.source[srcnum], dobs=dobs_loc, ps=J.ps, opt=subsample(J.options, srcnum))
end

function adjbornop(J::judiJacobianP, w)
    srcnum = 1:J.info.nsrc
    return time_modeling(J.model, J.source, w, srcnum, J.dobs, J.options, J.ps)
end

function bornop(J::judiJacobianP, v)
    srcnum = 1:J.info.nsrc
    return time_modeling(J.model, J.source.geometry, J.source.data, J.dobs.geometry, nothing, v, srcnum, 'J', 1, J.options)
end

###### JUDI fix to be moved there
Base.similar(x::Array{T, N}, m::Model) where {T<:Real, N} = zeros(T, m.n)

max_stars_repo_name: slimgroup/TimeProbeSeismic

max_issues_repo_name: slimgroup/TimeProbeSeismic

max_forks_repo_name: slimgroup/TimeProbeSeismic


text:
using BilliardModels

function run_sinai(num_collisions)
    billiard_table = Sinai_billiard(0.354, true, true)  # periodic in x and y

    # x = Vector2D(0.3, 0.)
    # v = Vector2D(0.99, 0.01)

    x, v = initial_condition(billiard_table, -.5, .5, -.5, .5)

    l = Vector2D(0, 0)
    p = ParticleOnLattice(x, v, l)

    xs, ls, free_paths = billiard_dynamics_on_lattice(p, billiard_table, num_collisions);
end

N = 0
try
    N = int(ARGS[1])
catch
    N = 1_000_000
end

println("# Using N=$N")

run_sinai(1)
@time run_sinai(N)

using BilliardModels
billiard_table = Sinai_billiard(0.354, true, true)  # periodic in x and y
x, v = initial_condition(billiard_table, -.5, .5, -.5, .5)
l = Vector2D(0, 0)
p = ParticleOnLattice(x, v, l)
num_collisions = 10
xs, ls, free_paths = billiard_dynamics_on_lattice(p, billiard_table, num_collisions);
continuous_time(xs, ls, free_paths, 0.25)

billiard_dynamics_on_lattice

max_stars_repo_name: dpsanders/BilliardModel

max_issues_repo_name: dpsanders/BilliardModel

max_forks_repo_name: dpsanders/BilliardModel


text:
using DSP
include("extend_to_dc.jl")
include("make_conjugate_symmetric.jl")
include("gen_impulse_response.jl")
include("clean_array.jl")
include("gen_step_response.jl")

function s_parameter_to_time_domain(freqs,s11; kb_beta=6)
   # function [timePts, step_response] = SParams2TDR(Freq,S11,varargin)
   # SParams2TDR takes in measured frequency data and outputs a step response

   # prepare s parameter for transformation
   res = extend_to_dc(freqs,s11);
   FreqToDC = res[1]; S11toDC = res[2]
   doubleS11 = make_conjugate_symmetric(S11toDC); # runs from DC -> - deltaF
   doubleFreq = vcat(FreqToDC, -flipdim(FreqToDC[2:end],1));

   filter_vals = kaiser(length(doubleS11),kb_beta);

   doubleS11 = flipdim(fftshift(filter_vals),1) .* doubleS11;
   res = gen_impulse_response(doubleFreq, doubleS11);
   timePts = res[1]; imp_response = res[2];
   # Generate impulse response
   imp_response = clean_array(imp_response,threshold=1e-6);
   # Check if it's real
   if ~isreal(imp_response)
      warn("Impulse response is not real. Results may not make sense.");
   end
   # Generate step response
   res = gen_step_response(timePts, imp_response);
   step_response = res[2]

   # Convert the step response to a vector of floats (if possible)
   step_response = convert(Vector{Float64}, step_response)
   println("Step response type: ", typeof(step_response))
   return (timePts, step_response)
end


text:
SUITE["points"] = BenchmarkGroup(["microbenchmark", "points"]) # add tags for eventual filtering

add_benchmark!(SUITE["points"], [
  :(Point(0., 1.)),
  :(Point([0., 1.])),
  :(Point([0., 1])),
  :(Point(0., convert(Float64, 1))),
])


text:
abstract type AbstractAgent
end
select_move(agent::AbstractAgent, game_state::GameState) = error("unimplemented")

function is_point_an_eye(board::Board, point::Point, color::Player)::Bool
    if !isvoid(board[point])
        return false
    end
    for nbr in nbrs(point)
        if is_on_grid(board, nbr)
            if board[nbr].color !== color
                return false
            end
        end
    end

    friendly_corners = 0
    off_board_corners = 0

    corners = [
        (row = point.row - 1, col = point.col - 1), (row = point.row - 1, col = point.col + 1),
        (row = point.row + 1, col = point.col - 1), (row = point.row + 1, col = point.col + 1)
    ]

    for corner in corners
        if is_on_grid(board, corner)
            if board[corner].color ≡ color
                friendly_corners += 1
            end
        else
            off_board_corners += 1
        end
    end

    ans = (off_board_corners > 0) ? (off_board_corners + friendly_corners == 4) : (friendly_corners >= 3)

    return ans
end


text:
import PowerSimulations:
    open_store,
    HdfSimulationStore,
    HDF_FILENAME,
    SimulationStoreParams,
    ModelStoreParams,
    SimulationModelStoreRequirements,
    CacheFlushRules,
    KiB,
    MiB,
    GiB,
    STORE_CONTAINER_VARIABLES,
    initialize_problem_storage!,
    add_rule!,
    write_result!,
    read_result,
    has_dirty,
    get_cache_hit_percentage

function _initialize!(store, sim, variables, model_defs, cache_rules)
    models = OrderedDict{Symbol, ModelStoreParams}()
    model_reqs = Dict{Symbol, SimulationModelStoreRequirements}()
    num_param_containers = 0
    for model in keys(model_defs)
        execution_count = model_defs[model]["execution_count"]
        horizon = model_defs[model]["horizon"]
        num_rows = execution_count * sim["num_steps"]

        model_params = ModelStoreParams(
            execution_count,
            horizon,
            model_defs[model]["interval"],
            model_defs[model]["resolution"],
            model_defs[model]["base_power"],
            model_defs[model]["system_uuid"],
        )
        reqs = SimulationModelStoreRequirements()

        for (key, array) in model_defs[model]["variables"]
            reqs.variables[key] = Dict(
                "columns" => model_defs[model]["names"],
                "dims" => (horizon, length(model_defs[model]["names"]), num_rows),
            )
            keep_in_cache = variables[key]["keep_in_cache"]
            add_rule!(cache_rules, model, key, keep_in_cache)
        end

        models[model] = model_params
        model_reqs[model] = reqs
        num_param_containers += length(reqs.variables)
    end

    params = SimulationStoreParams(
        sim["initial_time"],
        sim["step_resolution"],
        sim["num_steps"],
        models,
        # Emulation Model Store requirements. No tests yet
        OrderedDict(
            :Emulator => ModelStoreParams(
                100, # Num Executions
                1,
                Minute(5), # Interval
                Minute(5), # Resolution
                100.0,
                Base.UUID("4076af6c-e467-56ae-b986-b466b2749572"),
            ),
        ),
    )
    em_reqs = SimulationModelStoreRequirements()
    initialize_problem_storage!(store, params, model_reqs, em_reqs, cache_rules)
    return
end

function _run_sim_test(path, sim, variables, model_defs, cache_rules, seed)
    rng = MersenneTwister(seed)
    type = STORE_CONTAINER_VARIABLES
    open_store(HdfSimulationStore, path, "w") do store
        sim_time = sim["initial_time"]
        _initialize!(store, sim, variables, model_defs, cache_rules)
        for step in 1:sim["num_steps"]
            for model in keys(model_defs)
                model_time = sim_time
                for i in 1:model_defs[model]["execution_count"]
                    for key in keys(variables)
                        data = rand(rng, size(model_defs[model]["variables"][key])...)
                        write_result!(store, model, key, model_time, model_time, data)
                        columns = model_defs[model]["names"]
                        _verify_data(data, store, model, key, model_time, columns)
                    end

                    model_time += model_defs[model]["resolution"]
                end
            end

            sim_time += sim["step_resolution"]
        end

        for output_cache in values(store.cache.data)
            if PSI.should_keep_in_cache(output_cache)
                @test get_cache_hit_percentage(output_cache) == 100.0
            else
                @test get_cache_hit_percentage(output_cache) < 100.0
            end
        end

        flush(store)
        @test !has_dirty(store.cache)
    end
end

function _verify_read_results(path, sim, variables, model_defs, seed)
    rng = MersenneTwister(seed)
    type = STORE_CONTAINER_VARIABLES
    open_store(HdfSimulationStore, path, "r") do store
        sim_time = sim["initial_time"]
        for step in 1:sim["num_steps"]
            for model in keys(model_defs)
                model_time = sim_time
                for i in 1:model_defs[model]["execution_count"]
                    for key in keys(variables)
                        data = rand(rng, size(model_defs[model]["variables"][key])...)
                        columns = model_defs[model]["names"]
                        _verify_data(data, store, model, key, model_time, columns)
                    end

                    model_time += model_defs[model]["resolution"]
                end
            end

            sim_time += sim["step_resolution"]
        end

        for output_cache in values(store.cache.data)
            @test get_cache_hit_percentage(output_cache) == 0.0
        end
    end
end

function _verify_data(expected, store, model, name, time, columns)
    expected_df = DataFrames.DataFrame(expected, columns)
    df = read_result(DataFrames.DataFrame, store, model, name, time)
    @test expected_df == df
end

@testset "Test SimulationStore 2-d arrays" begin
    sim = Dict(
        "initial_time" => Dates.DateTime("2020-01-01T00:00:00"),
        "step_resolution" => Dates.Hour(24),
        "num_steps" => 50,
    )
    variables = Dict(
        PSI.VariableKey(ActivePowerVariable, ThermalStandard) =>
            Dict("keep_in_cache" => true),
        PSI.VariableKey(ActivePowerVariable, RenewableDispatch) =>
            Dict("keep_in_cache" => true),
        PSI.VariableKey(ActivePowerVariable, InterruptibleLoad) =>
            Dict("keep_in_cache" => false),
        PSI.VariableKey(ActivePowerVariable, RenewableFix) =>
            Dict("keep_in_cache" => false),
    )
    model_defs = OrderedDict(
        :ED => Dict(
            "execution_count" => 24,
            "horizon" => 12,
            "names" => [:dev1, :dev2, :dev3, :dev4, :dev5],
            "variables" => Dict(x => ones(12, 5) for x in keys(variables)),
            "interval" => Dates.Hour(1),
            "resolution" => Dates.Hour(1),
            "base_power" => 100.0,
            "system_uuid" => Base.UUID("4076af6c-e467-56ae-b986-b466b2749572"),
        ),
        :UC => Dict(
            "execution_count" => 1,
            "horizon" => 24,
            "names" => [:dev1, :dev2, :dev3],
            "variables" => Dict(x => ones(24, 3) for x in keys(variables)),
            "interval" => Dates.Hour(1),
            "resolution" => Dates.Hour(24),
            "base_power" => 100.0,
            "system_uuid" => Base.UUID("4076af6c-e467-56ae-b986-b466b2749572"),
        ),
    )
    cache_rules = CacheFlushRules(max_size=1 * MiB, min_flush_size=4 * KiB)

    path = mktempdir()
    # Use this seed to produce the same randomly generated arrays for write and verify.
    seed = 1234
    _run_sim_test(path, sim, variables, model_defs, cache_rules, seed)
    # TODO: Re-enable later when we serialize the keys to be deserialized later
    # _verify_read_results(path, sim, variables, model_defs, seed)
end

@testset "Test OptimizationOutputCache" begin
    key = PSI.OptimizationResultCacheKey(
        :ED,
        PSI.VariableKey(ActivePowerVariable, InterruptibleLoad),
    )
    cache = PSI.OptimizationOutputCache(key, PSI.CacheFlushRule(true))
    @test !PSI.has_clean(cache)
    @test !PSI.is_dirty(cache, Dates.now())

    timestamp1 = Dates.DateTime("2020-01-01T00:00:00")
    timestamp2 = Dates.DateTime("2020-01-01T01:00:00")
    timestamp3 = Dates.DateTime("2020-01-01T02:00:00")
    timestamp4 = Dates.DateTime("2020-01-01T03:00:00")
    PSI.add_result!(cache, timestamp1, ones(2), false)
    @test PSI.is_dirty(cache, timestamp1)
    PSI.add_result!(cache, timestamp2, ones(2), false)
    @test PSI.is_dirty(cache, timestamp2)

    @test_throws IS.InvalidValue PSI.add_result!(cache, timestamp2, ones(2), false)

    @test length(cache.data) == 2
    @test length(cache.dirty_timestamps) == 2

    popfirst!(cache.dirty_timestamps)
    @test !PSI.is_dirty(cache, timestamp1)
    @test PSI.has_clean(cache)
    @test length(cache.data) == 2
    @test length(cache.dirty_timestamps) == 1

    PSI.add_result!(cache, timestamp3, ones(2), false)
    @test length(cache.data) == 3
    @test length(cache.dirty_timestamps) == 2

    PSI.add_result!(cache, timestamp4, ones(2), true)
    @test length(cache.data) == 3
    @test length(cache.dirty_timestamps) == 3

    popfirst!(cache.dirty_timestamps)
    @test PSI.has_clean(cache)

    empty!(cache.dirty_timestamps)
    empty!(cache)
    @test isempty(cache.data)
    @test isempty(cache.dirty_timestamps)

    PSI.add_result!(cache, timestamp1, ones(2), false)
    PSI.add_result!(cache, timestamp2, ones(2), false)
    PSI.discard_results!(cache, [timestamp1, timestamp2])
    @test isempty(cache.data)
end

# TODO: test optimizer stats
# TODO: unit tests of individual functions, size checks
# TODO: 3-d arrays
# TODO: profiling of memory performance and GC


text:
"""
    train!(model::AbstractGPModel; iterations::Integer=100, callback, convergence)

Function to train the given GP `model`.
## Arguments
- `model` : AbstractGPModel model with either an `Analytic`, `AnalyticVI` or `NumericalVI` type of inference

## Keyword Arguments
- `iterations::Int` : Number of iterations (not necessarily epochs!)for training
- `callback::Function=nothing` : Callback function called at every iteration. Should be of type `function(model,iter) ...  end`
- `convergence::Function=nothing` : Convergence function to be called every iteration, should return a scalar and take the same arguments as `callback`
"""
function train!(
    model::AbstractGPModel{T},
    X::AbstractArray,
    y,
    iterations::Int=100;
    callback=nothing,
    convergence=nothing,
    state=nothing,
    obsdim=1,
) where {T}
    iterations > 0 || error("Number of iterations should be positive")
    X, Tx = wrap_X(X, obsdim)
    data = wrap_data(X, y, likelihood(model))
    if is_stochastic(model)
        0 < batchsize(inference(model)) <= n_sample(data) || error(
            "The size of mini-batch $(batchsize(inference(model))) is incorrect (negative or bigger than number of samples), please set `batchsize` correctly in the inference object",
        )
        set_ρ!(model, n_sample(data) / batchsize(inference(model)))
    else
        set_batchsize!(inference(model), n_sample(data))
    end

    if verbose(model) > 0
        @info "Starting training $model with $(n_sample(data)) samples, $(n_dim(data)) features and $(n_latent(likelihood(model))) latent GP" *
            (n_latent(model) > 1 ? "s" : "")
    end
    # model.evol_conv = [] # Array to check on the evolution of convergence
    local_iter = 1
    if isnothing(state)
        setHPupdated!(inference(model), true)
    end
    state = isnothing(state) ? init_state(model) : state
    conv = Inf
    p = Progress(iterations; dt=0.2, desc="Training Progress: ")
    prev_elbo = -Inf
    while true # loop until one condition is matched
        try
            if is_stochastic(model)
                minibatch = StatsBase.sample(
                    1:n_sample(data), batchsize(model); replace=false
                )
                x = view_x(data, minibatch)
                y = view_y(likelihood(model), data, minibatch)
            else
                x = input(data)
                y = view_y(likelihood(model), data, 1:n_sample(data))
            end
            state = update_parameters!(model, state, x, y) #Update all the variational parameters
            set_trained!(model, true)
            if !isnothing(callback)
                callback(model, state, n_iter(model)) #Use a callback method if set by user
            end
            if (n_iter(model) % model.atfrequency == 0) &&
                (n_iter(model) >= 3) &&
                (local_iter != iterations)
                state = update_hyperparameters!(model, state, x, y) #Update the hyperparameters
            end
            # Print out informations about the convergence
            if verbose(model) > 2 || (verbose(model) > 1 && local_iter % 10 == 0)
                if inference(model) isa GibbsSampling
                    next!(p; showvalues=[(:samples, local_iter)])
                else
                    if (verbose(model) == 2 && local_iter % 10 == 0)
                        elbo = objective(model, state, y)
                        prev_elbo = elbo
                        ProgressMeter.update!(p, local_iter - 1)
                        ProgressMeter.next!(
                            p; showvalues=[(:iter, local_iter), (:ELBO, elbo)]
                        )
                    elseif verbose(model) > 2
                        elbo = objective(model, state, y)
                        prev_elbo = elbo
                        ProgressMeter.next!(
                            p; showvalues=[(:iter, local_iter), (:ELBO, prev_elbo)]
                        )
                    end
                end
            end
            local_iter += 1
            model.inference.n_iter += 1
            (local_iter <= iterations) || break #Verify if the number of maximum iterations has been reached
        # (iter < model.nEpochs && conv > model.ϵ) || break; #Verify if any condition has been broken
        catch e
            if isa(e, InterruptException)
                @warn "Training interrupted by user at iteration $local_iter"
                break
            else
                rethrow(e)
            end
        end
    end
    if verbose(model) > 0
        @info "Training ended after $(local_iter - 1) iterations. Total number of iterations $(n_iter(model))"
    end
    state = merge(state, compute_Ks(model)) # Compute final version of the matrices for predictions
    post_step!(model, state)
    set_trained!(model, true)
    return model, state
end

@traitfn function train!(
    model::TGP, iterations; callback=nothing, convergence=nothing, state=nothing
) where {TGP <: AbstractGPModel; IsFull{TGP}}
    return train!(
        model,
        input(model.data),
        output(model.data),
        iterations;
        state,
        callback,
        convergence,
    )
end

function update_parameters!(model::GP, state, x, y)
    state = compute_kernel_matrices(model, state, x) # Recompute the matrices if necessary (when hyperparameters have been updated)
    state = analytic_updates(model, state, y)
    return state
end

function update_parameters!(model::VGP, state, x, y)
    state = compute_kernel_matrices(model, state, x) # Recompute the matrices if necessary (always for the stochastic case, or when hyperparameters have been updated)
    state = variational_updates(model, state, y)
    return state
end

## Update all variational parameters of the sparse variational GP Model ##
function update_parameters!(model::SVGP, state, x, y)
    state = compute_kernel_matrices(model, state, x) # Recompute the matrices if necessary (always for the stochastic case, or when hyperparameters have been updated)
    state = variational_updates(model, state, y)
    return state
end

function update_parameters!(model::MOVGP, state, x, y)
    state = compute_kernel_matrices(model, state, x) # Recompute the matrices if necessary (always for the stochastic case, or when hyperparameters have been updated)
    state = update_A!(model, state, y)
    state = variational_updates(model, state, y)
    return state
end

function update_parameters!(m::MOSVGP, state, x, y)
    state = compute_kernel_matrices(m, state, x) # Recompute the matrices if necessary (always for the stochastic case, or when hyperparameters have been updated)
    update_A!(m, state, y)
    state = variational_updates(m, state, y)
    return state
end

function update_parameters!(m::VStP, state, x, y)
    state = compute_kernel_matrices(m, state, x) # Recompute the matrices if necessary (always for the stochastic case, or when hyperparameters have been updated)
    state = local_prior_updates!(m, state, x)
    state = variational_updates(m, state, y)
    return state
end

function compute_kernel_matrices(m::GP{T}, state, x, ::Bool) where {T}
    kernel_matrices = (; K=compute_K(m.f, x, T(jitt)))
    setHPupdated!(inference(m), false)
    return merge(state, (; kernel_matrices))
end

@traitfn function compute_kernel_matrices(
    m::TGP, state, x, update::Bool=false
) where {T,TGP<:AbstractGPModel{T};IsFull{TGP}}
    if isHPupdated(inference(m)) || update
        kernel_matrices = map(m.f) do gp
            (; K=compute_K(gp, x, T(jitt)))
        end
        setHPupdated!(inference(m), false)
        return merge(state, (; kernel_matrices))
    else
        return state
    end
end

@traitfn function compute_kernel_matrices(
    m::TGP, state, x, update::Bool=false
) where {T,TGP<:AbstractGPModel{T};!IsFull{TGP}}
    kernel_matrices = if isHPupdated(inference(m)) || update
        map(m.f) do gp
            (; K=compute_K(gp, T(jitt)))
        end
    else
        state.kernel_matrices
    end
    # If change of hyperparameters or if stochatic
    kernel_matrices = if isHPupdated(inference(m)) || is_stochastic(inference(m)) || update
        κs = map(m.f, kernel_matrices) do gp, kernel_matrix
            compute_κ(gp, x, kernel_matrix.K, T(jitt))
        end
        merge.(kernel_matrices, κs)
    else
        kernel_matrices
    end
    setHPupdated!(inference(m), false)
    return merge(state, (; kernel_matrices)) # update the kernel_matrices state
end

function compute_Ks(m::AbstractGPModel{T}) where {T}
    return (; kernel_matrices=broadcast(m.f, Zviews(m)) do gp, x
        K = compute_K(gp, x, T(jitt))
        return (; K)
    end)
end

function compute_Ks(m::GP{T}) where {T}
    return (; kernel_matrices=(; K=compute_K(m.f, input(m.data), T(jitt))))
end


text:
# Functions for creating "word list" for Nerdle.

import Base: isless, show

"""
Represents a mathematical expression.
"""
struct Expr
    # Numerical value of the expression
    value::Int32

    # String representation of expression
    rep::String
end

show(o::IO, x::Expr) = print(o, x.rep)

"""
Order expressions by length of their strings. Expressions of equal length are ordered by numerical value.
"""
function isless(x::Expr, y::Expr)
    if length(x.rep) ≠ length(y.rep)
        return length(x.rep) < length(y.rep)
    end
    return x.value < y.value
end

"""
    op(f, op, left, right)

Create plausible expressions from operands chosen from left and right.
f implements the operation and op is the character reprsentation of it.
f should return nothing if the operation is invalid.
"""
function op(f::Function, op::Char, left::Vector{Expr}, right::Vector{Expr})
    @assert issorted(left)
    @assert issorted(right)

    result = Expr[]
    for x in left
        for y in right
            if length(x.rep) + 1 + length(y.rep) > 6
                break
            end
            z = f(x.value, y.value)
            if z ≠ nothing && 0 ≤ z
                push!(result, Expr(z, x.rep * op * y.rep))
            end
        end
    end
    return result
end

"""
    divide(x,y)

Return x/y if the quotient is defined and exact, nothing otherwise.
"""
function divide(x::Integer, y::Integer)
    if y == 0 || x % y ≠ 0
        return nothing
    end
    return div(x,y)
end

"""
Return equations allowed by Nerdle.
"""
function equations()
    # Literals
    lits = [Expr(i, string(i)) for i in 1:999]

    # Terms with two literals
    mul2 = op(*, '*', lits, lits)
    div2 = op(divide, '/', lits, lits)
    term2 = sort([mul2;div2])

    # Terms with three literals
    mul3 = op(*, '*', term2, lits)
    div3 = op(divide, '/', term2, lits)
    term3 = [mul3;div3]
    terms = sort([lits; term2; term3])

    # Expressions with two terms
    add2 = op(+, '+', terms, terms)
    sub2 = op(-, '-', terms, terms)
    expr2 = sort([add2;sub2])

    # Expressions with three terms
    add3 = op(+, '+', expr2, terms)
    sub3 = op(-, '-', expr2, terms)
    expr3 = sort([add3;sub3])

    # Expressions without final filtering for equation length
    exprs = sort([lits; terms; expr2; expr3]; by=x->x.value)

    # Get equations that are 8 characters
    result = String[]
    for e in exprs
        rep = e.rep * '=' * string(e.value)
        if length(rep) == 8
            push!(result, rep)
        end
    end
    return result
end

"""
Write word lists for Nerdle.

Overwrites files `answers.txt` and `guesses-other.text`
"""
function makeNerdleWordLists()
    eq = equations();
    open("answers.txt","w") do f
        for e in eq
            println(f, e)
        end
    end
    open("guesses-other.txt","w") do f
    end
end

max_stars_repo_name: ArchRobison/WordleAnalysis

max_issues_repo_name: ArchRobison/WordleAnalysis

max_forks_repo_name: ArchRobison/WordleAnalysis


text:
#=
Discrete point prediction with subpixel location estimation
=#

function calculate_whiskers(han::Tracker_Handles,total_frames=han.max_frames,batch_size=32,loading_size=128)

    calculate_whiskers(han.nn,han.wt.vid_name,total_frames,batch_size,loading_size,prog_bar=han.b["dl_predict_prog"])
end

function calculate_whiskers(nn,vid_name,total_frames,batch_size=32,loading_size=128; prog_bar=nothing)

    vid_w=640
    vid_h=480
    fps=25

    Knet.knetgcinit(Knet._tape)
    Knet.gc()
    @assert rem(loading_size,batch_size) == 0

    w=size(nn.imgs,1)
    h=size(nn.imgs,2)

    batch_per_load = div(loading_size,batch_size)

    k_mean = convert(CuArray,nn.norm.mean_img)

    temp_frames=zeros(UInt8,vid_w,vid_h,1,loading_size)
    temp_frames_cu=convert(CuArray,zeros(UInt8,vid_w,vid_h,1,loading_size))
    input_images_cu_r = convert(CuArray,zeros(Float32,w,h,1,loading_size))

    sub_input_images=convert(KnetArray{Float32,4},zeros(Float32,w,h,1,batch_size))

    input_f=convert(KnetArray{Float32,4},zeros(Float32,64,64,nn.features,loading_size))
    input=zeros(Float32,64,64,nn.features,loading_size)
    input_fft=zeros(Complex{Float32},64,64,nn.features,loading_size)
    input_fft=convert(SharedArray,input_fft)

    input_cu = convert(CuArray,zeros(Float32,64,64,nn.features,loading_size))
    input_cu_com = convert(CuArray,zeros(Complex{Float32},64,64,nn.features,loading_size))

    preds=zeros(Float32,nn.features,3,total_frames)
    preds=convert(SharedArray,preds)

    kernel_pad = StackedHourglass.create_padded_kernel(size(nn.labels,1),size(nn.labels,2),1)
    k_fft = fft(kernel_pad)

    if prog_bar != nothing
        set_gtk_property!(prog_bar,:fraction,0.0)
    end
    set_testing(nn.hg,false)

    frame_num = 1

    while (frame_num < div(total_frames,loading_size)*loading_size)

        @ffmpeg_env run(WhiskerTracking.ffmpeg_cmd(frame_num/fps,vid_name,loading_size,"test5.yuv"))
        read!("test5.yuv",temp_frames)

        temp_frames_cu[:]=convert(CuArray,temp_frames)
        StackedHourglass.CUDA_preprocess(temp_frames_cu,input_images_cu_r)
        StackedHourglass.CUDA_normalize_images(input_images_cu_r,k_mean)

        input_images = convert(KnetArray,input_images_cu_r)

        for k=0:(batch_per_load-1)

            copyto!(sub_input_images,1,input_images,k*w*h*batch_size+1,w*h*batch_size)
            myout=nn.hg(sub_input_images)
            copyto!(input_f,k*64*64*nn.features*batch_size+1,myout[4],1,length(myout[4]))
            for kk=1:length(myout)
                Knet.freeKnetPtr(myout[kk].ptr)
            end
        end
        Knet.freeKnetPtr(input_images.ptr)

        input_cu[:]=(CuArray(input_f))
        input_cu_com[:]=fft(input_cu,(1,2))
        input_fft[:] = convert(Array{Complex{Float32},4},input_cu_com)

        input[:]=convert(Array,input_f)
        hi=argmax(input,dims=(1,2))
        for jj=1:size(hi,4)
            for kk=1:size(hi,3)
                preds[kk,3,jj + frame_num-1] = input[hi[1,1,kk,jj][1],hi[1,1,kk,jj][2],kk,jj]
            end
        end

        StackedHourglass.calculate_subpixel(preds,frame_num-1,input_fft,k_fft)

        if prog_bar != nothing
            set_gtk_property!(prog_bar,:fraction,frame_num/total_frames)
        end
        frame_num += loading_size
        sleep(0.0001)
    end

    preds[:,1,:] = preds[:,1,:] ./ 64 .* vid_w
    preds[:,2,:] = preds[:,2,:] ./ 64 .* vid_h

    if prog_bar != nothing
        set_gtk_property!(prog_bar,:fraction,1.0)
    end

    set_testing(nn.hg,true)

    convert(Array,preds)
end

function predict_single_frame(han::Tracker_Handles)

    k_mean = han.nn.norm.mean_img

    temp_frame = convert(Array{Float32,2},han.current_frame)
    temp_frame = imresize(temp_frame,(256,256))

    temp_frame = StackedHourglass.normalize_new_images(temp_frame,k_mean)

    temp_frame = convert(KnetArray,reshape(temp_frame,(256,256,1,1)))

    set_testing(han.nn.hg,false) #Turn off batch normalization for prediction
    myout=han.nn.hg(temp_frame)[4]
    set_testing(han.nn.hg,true) #Turn back on

    hi=argmax(myout,dims=(1,2))
    preds=zeros(Float32,2,han.nn.features)
    confidences=zeros(Float32,han.nn.features)

    for kk=1:size(hi,3)
        confidences[kk] = myout[hi[1,1,kk,1][1],hi[1,1,kk,1][2],kk,1]
    end

    kernel_pad = StackedHourglass.create_padded_kernel(size(myout,1),size(myout,2),1)

    k_fft = fft(kernel_pad)
    input=fft(convert(Array,myout),(1:2))

    for i=1:size(myout,3)
        preds[:,i]=StackedHourglass.subpixel(input[:,:,i,1],k_fft,4) .+ 32.0
    end

    (preds', confidences)
end


text:
module ProceduralNoise

using LinearAlgebra
using StaticArrays
using Base.Threads:@threads

include("utils.jl")
include("gradient.jl")

export perlin, perlin!, perlin2d!

end # module


text:
module ONSCodes

    using Query
    using CSV
    using DataFrames

    using GlobalDecls
    using Utils

    export load_las, region_name_from_code, region_name_from_name, region_code_from_code, region_code_from_ccode
    export region_code_from_name, code_from_name, is_aggregate, create_brma_lookup, pick_brma_at_random

    """
    This loads the file of ONS codes and names I downloaded from [here]().
    """
    function load_las( name :: AbstractString ) :: DataFrame
        df = CSV.File(
            name,
            delim='\t' ) |> DataFrame
        lcnames = Symbol.(lowercase.(basiccensor.(string.(names(df)))))
        rename!(df,lcnames)
        for n in names(df)
            df[!,n]=map( n->strip(n), df[!,n])
        end
        df
    end

    """
    Broad Rental Market Areas (BRMA)
    """
    function create_brma_lookup()
        las = load_las( DATADIR*"las/all_las.tab" )
        brmap = CSV.File( DATADIR*"las/brmas/ladistrict_2_brma.csv" ) |> DataFrame
        lcnames = Symbol.(lowercase.(basiccensor.(string.(names(brmap)))))
        rename!(brmap,lcnames)
        for n in names(brmap)
            if typeof( brmap[n])==String
                brmap[n]=map( n->strip(n), brmap[n])
            end
        end
        println( brmap )
        brlook = make_brma_lookup(0)
        nlas = size(las)[1]
        for r in 1:nlas
            la = las[r,:]
            which = brmap.la .== la.old_ons_code
            brlas = brmap[which,:]
            ccode = la.new_gss_code
            n = size( brlas )[1]
            println( la.new_gss_code )
            println( brlas )
            println( n )
            out = [ccode, n, missing, missing, missing, missing, missing, missing, missing, missing, missing, missing ]
            for k in 1:n
                brla = brlas[k,:]
                out[k+2] = basiccensor(brla.brmaname)
            end
            println( out )
            push!( brlook, out )
        end
        # FIXME general load of this sort
        brvalues = CSV.File( DATADIR*"las/brmas/2019-20_LHA_TABLES.csv" ) |> DataFrame
        lcnames = Symbol.(lowercase.(basiccensor.(string.(names(brvalues)))))
        rename!(brvalues,lcnames)
        brvalues[:brma]=map( n->basiccensor(n), brvalues[:brma])

        CSV.write( DATADIR*"las/brmas/2019-20_LHA_TABLES_EDITED.csv", brvalues )
        CSV.write( DATADIR*"las/brmas/brma_lookup.csv", brlook )

        (brlook=brlook,brvalues=brvalues)
    end

    LAMAPPINGS = load_las( DATADIR*"las/all_las.tab" )
    BRVALUES_2019 = CSV.File( DATADIR*"las/brmas/2019-20_LHA_TABLES_EDITED.csv" ) |> DataFrame
    BRVALUES_2020 = CSV.File( DATADIR*"las/brmas/2020-21_LHA_TABLES_EDITED.csv" ) |> DataFrame
    BRVALUES = BRVALUES_2020
    BRLOOKUP = CSV.File( DATADIR*"las/brmas/brma_lookup.csv" ) |> DataFrame

    function make_brma_lookup( n :: Integer ) :: DataFrame
        DataFrame(
            ccode     = Vector{AbstractString}(undef,n),
            n         = Vector{Integer}(undef,n),
            brma1     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma2     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma3     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma4     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma5     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma6     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma7     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma8     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma9     = Vector{Union{AbstractString,Missing}}(missing,n),
            brma10     = Vector{Union{AbstractString,Missing}}(missing,n))
    end

    function pick_brma_at_random( ccode :: AbstractString ) :: AbstractString
        which =BRLOOKUP.ccode .== ccode
        row = BRLOOKUP[which,:]
        # println(typeof(row))
        if size( row )[1] > 0
            n = row[1,:].n
            if n > 0
                r = rand(1:n)
                brs = Symbol("brma$r")
                return row[1,brs]
            end
        end
        return ""
    end

    function get_brma_value( ccode :: AbstractString, field :: Symbol = :cat_a ) :: Real
        name = pick_brma_at_random( ccode )
        v = -1.0
        if( name != "")
            which = BRVALUES.brma .== name
            vv = BRVALUES[which,field]
            if length( vv )[1] > 0
                v = vv[1]
            else
                println( "getbrvalue failed for ccode=|$ccode| name=|$name|")
            end
        end
        v
    end

    function region_name_from_name( name :: AbstractString )
           q = @from i in LAMAPPINGS begin
               @where (i.name == name )
               @select lad = i.region
           end
           s=missing
           for i in q
               s = i
           end
           s
    end

    function region_name_from_code( code :: AbstractString )
           q = @from i in LAMAPPINGS begin
               @where (i.new_gss_code == code )
               @select lad = i.region
           end
           s=missing
           for i in q
               s = i
           end
           s
    end

    function region_code_from_code( name :: AbstractString )
        region = region_name_from_code( name )
        return code_from_name( region )
    end

    function region_code_from_name( name :: AbstractString )
        region = region_name_from_name( name )
        return code_from_name( region )
    end

    function code_from_name( name :: Missing )
        missing
    end

    function code_from_name( name :: Nothing )
        missing
    end

    """
    returns the ONS code for the given LA/Region name
    """
    function code_from_name( name :: AbstractString )
        # println( "name $name " )
       q = @from i in LAMAPPINGS begin
           @where basiccensor(i.name) == basiccensor(name)
           @select lad = i.new_gss_code
       end
       s=missing
       for i in q
           s = i
       end
       s
    end

    """
    Another hack - faster lookup for ccode->region since the Query version
    is paralyising things.
    """
    function make_region_lookup( LAS :: DataFrame ) :: Dict
        d = Dict()
        for i in eachrow( LAS )
            d[i.new_gss_code] = region_code_from_code( i.new_gss_code )
        end
        d
    end

    CCODE_TO_RCODE_LOOKUP = make_region_lookup( LAMAPPINGS )

    """
    faster version of below, using a simple Dict
    """
    function region_code_from_ccode( ccode :: AbstractString ) :: AbstractString
        return CCODE_TO_RCODE_LOOKUP[ccode]
    end

    function is_aggregate( ccode :: Union{Missing,AbstractString} )
        ctype = typeof( ccode )
        println( "is_aggregate; ccode $ccode type $ctype " )
        if ccode === missing
            return true
        end
        if length( ccode ) <= 1 || ccode=="NA" # different ways of loading a dataframe produce either missing or 0 length string
            return true
        end
        (ccode[1:3] in ["E92","E12"])
    end

end # LAMappings