text:
### A Pluto.jl notebook ###
# v0.17.3

using Markdown
using InteractiveUtils

# ╔═╡ c730d000-5ed6-11ec-36b6-39741ccad6c7
begin
	using Pkg; Pkg.activate(@__DIR__); Pkg.instantiate()
	Pkg.precompile()

	using PlutoUI
	using BenchmarkTools
	using DataFrames
	using DelimitedFiles
	using CSV
	using XLSX
	using Downloads
	using JLD, NPZ, MAT, RData
	using RCall
end;

# ╔═╡ 62c5d4f2-9776-43ae-8125-0d429d6cde42
PlutoUI.TableOfContents(aside=true, indent=true, depth=3)

# ╔═╡ 34f92fb1-6c1e-4db6-b795-8a526007ea92
html"""

"""

# ╔═╡ 13a10dac-8c7e-4e72-9fa9-d58ca8dd0202
md"""
# 💾 Data

Being able to easily load and process data is a crucial task that can make any data science more pleasant. In this notebook, we will cover most common types often encountered in data science tasks, and we will be using this data throughout the rest of this tutorial.

[YouTube link](https://www.youtube.com/watch?v=iG1dZBaxS-U)
"""

# ╔═╡ 5100df3f-c5d3-4346-bcd9-8afc92bc559c
md"""
## Get some data

In Julia, it's pretty easy to dowload a file from the web using the `download` function. But also, you can use your favorite command line commad to download files by easily switching from Julia via the `;` key. Let's try both.

**Note:** `download` depends on external tools such as curl, wget or fetch. So you must have one of these.
"""

# ╔═╡ abe62085-b630-4816-af11-b39133892849
data₁ = Downloads.download("https://raw.githubusercontent.com/nassarhuda/easy_data/master/programming_languages.csv", "programminglanguages.csv")

# ╔═╡ 51c909b1-1d37-458f-acd6-68f4041bc92f
md"""
## Read your data from text files

The key question here is to load data from files such as `csv` files, `xlsx` files, or just raw text files. We will go over some Julia packages that will allow us to read such files very easily.
"""

# ╔═╡ 6483858c-2e58-4930-b4e9-76b47032eac4
md"""
### DelimitedFiles.jl

Let's start with the package `DelimitedFiles` which is in the standard library. When loaded, the file is stored as a `Matrix`.

> **readdlm**(source,
> delim::AbstractChar,
> T::Type,
> eol::AbstractChar;
> header=false,
> skipstart=0,
> skipblanks=true,
> use_mmap,
> quotes=true,
> dims,
> comments=false,
> comment_char='#')
"""

# ╔═╡ 89a5a316-5058-4ff2-8d71-fc832a84c58d
begin
	data₂, header = readdlm("programming_languages.csv",',';header=true)
	header
end

# ╔═╡ 853a4e90-ac39-4d73-9953-c7b70e8f78ef
data₂

# ╔═╡ 3a22c018-2ca7-41df-9ee4-91dc8b64d940
md"""
> **Note:** the output table is a Matrix (2D Array).

We can use the function `writedlm`, from `DelimitedFiles.jl` package, to write to a text file...
"""

# ╔═╡ b4fcdcf9-793f-4d4a-a5fb-950108cb8188
writedlm("output/programminglanguages_dlm.txt", data₂, '-')

# ╔═╡ cfa37c7c-810c-4095-8887-262dee4a9d04
md"""
### CSV.jl

A more powerful package to use here is the `CSV.jl` package. By default, this package imports the data to a `DataFrame`, which can have several advantages as we will see below.

_In general, `CSV.jl` is the recommended way to load CSVs in Julia_. Only use `DelimitedFiles.jl` when you have a more complicated file where you want to specify several things.
"""

# ╔═╡ 4b651b5a-e562-46ef-a3b5-8ee85b5a61ed
df = CSV.read("programming_languages.csv", DataFrame)

# ╔═╡ 11471bbf-b7b4-4b6e-90cb-9e9ff47bbe4a
md"""
We can use the `typeof` function to make sure that the output is a DataFrame.
"""

# ╔═╡ 2652dc01-d80d-46d9-ae7a-9afa534e88cb
typeof(df)

# ╔═╡ b2f35e6b-8c19-4729-87a3-292858bfbcb5
md"""
We can use the macro `@btime`, from `BenchMarkTools.jl`, to prove that `CSV.jl` is faster than `DelimitedFiles.jl`...

**Note:** Check Julia REPL to see the processing time.
"""

# ╔═╡ e1210609-a216-4d54-be63-04f10c5c15bc
@btime df₁, h₁ = readdlm("programming_languages.csv",',';header=true);

# ╔═╡ 008b635f-c4e5-475a-83af-4bc3fc722344
@btime df₂ = CSV.read("programming_languages.csv", DataFrame);

# ╔═╡ 934c8549-c86c-45e6-a9f5-56297c73513d
md"""
We can use the function `write`, from `CSV.jl` package, to write to a text file...

**Note:** if we want to write a Matrix, we could first convert it into a DataFrame (using the `DataFrame` function) and, then, write it using the `write` function.
"""

# ╔═╡ 92708ece-3620-4cf5-b09d-098002044623
CSV.write("output/programminglanguages_CSV.csv", df)

# ╔═╡ cac4f1b5-f9e3-469e-9576-553a1cf1b363
md"""
### Some DataFrame Operations

There are different ways to slice a DataFrame. Suppose that we want just the first 10 rows of a DataFrame...
"""

# ╔═╡ 3d378ec4-659f-4032-a5b7-6d6b6d402c2f
df[1:10,:]

# ╔═╡ e50ae64c-ff64-4597-bf75-d7c921d23225
md"""
Suppose that we want all the instances of a given feature (column)...
"""

# ╔═╡ d9ea3574-2ffb-4fdb-9ced-db1dc669ecd9
df[!,"year"]

# ╔═╡ 09ff6d8b-15f2-4d8e-a775-dee357a417e4
df.year

# ╔═╡ 843a668f-7b08-4bce-b02a-3153d0eb7d0d
df[!,"year"] == df.year

# ╔═╡ 159a35f4-09fd-47bd-9935-88668a7e7f27
typeof(df.year)

# ╔═╡ 494ebede-80d0-430e-91aa-0a9050a1bb29
md"""
We can use the `names` function to get an array of all column names...
"""

# ╔═╡ 00d12274-3057-4a52-8400-0c8c3c358e6b
names(df)

# ╔═╡ fede3ce2-da03-4ff9-a13e-7ce55ec05bf6
md"""
We can use the `describe` function to check some data stats...
"""

# ╔═╡ 74783bbb-3c9d-4e15-bc54-b3d8e7c2d230
describe(df)

# ╔═╡ e4b3a444-c487-4fe8-8048-873b8a3affb7
md"""
Let's create two DataFrames...
"""

# ╔═╡ 0becba29-d5ff-4595-86ab-6b0575674047
begin
	foods = ["Apple","Cucumber","Tomato","Banana"]
	calories = [105,47,22,105]

	df_cal = DataFrame(Item=foods, Calories=calories)
end

# ╔═╡ 9d522bdc-d066-40ac-84c4-47287d9e07e0
begin
	prices = [0.85,1.6,0.8,0.6]

	df_price = DataFrame(Item=foods, Price=prices)
end

# ╔═╡ 79de9741-c913-4c00-8833-4f1b68f80ea5
md"""
Now, we can use the `innerjoin` function to join both DataFrames on `Item` column...
"""

# ╔═╡ fd724d83-41bb-422c-860e-2bbeff17e0dc
df_comb = innerjoin(df_cal, df_price, on=:Item)

# ╔═╡ 51301eeb-709a-4ea9-a1a5-d7967f8dd55d
md"""
### XLSX.jl

Another type of files that we may often need to read is XLSX files. Let's try to read this type of file using the `XLSX.jl` package...

> **readdata**(filepath, sheetname, cellrange)
"""

# ╔═╡ 64e7e90b-cd7d-4569-a715-4f27f1787e40
xlsx₁ = XLSX.readdata("data/zillow_data_download_april2020.xlsx",
    			  	  "Sale_counts_city",
    			  	  "A1:F9")

# ╔═╡ e5f2c69b-fbd8-4048-aaed-792a64a4d66c
md"""
If you don't want to specify cell ranges, you can use the `readtable` function. However, this will take a little longer...

> **readtable**(filepath, sheetname)
"""

# ╔═╡ 26047637-1b77-4203-b705-e34f74137295
xlsx₂ = XLSX.readtable("data/zillow_data_download_april2020.xlsx",
					   "Sale_counts_city")

# ╔═╡ 0f3d505c-a575-419f-8155-fd071f510714
typeof(xlsx₂)

# ╔═╡ c2f1eab2-862a-4eb8-b233-71567cdc9963
md"""
Here, `xlsx₂` is a tuple of two items. The first item is an vector of vectors where each vector corresponds to a column in the excel file...
"""

# ╔═╡ 3a924b3e-3dbe-49b6-9e5e-5a48b0ffd385
xlsx₂[1]

# ╔═╡ 58bad348-0586-4d0f-8d54-7c3e1a7a2c40
md"""
We can access the first column by typing...
"""

# ╔═╡ 036de982-36ad-431a-8445-517178626911
xlsx₂[1][1]

# ╔═╡ 4ead9039-f4c2-4f82-bf1e-2146718014a3
md"""
We can access the third column first 10 elements...
"""

# ╔═╡ bb04dc6c-e62e-4b2c-a247-ebe4836d8013
xlsx₂[1][3][begin:10]

# ╔═╡ d9550872-fa7c-4ebd-9a4d-828182c8e49d
md"""
On the other hand, the second is the header with the column names...
"""

# ╔═╡ e01a65ab-394a-4718-b9ae-ca2bb30e20b5
xlsx₂[2]

# ╔═╡ 1eb26b3c-8a5a-4648-bb73-f8291b656edc
md"""
We could check the names of the last 10 columns...
"""

# ╔═╡ cb027055-2cf6-4f74-b265-9d01c23c2967
xlsx₂[2][end-9:end]

# ╔═╡ 12dccfc3-e976-46e6-81b0-dc51b48309f4
md"""
And we can easily store this data in a `DataFrame`. We can use the "splat" operator to unwrap these arrays and pass them to the DataFrame constructor...
"""

# ╔═╡ 403ac783-9c02-4fd1-8506-afe0af36a14b
df_xlsx = DataFrame(xlsx₂...)
# df_xlsx = DataFrame(xlsx₂[1], xlsx₂[2])

# ╔═╡ 36d8898e-980c-4f54-bbff-277ded2229ea
md"""
**Note:** the first argument is the actual data and the second one comprises the column names.

We can also easily write data to an XLSX file (takes several amount of time)...
"""

# ╔═╡ 8d769d53-1902-4a09-8228-ce60174b22bb
# XLSX.writetable("output/writefile_using_XLSX.xlsx",xlsx₂[1],xlsx₂[2])

# ╔═╡ bc29d2b5-e6c2-4b45-8b79-cbb012659be1
md"""
## Importing your data

Often, the data you want to import is not stored in plain text, and you might want to import different kinds of types. Here we will go over importing `jld`, `npz`, `rda`, and `mat` files. Hopefully, these four will capture the types from four common programming languages used in Data Science (Julia, Python, R and Matlab).

We will use a toy example here of a very small matrix. But the same syntax will hold for bigger files.
"""

# ╔═╡ 7c2fef9c-4e9a-4637-bc8e-6636af5f98ae
md"""
### Julia Data (JLD)

JLD, for which files conventionally have the extension `.jld`, is a widely-used format for data storage with the Julia programming language.

JLD is a specific "dialect" of HDF5, a cross-platform, multi-language data storage format most frequently used for scientific data. By comparison with "plain" HDF5, JLD files automatically add attributes and naming conventions to preserve type information for each object.
"""

# ╔═╡ 8f51aeb5-3e75-4e5a-8c8f-ed0201eb8ce0
# load JLD file
jld_data = JLD.load("data/mytempdata.jld")

# ╔═╡ 879ac394-8598-4399-957f-1280ecd99478
# see the data
jld_data["tempdata"]

# ╔═╡ 0d27e3b3-79e3-424d-bc6b-94252900f7fb
# save JLD file
save("output/mywrite.jld", "A", jld_data)

# ╔═╡ 13053d54-8680-41e2-b416-8a7bc50f6c92
# JLD type
typeof(jld_data)

# ╔═╡ e0f49c77-0c0d-4740-9227-5619d14748b6
md"""
### Numpy Data (NPZ)

Several arrays into a single file in uncompressed Numpy `.npz` format.
"""

# ╔═╡ f4ddf1bf-0cbf-47c5-bb5f-f8abf0fb8ac2
# load NPZ file
npz_data = npzread("data/mytempdata.npz")

# ╔═╡ a979107e-f2cf-42da-aefa-5cb249bc7073
# save NPZ file
npzwrite("output/mywrite.npz", npz_data)

# ╔═╡ edd2093e-f84a-4563-acb7-219f2bd07399
# NPZ type
typeof(npz_data)

# ╔═╡ be91882d-fe73-40d0-8e75-30c2e8ddd564
md"""
### R Data (RDA)

The RData format (usually with extension `.rdata` or `.rda`) is a format designed for use with R, a system for statistical computation and related graphics, for storing a complete R workspace or selected "objects" from a workspace in a form that can be loaded back by R.
"""

# ╔═╡ 1f3092e8-3f3a-4d29-a9f5-e64ef96c13a8
# load RDA file
R_data = RData.load("data/mytempdata.rda")

# ╔═╡ cf9a1b00-79dd-4305-93ad-69f229319ce2
# see the data
R_data["tempdata"]

# ╔═╡ 1b49419d-18cd-4617-9442-15080e3ef588
begin
	# save RDA file
	@rput R_data
	R"save(R_data, file=\"output/mywrite.rda\")"
end

# ╔═╡ c3bce6db-d503-4a68-b83f-278c6d603f07
md"""
**Note:** we must use the `RCall.jl` package to save `.rda` files.
"""

# ╔═╡ 7131ac1c-984e-4ad9-a1cd-b1d46a4f240e
# RDA type
typeof(R_data)

# ╔═╡ 4f70b5df-0026-43cd-b769-d4ca50145fbd
md"""
### MATLAB® Data (MAT)

MAT-files are binary MATLAB® `.mat` files that store workspace variables.
"""

# ╔═╡ 0c1201b9-6acb-4206-a806-e09e04c6a65e
# load MAT file
matlab_data = matread("data/mytempdata.mat")

# ╔═╡ cc555ffc-0eea-4e35-b0d1-36d3a734f9f2
# see the data
matlab_data["tempdata"]

# ╔═╡ 5516dd98-3a87-4801-a62e-e91a95e4923a
# save MAT file
matwrite("output/mywrite.mat", matlab_data)

# ╔═╡ 53be8986-1f06-45a5-9303-5018a3ddf06f
# MAT type
typeof(matlab_data)

# ╔═╡ ecefa934-49a8-4abf-880d-d91d3f9f9d05
md"""
All files, when loaded, are `Dict`. The only exception is NPZ data, which is loaded as an `Matrix`.
"""

# ╔═╡ 01be19ba-2aaa-4746-af5d-8ce7f1cf3a55
md"""
## Processing different types of data

We will mainly cover `Vector` (`Matrix` included), `DataFrame`, and `Dict`. Let's bring back our programming languages dataset `data₂` and start playing it the matrix it's stored in.
"""

# ╔═╡ 3a6f9dc0-2186-440a-9d78-5da6acabcdae
md"""
### Matrix
"""

# ╔═╡ 4f9dca90-9f1a-4c0e-a510-fa640d109b0d
data₂

# ╔═╡ 57b8265f-be5c-4f05-91c6-6c4fec4f0a6b
md"""
Here are some quick questions we might want to ask about this simple data.

> **Q1:** Which year was was a given language invented?
"""

# ╔═╡ 157e9e3d-2e10-418e-997e-0178c14d7b3d
function year_created_mtx(data, language::String)
    loc = findfirst(data[:,2] .== language)
    !isnothing(loc) && return data[loc,1]
	error("Error: Language not found!")
end

# ╔═╡ e7bd695e-c7d6-4b38-8520-7d915e740015
md"""
**Note:** this function return the year just when `loc` local variable is not `nothing`. Otherwise, it returns an error message.
"""

# ╔═╡ c9ead4e0-aa01-43bc-8107-72d8ec4c073b
begin
	julia_year1 = year_created_mtx(data₂, "Julia")

	"Julia was created in $julia_year1."
end

# ╔═╡ 92b4a2d9-b271-42b3-97d1-647300a9439a
begin
	cobol_year1 = year_created_mtx(data₂, "COBOL")

	"COBOL was created in $cobol_year1."
end

# ╔═╡ 01a73335-69b9-4297-9910-a441848fe2c5
md"""
> **Q2:** How many languages were created in a given year?
"""

# ╔═╡ fbf69873-af41-428c-9bba-d63cdb20149b
function langs_per_year_mtx(data, year::Int64)
	count = length(findall(data[:,1] .== year))

	return count
end

# ╔═╡ 1ea19d7e-5748-4559-b2d2-fd4f4d449516
begin
	langs_1988₁ = langs_per_year_mtx(data₂, 1988)

	"In 1988, $langs_1988₁ language(s) was/were created."
end

# ╔═╡ 2592f7cd-95ec-493c-93e1-bbedf433367f
begin
	langs_2006₁ = langs_per_year_mtx(data₂, 2006)

	"In 2006, $langs_2006₁ language(s) was/were created."
end

# ╔═╡ 12121f45-8a7b-4709-bc85-b2fde7313a21
md"""
### DataFrame

Now let's try to store this data in a DataFrame...
"""

# ╔═╡ d455f80e-c5b5-4b1a-9667-86fade5ee3d3
# anonymous column names
data₃ = DataFrame(data₂, :auto)

# ╔═╡ 42c72fde-8331-4e40-b761-c50d0e2e8486
# specifying column names
data₄ = DataFrame(Year = data₂[:,1], Language = data₂[:,2])

# ╔═╡ 201a8534-91ee-4ac9-a120-51b29f196c23
# specifying both column names and data types
data₅ = DataFrame(Year = Int.(data₂[:,1]), Language = string.(data₂[:,2]))

# ╔═╡ 0b14e9fc-62e6-4ad7-ab67-eb36e340749c
md"""
> **Q1:** Which year was was a given language invented?
"""

# ╔═╡ 679b2b83-07f9-40bd-bdd1-89e7233bfade
function year_created_df(df,language::String)
    loc = findfirst(df.Language .== language)
	!isnothing(loc) && return df.Year[loc]
	error("Error: Language not found!")
end

# ╔═╡ f4738476-8b05-4dbc-a15e-6b3d59c90c50
md"""
**Note:** this function return the year just when `loc` local variable is not `nothing`. Otherwise, it returns an error message.
"""

# ╔═╡ c6b170be-6576-480e-b2ba-65d3b853871a
begin
	julia_year2 = year_created_df(data₅, "Julia")

	"Julia was created in $julia_year2."
end

# ╔═╡ 9977fdaa-db30-43a5-bd58-14087c2163e4
begin
	cobol_year2 = year_created_df(data₅, "COBOL")

	"COBOL was created in $cobol_year2."
end

# ╔═╡ a137ebd7-9db4-4c29-a694-b1acad5214a3
md"""
> **Q2:** How many languages were created in a given year?
"""

# ╔═╡ 2cc2938b-58d5-4c27-bdab-dddf22231bc8
function langs_per_year_df(df, year::Int64)
    count = length(findall(df.Year.==year))
    return count
end

# ╔═╡ ae7a1714-2245-4d5a-89bb-10b3adff787a
begin
	langs_1988₂ = langs_per_year_mtx(data₅, 1988)

	"In 1988, $langs_1988₂ language(s) was/were created."
end

# ╔═╡ ce4ac86d-5e06-4c3e-93d6-ebdee46439bd
begin
	langs_2006₂ = langs_per_year_mtx(data₅, 2006)

	"In 2006, $langs_2006₂ language(s) was/were created."
end

# ╔═╡ 515b7479-97d8-464b-8245-1034f3cd8bdc
md"""
### Dictionary

Next, we'll use dictionaries. A quick way to create a dictionary is with the `Dict()` command. But this creates a dictionary without types. Here, we will specify the types of this dictionary.
"""

# ╔═╡ 3b08a2c8-546b-4a6c-8ae0-2a1e64578af8
# dict from a list of tuples
Dict([("A", 1), ("B", 2)])

# ╔═╡ 21551a38-57fc-42c2-81a2-b314bb06250b
md"""
**Note:** the key type is `String`, while the value type is `Int64`.
"""

# ╔═╡ 6da1f5a7-56a2-4a7a-abfc-25ec3f44f892
# dictionary from a list of tuples
Dict([("A", 1), ("B", 2), ("C", [1,2,3])])

# ╔═╡ 43ad8297-d94c-45a0-941f-ab0d5bfd2ddb
md"""
**Note:** the key type is `String`, while the value type is `Any`, since we have integers and arrays as values.
"""

# ╔═╡ 9390d98f-94eb-48bb-ba9c-e6b3f67f762f
# empty dict
Dict()

# ╔═╡ 30718df7-5085-4d0d-8a86-7b4a08ff20b2
# empty dict, specifying that keys are Integers and values are Vectors of Strings
dict₁ = Dict{Integer, Vector{String}}()

# ╔═╡ f77a5189-19b4-4f99-b394-72190253912f
# appending a key-value pair to dict
dict₁[2012] = ["Julia", "Programming", "Language"]

# ╔═╡ 50ba8f3b-5104-4d4f-85ea-831d4afe9164
# updated dict
dict₁

# ╔═╡ bb814af7-62e3-4744-955f-31b557540057
md"""
Now, let's populate the dictionary with years as keys and vectors that hold all the programming languages created in each year as their values. Even though this looks like more work, we often need to do it just once.
"""

# ╔═╡ 9193df28-7ed1-4853-a97b-aa5a1232e07b
begin
	lang_dict = Dict{Integer, Vector{String}}()

	for i in 1:size(data₂,1)
		year, lang = data₂[i,:]
		if year ∈ keys(lang_dict)
			lang_dict[year] = push!(lang_dict[year], lang)
		else
			lang_dict[year] = [lang]
		end
	end
end

# ╔═╡ c5aca233-9a26-4800-9b8d-7f0f03852626
lang_dict

# ╔═╡ 7b2d5919-9629-4ca9-8195-f26183734044
md"""
> **Note:** there is an `enumerate()` method in Julia which is useful when you need not only the values `x` over which you are iterating, but also the number of iterations (index) so far.
**Syntax:**
```julia
for (index, value) in enumerate(df.column)
```
"""

# ╔═╡ d8054919-1554-4c73-a4a2-09383ee09224
md"""
We can check the size of the dictionary...
"""

# ╔═╡ 7d57f6cc-ab44-4658-a73e-92c873111155
length(keys(lang_dict))

# ╔═╡ a322a0e7-3984-4c00-95a5-7b23fc6ec06f
md"""
Or we can check the size of the vector made up by unique `Year` values...
"""

# ╔═╡ e0a28251-a7d6-4c17-b49e-84ffc91c60b6
length(unique(data₂[:,1]))

# ╔═╡ 9073e71f-b922-44a4-a470-5cb081588fcd
md"""
> **Q1:** Which year was was a given language invented?
"""

# ╔═╡ f9325e7b-8df8-448a-98b2-9ea8add93da6
function year_created_dict(dict,language::String)
    keys_vec = collect(keys(dict))
    lookup = map(keyid -> findfirst(dict[keyid] .== language), keys_vec)

    return keys_vec[findfirst((!isnothing).(lookup))]
end

# ╔═╡ c33df74d-dd81-4efa-81d6-a5deb4685f1d
begin
	julia_year3 = year_created_dict(lang_dict, "Julia")

	"Julia was created in $julia_year3."
end

# ╔═╡ a08ad2fd-5d3c-4e76-a059-0608de3e36aa
md"""
> **Q2:** How many languages were created in a given year?
"""

# ╔═╡ fb6df54d-0765-4493-b889-17707ff0cf14
langs_per_year_dict(dict, year::Int64) = length(dict[year])

# ╔═╡ d5e60574-5afd-498c-aedd-1f5ffc4c6bd7
begin
	langs_1988₃ = langs_per_year_dict(lang_dict, 1988)

	"In 1988, $langs_1988₃ language(s) was/were created."
end

# ╔═╡ ee8bcf97-9efa-47f0-a82e-cfaa2efd68e6
md"""
## Missing data

Let's remove the first year value from our data (Matrix) and, after that, create a DataFrame...
"""

# ╔═╡ b3927e97-7516-4a80-bcc1-49614cf67166
begin
	data₂[1,1] = missing

	data₆ = DataFrame(Year=data₂[:,1], Language=data₂[:,2])
end

# ╔═╡ dc7ed277-b044-464e-8ad3-998a83b00d1b
md"""
We can now use the `dropmissing!` function to remove (inplace) the line with missing value...
"""

# ╔═╡ e2b6b920-2e05-4168-af16-07e9611d895d
dropmissing!(data₆)

# ╔═╡ Cell order:
# ╟─c730d000-5ed6-11ec-36b6-39741ccad6c7
# ╟─62c5d4f2-9776-43ae-8125-0d429d6cde42
# ╟─34f92fb1-6c1e-4db6-b795-8a526007ea92
# ╟─13a10dac-8c7e-4e72-9fa9-d58ca8dd0202
# ╟─5100df3f-c5d3-4346-bcd9-8afc92bc559c
# ╠═abe62085-b630-4816-af11-b39133892849
# ╟─51c909b1-1d37-458f-acd6-68f4041bc92f
# ╟─6483858c-2e58-4930-b4e9-76b47032eac4
# ╠═89a5a316-5058-4ff2-8d71-fc832a84c58d
# ╠═853a4e90-ac39-4d73-9953-c7b70e8f78ef
# ╟─3a22c018-2ca7-41df-9ee4-91dc8b64d940
# ╠═b4fcdcf9-793f-4d4a-a5fb-950108cb8188
# ╟─cfa37c7c-810c-4095-8887-262dee4a9d04
# ╠═4b651b5a-e562-46ef-a3b5-8ee85b5a61ed
# ╟─11471bbf-b7b4-4b6e-90cb-9e9ff47bbe4a
# ╠═2652dc01-d80d-46d9-ae7a-9afa534e88cb
# ╟─b2f35e6b-8c19-4729-87a3-292858bfbcb5
# ╠═e1210609-a216-4d54-be63-04f10c5c15bc
# ╠═008b635f-c4e5-475a-83af-4bc3fc722344
# ╟─934c8549-c86c-45e6-a9f5-56297c73513d
# ╠═92708ece-3620-4cf5-b09d-098002044623
# ╟─cac4f1b5-f9e3-469e-9576-553a1cf1b363
# ╠═3d378ec4-659f-4032-a5b7-6d6b6d402c2f
# ╟─e50ae64c-ff64-4597-bf75-d7c921d23225
# ╠═d9ea3574-2ffb-4fdb-9ced-db1dc669ecd9
# ╠═09ff6d8b-15f2-4d8e-a775-dee357a417e4
# ╠═843a668f-7b08-4bce-b02a-3153d0eb7d0d
# ╠═159a35f4-09fd-47bd-9935-88668a7e7f27
# ╟─494ebede-80d0-430e-91aa-0a9050a1bb29
# ╠═00d12274-3057-4a52-8400-0c8c3c358e6b
# ╟─fede3ce2-da03-4ff9-a13e-7ce55ec05bf6
# ╠═74783bbb-3c9d-4e15-bc54-b3d8e7c2d230
# ╟─e4b3a444-c487-4fe8-8048-873b8a3affb7
# ╠═0becba29-d5ff-4595-86ab-6b0575674047
# ╠═9d522bdc-d066-40ac-84c4-47287d9e07e0
# ╟─79de9741-c913-4c00-8833-4f1b68f80ea5
# ╠═fd724d83-41bb-422c-860e-2bbeff17e0dc
# ╟─51301eeb-709a-4ea9-a1a5-d7967f8dd55d
# ╠═64e7e90b-cd7d-4569-a715-4f27f1787e40
# ╟─e5f2c69b-fbd8-4048-aaed-792a64a4d66c
# ╠═26047637-1b77-4203-b705-e34f74137295
# ╠═0f3d505c-a575-419f-8155-fd071f510714
# ╟─c2f1eab2-862a-4eb8-b233-71567cdc9963
# ╠═3a924b3e-3dbe-49b6-9e5e-5a48b0ffd385
# ╟─58bad348-0586-4d0f-8d54-7c3e1a7a2c40
# ╠═036de982-36ad-431a-8445-517178626911
# ╟─4ead9039-f4c2-4f82-bf1e-2146718014a3
# ╠═bb04dc6c-e62e-4b2c-a247-ebe4836d8013
# ╟─d9550872-fa7c-4ebd-9a4d-828182c8e49d
# ╠═e01a65ab-394a-4718-b9ae-ca2bb30e20b5
# ╟─1eb26b3c-8a5a-4648-bb73-f8291b656edc
# ╠═cb027055-2cf6-4f74-b265-9d01c23c2967
# ╟─12dccfc3-e976-46e6-81b0-dc51b48309f4
# ╠═403ac783-9c02-4fd1-8506-afe0af36a14b
# ╟─36d8898e-980c-4f54-bbff-277ded2229ea
# ╠═8d769d53-1902-4a09-8228-ce60174b22bb
# ╟─bc29d2b5-e6c2-4b45-8b79-cbb012659be1
# ╟─7c2fef9c-4e9a-4637-bc8e-6636af5f98ae
# ╠═8f51aeb5-3e75-4e5a-8c8f-ed0201eb8ce0
# ╠═879ac394-8598-4399-957f-1280ecd99478
# ╠═0d27e3b3-79e3-424d-bc6b-94252900f7fb
# ╠═13053d54-8680-41e2-b416-8a7bc50f6c92
# ╟─e0f49c77-0c0d-4740-9227-5619d14748b6
# ╠═f4ddf1bf-0cbf-47c5-bb5f-f8abf0fb8ac2
# ╠═a979107e-f2cf-42da-aefa-5cb249bc7073
# ╠═edd2093e-f84a-4563-acb7-219f2bd07399
# ╟─be91882d-fe73-40d0-8e75-30c2e8ddd564
# ╠═1f3092e8-3f3a-4d29-a9f5-e64ef96c13a8
# ╠═cf9a1b00-79dd-4305-93ad-69f229319ce2
# ╠═1b49419d-18cd-4617-9442-15080e3ef588
# ╟─c3bce6db-d503-4a68-b83f-278c6d603f07
# ╠═7131ac1c-984e-4ad9-a1cd-b1d46a4f240e
# ╟─4f70b5df-0026-43cd-b769-d4ca50145fbd
# ╠═0c1201b9-6acb-4206-a806-e09e04c6a65e
# ╠═cc555ffc-0eea-4e35-b0d1-36d3a734f9f2
# ╠═5516dd98-3a87-4801-a62e-e91a95e4923a
# ╠═53be8986-1f06-45a5-9303-5018a3ddf06f
# ╟─ecefa934-49a8-4abf-880d-d91d3f9f9d05
# ╟─01be19ba-2aaa-4746-af5d-8ce7f1cf3a55
# ╟─3a6f9dc0-2186-440a-9d78-5da6acabcdae
# ╠═4f9dca90-9f1a-4c0e-a510-fa640d109b0d
# ╟─57b8265f-be5c-4f05-91c6-6c4fec4f0a6b
# ╠═157e9e3d-2e10-418e-997e-0178c14d7b3d
# ╟─e7bd695e-c7d6-4b38-8520-7d915e740015
# ╠═c9ead4e0-aa01-43bc-8107-72d8ec4c073b
# ╠═92b4a2d9-b271-42b3-97d1-647300a9439a
# ╟─01a73335-69b9-4297-9910-a441848fe2c5
# ╠═fbf69873-af41-428c-9bba-d63cdb20149b
# ╠═1ea19d7e-5748-4559-b2d2-fd4f4d449516
# ╠═2592f7cd-95ec-493c-93e1-bbedf433367f
# ╟─12121f45-8a7b-4709-bc85-b2fde7313a21
# ╠═d455f80e-c5b5-4b1a-9667-86fade5ee3d3
# ╠═42c72fde-8331-4e40-b761-c50d0e2e8486
# ╠═201a8534-91ee-4ac9-a120-51b29f196c23
# ╟─0b14e9fc-62e6-4ad7-ab67-eb36e340749c
# ╠═679b2b83-07f9-40bd-bdd1-89e7233bfade
# ╟─f4738476-8b05-4dbc-a15e-6b3d59c90c50
# ╠═c6b170be-6576-480e-b2ba-65d3b853871a
# ╠═9977fdaa-db30-43a5-bd58-14087c2163e4
# ╟─a137ebd7-9db4-4c29-a694-b1acad5214a3
# ╠═2cc2938b-58d5-4c27-bdab-dddf22231bc8
# ╠═ae7a1714-2245-4d5a-89bb-10b3adff787a
# ╠═ce4ac86d-5e06-4c3e-93d6-ebdee46439bd
# ╟─515b7479-97d8-464b-8245-1034f3cd8bdc
# ╠═3b08a2c8-546b-4a6c-8ae0-2a1e64578af8
# ╠═21551a38-57fc-42c2-81a2-b314bb06250b
# ╠═6da1f5a7-56a2-4a7a-abfc-25ec3f44f892
# ╟─43ad8297-d94c-45a0-941f-ab0d5bfd2ddb
# ╠═9390d98f-94eb-48bb-ba9c-e6b3f67f762f
# ╠═30718df7-5085-4d0d-8a86-7b4a08ff20b2
# ╠═f77a5189-19b4-4f99-b394-72190253912f
# ╠═50ba8f3b-5104-4d4f-85ea-831d4afe9164
# ╟─bb814af7-62e3-4744-955f-31b557540057
# ╠═9193df28-7ed1-4853-a97b-aa5a1232e07b
# ╠═c5aca233-9a26-4800-9b8d-7f0f03852626
# ╟─7b2d5919-9629-4ca9-8195-f26183734044
# ╟─d8054919-1554-4c73-a4a2-09383ee09224
# ╠═7d57f6cc-ab44-4658-a73e-92c873111155
# ╟─a322a0e7-3984-4c00-95a5-7b23fc6ec06f
# ╠═e0a28251-a7d6-4c17-b49e-84ffc91c60b6
# ╟─9073e71f-b922-44a4-a470-5cb081588fcd
# ╠═f9325e7b-8df8-448a-98b2-9ea8add93da6
# ╠═c33df74d-dd81-4efa-81d6-a5deb4685f1d
# ╟─a08ad2fd-5d3c-4e76-a059-0608de3e36aa
# ╠═fb6df54d-0765-4493-b889-17707ff0cf14
# ╠═d5e60574-5afd-498c-aedd-1f5ffc4c6bd7
# ╟─ee8bcf97-9efa-47f0-a82e-cfaa2efd68e6
# ╠═b3927e97-7516-4a80-bcc1-49614cf67166
# ╟─dc7ed277-b044-464e-8ad3-998a83b00d1b
# ╠═e2b6b920-2e05-4168-af16-07e9611d895d

max_stars_repo_name: fnaghetini/DataScience

max_issues_repo_name: fnaghetini/DataScience

max_forks_repo_name: fnaghetini/DataScience


text:
module MatrixNetworks

using DataStructures
using SparseArrays
using LinearAlgebra
using Printf
using Arpack
using DelimitedFiles
using Random
using Statistics
using KahanSummation

"""
Module ``MatrixNetworks``: Documentation on the module

- Option 1: start with a sparse matrix A:
- example: ``M = MatrixNetwork(A)``
- Option 2: start with row and column indexes for the nonzeros in the matrix
- example: ``M = MatrixNetwork(ei,ej)``

Some available functions: (use ?function_name to get more documentation)
- bfs
- bipartite_matching
- clustercoeffs
- corenums
- cosineknn
- csr_to_sparse
- dfs
- biconnected_components
- dijkstra
- dirclustercoeffs
- floydwarshall
- largest_component
- mst_prim
- scomponents
- sparse_to_csr
- pagerank
- erdos_renyi_undirected
- biconnected_components
- triangles

You can check the readme file here: \n
"https://github.com/nassarhuda/MatrixNetworks.jl/blob/master/README.md"
"""
MatrixNetworks

include("MatrixNetwork.jl")
export MatrixNetwork, sparse_transpose,
    is_undirected, is_connected, is_empty, empty_graph,
    random_edge, undirected_edges, directed_edges

include("scomponents.jl")
include("csr_to_sparse.jl")
include("sparse_to_csr.jl")
include("bipartite_matching.jl")
include("bfs.jl")
include("dfs.jl")
include("clustercoeffs.jl")
include("corenums.jl")
include("floydwarshall.jl")

include("manage_data.jl")
export load_matrix_network, load_matrix_network_metadata, load_matrix_network_all,
    matrix_network_datasets
include("largest_component.jl")
include("cosineknn.jl")
include("dirclustercoeffs.jl")
include("dijkstra.jl")
include("mst_prim.jl")

include("spectral.jl")
export fiedler_vector, sweepcut, spectral_cut, bestset, SweepcutProfile

# export everything to make them accessible as functions
export bipartite_matching, edge_list, create_sparse,
bipartite_matching_setup, bipartite_matching_indicator, bfs,
dfs, clustercoeffs, corenums, scomponents, strong_components_map,
enrich, csr_to_sparse, floydwarshall, largest_component,
sparse_to_csr, cosineknn, dirclustercoeffs, dijkstra, mst_prim, mst_prim_matrix,
csr_to_sparse_matrix, edge_indicator

include("diffusions.jl")
export pagerank, pagerank_power!, personalized_pagerank, seeded_pagerank, stochastic_mult!,
        seeded_stochastic_heat_kernel, stochastic_heat_kernel_series!

include("generators.jl")
export erdos_renyi_undirected, erdos_renyi_directed,
    erdős_rényi_undirected, erdős_rényi_directed,
    chung_lu_undirected, is_graphical_sequence, havel_hakimi_graph,
    pa_graph, preferential_attachment_graph,
    pa_edges!, preferential_attachment_edges!,
    gpa_graph, generalized_preferential_attachment_graph,
    gpa_edges!, generalized_preferential_attachment_edges!,
    roach_graph, lollipop_graph

include("biconnected.jl")
export biconnected_components, biconnected_components!

include("triangles.jl")
export unzip_triangles,triangles

end # end module

ext: jl


text:
# Copyright (c) 2022 Code Komali
#
# This software is released under the MIT License.
# https://opensource.org/licenses/MIT

module HKOpinionDynamicsExample

export hk_model_run_and_plot!

using Agents
using Statistics: mean
using Plots
using Random

mutable struct HKAgent <: AbstractAgent
    id::Int
    old_opinion::Float64
    new_opinion::Float64
    previous_opinion::Float64
end

function hk_model(; numagents = 100, ϵ = 0.2)
    model = ABM(HKAgent, scheduler = Schedulers.fastest, properties= Dict(:ϵ => ϵ))
    Random.seed!(1234)
    for i in 1:numagents
        o = rand()
        add_agent!(model, o, o, -1)
    end
    return model
end

function boundfilter(agent, model)
    filter(
        j -> abs(agent.old_opinion -j) < model.ϵ,
        [a.old_opinion for a in allagents(model)]
    )
end

function agent_step!(agent, model)
    agent.previous_opinion = agent.old_opinion
    agent.new_opinion = mean(boundfilter(agent, model))
end

function model_step!(model)
    for a in allagents(model)
        a.old_opinion = a.new_opinion
    end
end

function terminate(model, s)
    if any(
        !isapprox(a.previous_opinion, a.new_opinion; rtol= 1e-12) for a in allagents(model)
    )
        return false
    else
        return true
    end
end

function model_run(; kwargs...)
    model = hk_model(; kwargs...)
    agent_data,_ = run!(
        model,
        agent_step!,
        model_step!,
        terminate;
        adata = [:new_opinion],
        # make sure to comment the 'when' otherwise you won't get the graph as expected
        #when = terminate # when to collect data, the default is to collect on every step
    )
    return agent_data
end

#data = model_run()

plotsim(data, ϵ) = plot(
    data.step,
    data.new_opinion,
    leg = false,
    group = data.id,
    title = "epsilon = $(ϵ)",
)

function hk_model_run_and_plot!(eps_list=[0.05, 0.15, 0.3] ; kwargs...)
    plots = []
    alldata = []
    for eps in eps_list
        data = model_run(; ϵ = eps)
        push!(alldata, data)
        plt = plotsim(data, eps)
        push!(plots, plt)
    end
    plot(plots..., layout=(3,1))
    savefig("HKOpinionDynamicsPlot.png")
    return alldata
end

#hk_model_run_and_plot!()

end


text:
"""
    TypeK(var_1::TypeL)

Lorem ipsum dolor sit amet.
"""
struct TypeK

  var_1::TypeL

end


text:
# To execute:
# julia> addprocs(4)  # start number of desired worker processes
# julia> require("test_scheduler.jl")
# julia> do_scheduler_test()

using HadoopBlocks.Scheduler

type TestTaskStrRev <: WorkerTask
    str::String
end

function _wrkrfn(wt::TestTaskStrRev)
    return reverse(wt.str)
end

function _callback(wt::TestTaskStrRev, ret)
    println("at callback: reversed string: $ret")
end

function do_scheduler_test()
    println("initializing...")
    prep_remotes()

    for i in 1:10
        qt = QueuedWorkerTask(TestTaskStrRev("parallel $i"), _wrkrfn, _callback, :wrkr_any)
        queue_worker_task(qt)
    end

    for i in 1:10
        qt = QueuedWorkerTask(TestTaskStrRev("sequential $i"), _wrkrfn, _callback, :wrkr_any)
        println(execute_worker_task(qt))
    end
end

max_stars_repo_name: tanmaykm/HadoopBlocks

max_issues_repo_name: tanmaykm/HadoopBlocks

max_forks_repo_name: tanmaykm/HadoopBlocks


text:
# in this I try to fit a model

using CSV, DataFrames, DataFramesMeta
respath = "benchmark/bench_string_results/"
paths = readdir(respath)
@time df = reduce(vcat, CSV.read.(respath.*paths))

# create some features
df[:logn] = log(10,df[:n])
df[:slogn] = df[:strlen].*log(10,df[:n])

only_radix_sort(test, strlen) = test == "julia_lsd_radixsort_elapsed" ? strlen : 0
df[:strlen_radix] = only_radix_sort.(df[:test],df[:strlen])
df[:loge] = log(df[:elapsed])

using GLM
lm(@formula(loge ~ test*logn + test*strlen - 1), df)

df_unique
df[:predicted] = exp.(predict(m, df))

df[[:test, :elapsed, :predicted, :strlen, :n]]


text:
abstract type AbstractRenewableFormulation <: AbstractDeviceFormulation end

abstract type AbstractRenewableDispatchFormulation <: AbstractRenewableFormulation end

struct RenewableFixed <: AbstractRenewableFormulation end

struct RenewableFullDispatch <: AbstractRenewableDispatchFormulation end

struct RenewableConstantPowerFactor <: AbstractRenewableDispatchFormulation end

########################### renewable generation variables ############################################

function activepower_variables(canonical_model::CanonicalModel,
                               devices::IS.FlattenIteratorWrapper{R}) where {R<:PSY.RenewableGen}

    add_variable(canonical_model,
                 devices,
                 Symbol("P_$(R)"),
                 false,
                 :nodal_balance_active;
                 lb_value = x -> 0.0,
                 ub_value = x -> PSY.get_rating(PSY.get_tech(x)) )

    return

end

function reactivepower_variables(canonical_model::CanonicalModel,
                                 devices::IS.FlattenIteratorWrapper{R}) where {R<:PSY.RenewableGen}

    add_variable(canonical_model,
                 devices,
                 Symbol("Q_$(R)"),
                 false,
                 :nodal_balance_reactive)

    return

end

####################################### Reactive Power Constraints ######################################
function reactivepower_constraints(canonical_model::CanonicalModel,
                                    devices::IS.FlattenIteratorWrapper{R},
                                    device_formulation::Type{RenewableFullDispatch},
                                    system_formulation::Type{S}) where {R<:PSY.RenewableGen,
                                                                         S<:PM.AbstractPowerModel}

    range_data = Vector{NamedMinMax}(undef, length(devices))

    for (ix, d) in enumerate(devices)
        tech = PSY.get_tech(d)
        name = PSY.get_name(d)
        if isnothing(PSY.get_reactivepowerlimits(tech))
            limits = (min = 0.0, max = 0.0)
            range_data[ix] = (PSY.get_name(d), limits)
            @warn("Reactive Power Limits of $(name) are nothing. Q_$(name) is set to 0.0")
        else
            range_data[ix] = (name, PSY.get_reactivepowerlimits(tech))
        end
    end

    device_range(canonical_model,
                range_data,
                Symbol("reactiverange_$(R)"),
                Symbol("Q_$(R)"))

    return

end

function reactivepower_constraints(canonical_model::CanonicalModel,
                                    devices::IS.FlattenIteratorWrapper{R},
                                    device_formulation::Type{RenewableConstantPowerFactor},
                                    system_formulation::Type{S}) where {R<:PSY.RenewableGen,
                                                                        S<:PM.AbstractPowerModel}

    names = (PSY.get_name(d) for d in devices)
    time_steps = model_time_steps(canonical_model)
    p_variable_name = Symbol("P_$(R)")
    q_variable_name = Symbol("Q_$(R)")
    constraint_name = Symbol("reactiverange_$(R)")
    canonical_model.constraints[constraint_name] = JuMPConstraintArray(undef, names, time_steps)

    for t in time_steps, d in devices
        name = PSY.get_name(d)
        pf = sin(acos(PSY.get_powerfactor(PSY.get_tech(d))))
        canonical_model.constraints[constraint_name][name, t] = JuMP.@constraint(canonical_model.JuMPmodel,
                                canonical_model.variables[q_variable_name][name, t] ==
                                canonical_model.variables[p_variable_name][name, t] * pf)
    end

    return

end

######################## output constraints without Time Series ###################################
function _get_time_series(devices::IS.FlattenIteratorWrapper{R},
                          time_steps::UnitRange{Int64}) where {R<:PSY.RenewableGen}

    names = Vector{String}(undef, length(devices))
    series = Vector{Vector{Float64}}(undef, length(devices))

    for (ix, d) in enumerate(devices)
        names[ix] = PSY.get_name(d)
        tech = PSY.get_tech(d)
        series[ix] = fill(PSY.get_rating(tech), (time_steps[end]))
    end

    return names, series

end

function activepower_constraints(canonical_model::CanonicalModel,
                                devices::IS.FlattenIteratorWrapper{R},
                                device_formulation::Type{D},
                                system_formulation::Type{S}) where {R<:PSY.RenewableGen,
                                                         D<:AbstractRenewableDispatchFormulation,
                                                         S<:PM.AbstractPowerModel}

    parameters = model_has_parameters(canonical_model)

    if parameters
        time_steps = model_time_steps(canonical_model)
        device_timeseries_param_ub(canonical_model,
                            _get_time_series(devices, time_steps),
                            Symbol("activerange_$(R)"),
                            UpdateRef{R}(Symbol("P_$(R)")),
                            Symbol("P_$(R)"))

    else
        range_data = [(PSY.get_name(d), (min = 0.0, max = PSY.get_rating(PSY.get_tech(d)))) for d in devices]
        device_range(canonical_model,
                    range_data,
                    Symbol("activerange_$(R)"),
                    Symbol("P_$(R)"))
    end

    return

end

######################### output constraints with Time Series ##############################################

function _get_time_series(forecasts::Vector{PSY.Deterministic{R}}) where {R<:PSY.RenewableGen}

    names = Vector{String}(undef, length(forecasts))
    ratings = Vector{Float64}(undef, length(forecasts))
    series = Vector{Vector{Float64}}(undef, length(forecasts))

    for (ix, f) in enumerate(forecasts)
        component = PSY.get_component(f)
        names[ix] = PSY.get_name(component)
        series[ix] = values(PSY.get_timeseries(f))
        ratings[ix] = PSY.get_tech(component).rating
    end

    return names, ratings, series

end

function activepower_constraints(canonical_model::CanonicalModel,
                                 forecasts::Vector{PSY.Deterministic{R}},
                                 device_formulation::Type{D},
                                 system_formulation::Type{S}) where {R<:PSY.RenewableGen,
                                                                     D<:AbstractRenewableDispatchFormulation,
                                                                     S<:PM.AbstractPowerModel}

    if model_has_parameters(canonical_model)
        device_timeseries_param_ub(canonical_model,
                                   _get_time_series(forecasts),
                                   Symbol("activerange_$(R)"),
                                   UpdateRef{R}(Symbol("P_$(R)")),
                                   Symbol("P_$(R)"))
    else
        device_timeseries_ub(canonical_model,
                            _get_time_series(forecasts),
                            Symbol("activerange_$(R)"),
                            Symbol("P_$(R)"))
    end

    return

end

##################################### renewable generation cost ######################################
function cost_function(canonical_model::CanonicalModel,
                       devices::IS.FlattenIteratorWrapper{PSY.RenewableDispatch},
                       device_formulation::Type{D},
                       system_formulation::Type{S}) where {D<:AbstractRenewableDispatchFormulation,
                                                           S<:PM.AbstractPowerModel}

    add_to_cost(canonical_model,
                devices,
                Symbol("P_RenewableDispatch"),
                :fixed,
                -1.0)

    return

end


text:
"""
    Canvas.jl

Julia package for interacting with the Canvas LMS API.
See https://fredrikekre.github.io/Canvas.jl/ for documentation.
"""
module Canvas

import HTTP, JSON, Dates, TimeZones

include("Internals.jl")

####################
## Authentication ##
####################
mutable struct OAuth2
    token::Base.SecretBuffer
    function OAuth2(auth::Base.SecretBuffer)
        finalizer(x -> Base.shred!(x), auth)
        return new(auth)
    end
    OAuth2(auth::String) = OAuth2(Base.SecretBuffer(auth))
    OAuth2(auth::OAuth2) = auth
end

##################
## API endpoint ##
##################
struct CanvasAPI
    uri::HTTP.URI
    auth::OAuth2
    CanvasAPI(uri, auth) = new(HTTP.URI(uri), OAuth2(auth))
end

# Default API
const API = Ref{CanvasAPI}()
setapi!(api::CanvasAPI) = API[] = api
function getapi()
    if isassigned(API)
        return API[]
    else # try to set
        url = get(ENV, "CANVAS_URL", nothing)
        auth = get(ENV, "CANVAS_TOKEN", nothing)
        if url !== nothing && auth !== nothing
            try
                setapi!(Canvas.CanvasAPI(url, auth))
                return getapi()
            catch
                @warn "Both ENV[\"CANVAS_URL\"] and ENV[\"CANVAS_TOKEN\"] found " *
                      "but did not manage to initialize the default API."
            end
        end
        throw(ArgumentError("no default CanvasAPI set"))
    end
end

################################
## Internal request utilities ##
################################

# Set Authorization and User-Agent
function Internals.canvas_headers(headers=nothing; auth=nothing)
    if headers === nothing
        headers = Dict{String,String}()
    else
        headers = Dict{String,String}(h for h in headers)
    end
    if isa(auth, OAuth2) && !haskey(headers, "Authorization")
        headers["Authorization"] = "Bearer $(String(copy(auth.token.data)))"
    end
    if !haskey(headers, "User-Agent")
        headers["User-Agent"] = "Canvas.jl" # Always overwrite this?
    end
    return headers
end

# Preprocess parameters before sending them to HTTP.jl
Internals.process_params(params::Union{AbstractString,Nothing}) = params
function Internals.process_params(params)
    params′ = Dict{String,Any}()
    # Assume it iterates pairs
    for (k, v) in params
        params′[Internals.process_key(k)] = Internals.process_val(v)
    end
    return params′
end
Internals.process_key(k) = string(k)
Internals.process_val(v) = string(v)
Internals.process_val(v::AbstractVector) = map(Internals.process_val, v)
function Internals.process_val(zdt::TimeZones.ZonedDateTime)
    zdt = TimeZones.astimezone(zdt, TimeZones.tz"Z")
    str = Dates.format(zdt, Dates.dateformat"yyyy-mm-ddTHH:MM:SSZ")
    return str
end

############################
## Public request methods ##
############################

"""
    Canvas.request(method, endpoint; kwargs...) -> JSON

General method for HTTP requests to `endpoint`.
Return the resulting JSON structure.

**Keyword arguments:**

 - `api`: the Canvas API with authentication;
 - `params`: dictionary with request parameters.

Remaining keyword arguments are passed to
[`HTTP.request`](https://juliaweb.github.io/HTTP.jl/stable/public_interface/#Requests-1).

**Examples**
Get a single course
```julia
Canvas.request("GET", "/api/v1/courses/:course_id")
```

Post an announcement
```julia
params = Dict(
    "title" => "Hello there!",
    "message" => "General Kenobi!",
    "is_announcement" => true,
)
Canvas.request("POST", "/api/v1/courses/:course_id/discussion_topics"; params=params)
```
"""
function request(method::String, endpoint::String=""; kwargs...)
    r = Internals.request(method, endpoint; kwargs...)
    json = JSON.parse(IOBuffer(HTTP.payload(r)))
    return json
end
# Internal request method, returning a HTTP.Response
function Internals.request(method::String, endpoint::String=""; api::CanvasAPI=getapi(),
                           headers=nothing, params=nothing, kwargs...)
    headers = Internals.canvas_headers(headers; auth=api.auth)
    uri = HTTP.merge(api.uri, path=endpoint)
    params = Internals.process_params(params)
    r = HTTP.request(method, uri, headers; query=params, kwargs...)
    @assert !HTTP.hasheader(r, "Link")
    return r
end

"""
    Canvas.paged_request(method, endpoint; kwargs...)

General method for HTTP requests to `endpoint`.
Return the resulting JSON structure and a dictionary
with page data with links.

# Examples
```julia
Canvas.request("GET", "/api/v1/courses")
```
"""
function paged_request(method::String, endpoint::String=""; kwargs...)
    rs, page_data = Internals.paged_request(method, endpoint; kwargs...)
    # Assume we get an JSON array in each request
    json = mapreduce(r->JSON.parse(IOBuffer(HTTP.payload(r))), append!, rs; init=Dict{String,Any}[])
    return json, page_data
end
# Internal request method for pagination, returning Vector{HTTP.Response} and page_data
function Internals.paged_request(
        method::String, endpoint::String=""; api::CanvasAPI=getapi(),
        headers=nothing, page_limit=typemax(Int),
        start_page="", params=nothing, kwargs...)
    headers = Internals.canvas_headers(headers; auth=api.auth)
    if isempty(start_page) # first request
        uri = HTTP.merge(api.uri, path=endpoint)
        params = Internals.process_params(params)
        r = HTTP.request(method, uri, headers; query=params, kwargs...)
    else
        # isempty(endpoint) || throw(ArgumentError("`start_page` kwarg is incompatible with `endpoint` argument"))
        params === nothing || throw(ArgumentError("`start_page` kwarg is incompatible with `params` kwarg"))
        r = HTTP.request(method, start_page, headers; kwargs...)
    end
    rs = [r]
    page_data = Dict{String, String}()
    if HTTP.hasheader(r, "Link")
        page_count = 1
        while page_count < page_limit
            links = split(HTTP.header(r, "Link"), ",")
            idx = findfirst(x->occursin("rel=\"next\"", x), links)
            idx === nothing && break
            r = HTTP.request(method, match(r"<(.*)>", links[idx]).captures[1], headers; kwargs...)
            push!(rs, r)
            page_count += 1
        end
        links = split(HTTP.header(r, "Link"), ",")
        for page in ("current", "next", "prev", "first", "last")
            idx = findfirst(x->occursin("rel=\"$(page)\"", x), links)
            if idx !== nothing
                page_data[page] = match(r"<(.*)>", links[idx]).captures[1]
            end
        end
    end
    return rs, page_data
end

##################
# Canvas objects #
##################
include("canvas_objects.jl")

#####################
# Exposed endpoints #
#####################
include("endpoints/file_uploads.jl")
include("endpoints/assignments.jl")
include("endpoints/courses.jl")
include("endpoints/files.jl")
include("endpoints/submissions.jl")
include("endpoints/submission_comments.jl")
include("endpoints/users.jl")
include("endpoints/tabs.jl")
# include("endpoints/groups.jl")

# function download(f::File, path=tempdir(); kwargs...)
#     mkpath(path)
#     r = HTTP.download(f.url, joinpath(path, f.display_name); kwargs...)
#     return r
# end

# function announcements(c; params=nothing, kwargs...)
#     params = something(params, Dict("context_codes"=>"course_$(Internals.id(c))"))
#     json, page_data = paged_request("GET", "/api/v1/announcements"; params=params, kwargs...)
#     return DiscussionTopic.(json), page_data
# end
# function create_announcement(c::Course; kwargs...)
#     json = request("POST", "/api/v1$(Internals.pid(c))/discussion_topics"; kwargs...)
#     return DiscussionTopic(json)
# end

# function create_conversation(c; params::Dict, kwargs...)
#     if !haskey(params, "recipients[]")
#         params = Dict("recipients[]"=>["$(Internals.id(c))"], params...)
#     end
#     json = request("POST", "/api/v1/conversations"; params=params, kwargs...)
#     @assert length(json) == 1
#     return Conversation(json[1])
# end

# function conversations(; kwargs...)
#     json, page_data = paged_request("GET", "/api/v1/conversations"; kwargs...)
#     return Conversation.(json), page_data
# end

# function conversation(c; kwargs...)
#     json = request("GET", "/api/v1$(Internals.pid(c))"; kwargs...)
#     return Conversation(json)
# end

end # module


text:
# elementary_svg.jl

real2str(x::Real) = string(Float64(round(x,digits=5)))

function svg_dot(
    CENTER::Tuple,
    RADIUS::Real,
    FILLCOLOR::String,
    EDGECOLOR::String,
    EDGEWIDTH::Real,
    OPACITY::Real=1
    )::String
    return "<circle cx=\"$(real2str(CENTER[1]))\" cy=\"$(real2str(CENTER[2]))\" r=\"$(real2str(RADIUS))\" " *
           " stroke=\"$(EDGECOLOR)\" stroke-width=\"$(real2str(EDGEWIDTH))\"  stroke-opacity=\"$(OPACITY)\"  fill-opacity=\"$(OPACITY)\"  fill=\"$(FILLCOLOR)\" />"
end

function svg_dots(
    CENTS::Vector,
    RS::Vector,
    FILLCOLOR::String,
    EDGECOLOR::String,
    EDGEWIDTH::Real,
    OPACITY::Real=1
    )::String
    @assert eltype(CENTS) <: Tuple
    @assert length(CENTS)==length(RS)
    d = [   "M $(real2str(C[1]-R)),$(real2str(C[2])) a $(R),$(R) 0 1,0 $(R*2),0  a $(R),$(R) 0 1,0 $(-R*2),0 "
            for (C,R) ∈ zip(CENTS,RS)    ]
    return "<path d=\"$(join(d," "))\" " *
           " stroke=\"$(EDGECOLOR)\" stroke-width=\"$(real2str(EDGEWIDTH))\"  stroke-opacity=\"$(OPACITY)\" fill-opacity=\"$(OPACITY)\"  fill=\"$(FILLCOLOR)\" />"
end

function svg_dots(CENTS::Vector, R::Real, FILLCOLOR::String, EDGECOLOR::String, EDGEWIDTH::Real, OPACITY::Real=1)
    return svg_dots(CENTS, R.*ones(length(CENTS)), FILLCOLOR, EDGECOLOR, EDGEWIDTH, OPACITY)
end

svg_point(CENTER::Tuple, RADIUS::Real, COLOR::String, OPACITY::Real=1) = svg_dot(CENTER, RADIUS, COLOR, COLOR, 0, OPACITY)

svg_points(CENTS::Vector, RADIUS, COLOR::String, OPACITY::Real=1) = svg_dots(CENTS, RADIUS, COLOR, COLOR, 0, OPACITY)

svg_circle(CENTER::Tuple, RADIUS::Real, COLOR::String, OPACITY::Real=1) = svg_dot(CENTER, RADIUS, "none", COLOR, 0.382*RADIUS, OPACITY)

svg_circles(CENTS::Vector, RADIUS::Real, COLOR::String, OPACITY::Real=1) = svg_dots(CENTS, RADIUS, "none", COLOR, 0.382*RADIUS, OPACITY)

svg_circles(CENTS::Vector, RADIUS::Vector, COLOR::String, OPACITY::Real=1) = svg_dots(CENTS, RADIUS, "none", COLOR, (0.382/length(RADIUS))*sum(RADIUS), OPACITY)

function svg_cross_circles(
    CENTS::Vector,
    RADIUS::Vector,
    COLOR::String,
    OPACITY::Real=1
    )
    @assert length(CENTS)==length(RADIUS)
    d = [[  "M $(real2str(C[1]-R)),$(real2str(C[2])) a $(R),$(R) 0 1,0 $(R*2),0  a $(R),$(R) 0 1,0 $(-R*2),0 ",
            "M $(real2str(C[1]-sqrt(0.5)*R)),$(real2str(C[2]+sqrt(0.5)*R)) L $(real2str(C[1]+sqrt(0.5)*R)),$(real2str(C[2]-sqrt(0.5)*R))",
            "M $(real2str(C[1]+sqrt(0.5)*R)),$(real2str(C[2]+sqrt(0.5)*R)) L $(real2str(C[1]-sqrt(0.5)*R)),$(real2str(C[2]-sqrt(0.5)*R))" ]
            for (C,R) ∈ zip(CENTS,RADIUS) ]
    return "<path d=\"$(join(vcat(d...)," "))\" stroke=\"$(COLOR)\" stroke-width=\"$(real2str(0.382*R))\" stroke-opacity=\"$(OPACITY)\" fill-opacity=\"$(OPACITY)\"  fill=\"none\" />"
end

function svg_dot_circles(
    CENTS::Vector,
    RADIUS::Vector,
    COLOR::String,
    OPACITY::Real=1;
    λ=0.382
    )
    Ravg = sum(RADIUS)/length(RADIUS)
    d = [[  "M $(real2str(C[1]-R)),$(real2str(C[2])) a $(R),$(R) 0 1,0 $(R*2),0  a $(R),$(R) 0 1,0 $(-R*2),0 ",
            "M $(real2str(C[1]-λ*R)),$(real2str(C[2])) a $(λ*R),$(λ*R) 0 1,0 $(λ*R*2),0  a $(λ*R),$(λ*R) 0 1,0 $(-λ*R*2),0 " ]
            for (C,R) ∈ zip(CENTS,RADIUS)  ]
    return "<path d=\"$(join(vcat(d...)," "))\" stroke=\"$(COLOR)\" stroke-width=\"$(real2str(λ*Ravg))\" stroke-opacity=\"$(OPACITY)\"  fill-opacity=\"$(OPACITY)\"  fill=\"none\" />"
end

function svg_lines(
    lsit_of_point_pairs::Vector{T},
    COLOR::String,
    LINEWIDTH::Real;
    dashed=false,
    dotted=false,
    dasharray=nothing
    )::String where {T<:Tuple}
    d = []
    for (p1,p2) ∈ lsit_of_point_pairs
        push!(d, "M $(real2str(p1[1])),$(real2str(p1[2])) L $(real2str(p2[1])),$(real2str(p2[2]))")
    end
    DA = (dasharray===nothing) ? (dashed ? (dotted ? " 8 2 2 2" : " 8") : (dotted ? " 2" : "")) : dasharray
    if dashed || dotted
        return "<path d=\"$(join(d," "))\"  style=\"stroke:$(COLOR);stroke-width:$(real2str(LINEWIDTH));stroke-dasharray:$(DA)\" />"
    else
        return "<path d=\"$(join(d," "))\"  style=\"stroke:$(COLOR);stroke-width:$(real2str(LINEWIDTH))\" />"
    end
end

function svg_line(
    P1::Tuple,
    P2::Tuple,
    COLOR::String,
    LINEWIDTH::Real;
    dashed=false,
    dotted=false,
    dasharray=nothing
    )::String
    svg_lines([(P1,P2),], COLOR, LINEWIDTH; dashed=dashed, dotted=dotted, dasharray=dasharray)
end

# TODO arrow

function svg_arrowheads(
    lsit_of_end_point_directions::Vector{T},
    COLOR::String,
    LINEWIDTH::Real;
    dashed=false,
    dotted=false,
    dasharray=nothing
    )::String where { T <: Tuple }
    rot45p = [[cos(π/4), sin(π/4)] [-sin(π/4), cos(π/4)]]
    rot45m = [[cos(π/4),-sin(π/4)] [ sin(π/4), cos(π/4)]]
    lp1 = [ (pt, pt.-sz.*(rot45p*dir)) for (pt,dir,sz) ∈ lsit_of_end_point_directions]
    lp2 = [ (pt, pt.-sz.*(rot45m*dir)) for (pt,dir,sz) ∈ lsit_of_end_point_directions]
    stroke1 = svg_lines( lp1, COLOR, LINEWIDTH;
                         dashed=dashed, dotted=dotted, dasharray=dasharray )
    stroke2 = svg_lines( lp2, COLOR, LINEWIDTH;
                         dashed=dashed, dotted=dotted, dasharray=dasharray )
    return ( stroke1 * "\n" * stroke2 )
end

function svg_arrows(
    lsit_of_point_pairs::Vector{T},
    COLOR::String,
    LINEWIDTH::Real;
    headsize=0.1,
    dashed=false,
    dotted=false,
    dasharray=nothing
    )::String where {T<:Tuple}
    t2v(x) = vcat(x...)
    lsit_of_end_point_directions = [ (t2v(p2), normalize(t2v(p2).-t2v(p1)), headsize)
                                        for (p1,p2) ∈ lsit_of_point_pairs ]
    arrow_bodys = svg_lines( lsit_of_point_pairs, COLOR, LINEWIDTH;
                             dashed=dashed, dotted=dotted, dasharray=dasharray )
    arrow_heads = svg_arrowheads( lsit_of_end_point_directions, COLOR, LINEWIDTH;
                                  dashed=dashed, dotted=dotted, dasharray=dasharray )
    return arrow_bodys #( arrow_bodys * "\n" * arrow_heads )
end

max_stars_repo_name: algorithmx/PlotSVG

max_issues_repo_name: algorithmx/PlotSVG

max_forks_repo_name: algorithmx/PlotSVG


text:
#############################################################################
# max.jl
# Return the maximum of the two arguments. Operates elementwise over arrays.
# All expressions and atoms are subtpyes of AbstractExpr.
# Please read expressions.jl first.
#############################################################################
import Base.max
export max, pos, hinge_loss

# TODO: This can easily be extended to work
### Max Atom
struct MaxAtom <: AbstractExpr
    head::Symbol
    id_hash::UInt64
    children::Tuple{AbstractExpr, AbstractExpr}
    size::Tuple{Int, Int}

    function MaxAtom(x::AbstractExpr, y::AbstractExpr)
        if sign(x) == ComplexSign() || sign(y) == ComplexSign()
            error("Both the arguments should be real instead they are $(sign(x)) and $(sign(y))")
        else
            if x.size == y.size
                sz = x.size
            elseif x.size == (1, 1)
                sz = y.size
            elseif y.size == (1, 1)
                sz = x.size
            else
                error("Got different sizes for x as $(x.size) and y as $(y.size)")
            end
        end

        children = (x, y)
        return new(:max, hash(children), children, sz)
    end
end

function sign(x::MaxAtom)
    sign_one = sign(x.children[1])
    sign_two = sign(x.children[2])
    if sign_one == Positive() || sign_two == Positive()
        return Positive()
    elseif sign_one == Negative() && sign_two == Negative()
        return Negative()
    else
        return sign_one + sign_two
    end
end

# The monotonicity
function monotonicity(x::MaxAtom)
    return (Nondecreasing(), Nondecreasing())
end

# If we have h(x) = f o g(x), the chain rule says h''(x) = g'(x)^T f''(g(x))g'(x) + f'(g(x))g''(x);
# this represents the first term
function curvature(x::MaxAtom)
    return ConvexVexity()
end

function evaluate(x::MaxAtom)
    return max.(evaluate(x.children[1]), evaluate(x.children[2]))
end

# x <= this and y <= this if max(x, y) = this
function conic_form!(x::MaxAtom, unique_conic_forms::UniqueConicForms=UniqueConicForms())
    if !has_conic_form(unique_conic_forms, x)
        this = Variable(x.size[1], x.size[2])
        objective = conic_form!(this, unique_conic_forms)
        for child in x.children
            conic_form!(this >= child, unique_conic_forms)
        end
        cache_conic_form!(unique_conic_forms, x, objective)
    end
    return get_conic_form(unique_conic_forms, x)
end

max(x::AbstractExpr, y::AbstractExpr) = MaxAtom(x, y)
max(x::AbstractExpr, y::Value) = max(x, Constant(y))
max(x::Value, y::AbstractExpr) = max(Constant(x), y)
pos(x::AbstractExpr) = max(x, Constant(0, Positive()))
hinge_loss(x::AbstractExpr) = pos(1 - x)


text:
const PCRE2_VERSION = "10.31"

using BinaryBuilder

# Collection of sources required to build pcre2
sources = [
    "https://ftp.pcre.org/pub/pcre/pcre2-$(PCRE2_VERSION).tar.gz" =>
    "e11ebd99dd23a7bccc9127d95d9978101b5f3cf0a6e7d25a1b1ca165a97166c4",
]

# Bash recipe for building across all platforms
script = """
cd \$WORKSPACE/srcdir
cd pcre2-$(PCRE2_VERSION)
./configure --prefix=\$prefix --host=\$target --enable-jit --enable-pcre2-16 --enable-pcre2-32
make
make install
"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
    Windows(:i686),
    Windows(:x86_64),
    Linux(:i686, :glibc),
    Linux(:x86_64, :glibc),
    Linux(:aarch64, :glibc),
    Linux(:armv7l, :glibc),
    Linux(:powerpc64le, :glibc),
    MacOS(:x86_64)
]

# The products that we will ensure are always built
products(prefix) = [
   LibraryProduct(prefix, ["libpcre2-8"],  :libpcre2_8)
   LibraryProduct(prefix, ["libpcre2-16"], :libpcre2_16)
   LibraryProduct(prefix, ["libpcre2-32"], :libpcre2_32)
]

# Dependencies that must be installed before this package can be built
dependencies = [

]

# Build the tarballs, and possibly a `build.jl` as well.
build_tarballs(ARGS, "PCRE2Builder", sources, script, platforms, products, dependencies)


text:
using PSCOPF

using Test
using Dates

@testset verbose=true "test_tso_actions" begin

    ts = Dates.DateTime("2015-01-01T14:00:00")
    ts2 = Dates.DateTime("2015-01-01T14:30:00")

    @testset "empty" begin
        tso_actions = PSCOPF.TSOActions()

        @test isempty( PSCOPF.get_limitations(tso_actions) )
        @test ismissing( PSCOPF.get_limitation(tso_actions, "lim_1", ts) )

        @test isempty( PSCOPF.get_impositions(tso_actions) )
        @test ismissing( PSCOPF.get_imposition(tso_actions, "imp", ts, "S1") )
    end

    @testset "definitive_value" begin
        tso_actions = PSCOPF.TSOActions()
        PSCOPF.set_limitation_value!(tso_actions, "lim_1", ts, 300.)
        PSCOPF.set_limitation_value!(tso_actions, "lim_2", ts, 90.)
        PSCOPF.set_imposition_value!(tso_actions, "imp", ts, "S1", 2., 10.)
        PSCOPF.set_imposition_value!(tso_actions, "imp", ts2, "S1", 55., 55.)

        @test length( PSCOPF.get_limitations(tso_actions) ) == 2
        @test PSCOPF.get_limitation(tso_actions, "lim_1", ts) == 300.
        @test PSCOPF.get_limitation(tso_actions, "lim_2", ts) == 90.

        @test length( PSCOPF.get_impositions(tso_actions) ) == 2
        @test PSCOPF.get_imposition(tso_actions, "imp", ts, "S1") == (2., 10.)
        @test PSCOPF.get_imposition(tso_actions, "imp", ts2, "S1") == (55., 55.)

        @test_throws ErrorException PSCOPF.get_imposition_level(tso_actions, "imp", ts, "S1") #cause imposition[1] !=  imposition[2] => bounds not a level
        @test PSCOPF.get_imposition_level(tso_actions, "imp", ts2, "S1") == 55.
    end

end

max_stars_repo_name: klorel/pscopf

max_issues_repo_name: klorel/pscopf

max_forks_repo_name: klorel/pscopf


text:
using Random

@testset "Testing Random Games Generating" begin

    @testset "test random game" begin
        nums_actions = (2, 3, 4)
        g = random_game(nums_actions)

        @test g.nums_actions == nums_actions
    end

    @testset "test covariance game" begin
        nums_actions = (2, 3, 4)
        N = length(nums_actions)

        rho = 0.5
        g = covariance_game(nums_actions, rho)
        @test g.nums_actions == nums_actions

        rho = 1
        g = covariance_game(nums_actions, rho)
        for a in CartesianIndices(nums_actions)
            payoff_profile = g[a]
            for i in 1:(N-1)
                @test payoff_profile[i] ≈ payoff_profile[end]
            end
        end

        rho = -1/(N-1)
        g = covariance_game(nums_actions, rho)
        for a in CartesianIndices(nums_actions)
            payoff_profile = g[a]
            @test sum(payoff_profile) ≈ 0 atol=1e-10
        end

    end

    @testset "test random game value error" begin
        nums_actions = ()

        @test_throws ArgumentError random_game(nums_actions)

    end

    @testset "test covariance game value error" begin
        nums_actions = () #empty

        @test_throws ArgumentError covariance_game(nums_actions, 0.5)

        nums_actions = (2,) #length one

        @test_throws ArgumentError covariance_game(nums_actions, 0.5)

        nums_actions = (2, 3, 4)
        rho = 1.1 # > 1

        @test_throws ArgumentError covariance_game(nums_actions, rho)

        rho = -1. # < -1/(N-1)

        @test_throws ArgumentError covariance_game(nums_actions, rho)

    end

    @testset "random_pure_actions" begin
        nums_actions = (2, 3, 4)
        seed = 1234
        action_profiles =
            [random_pure_actions(MersenneTwister(seed), nums_actions)
             for i in 1:2]
        @test action_profiles[1] <= nums_actions
        @test action_profiles[2] == action_profiles[1]
    end

    @testset "random_mixed_actions" begin
        nums_actions = (2, 3, 4)
        seed = 1234
        action_profiles =
            [random_mixed_actions(MersenneTwister(seed), nums_actions)
             for i in 1:2]
        @test length.(action_profiles[1]) == nums_actions
        for i in 1:length(nums_actions)
            @test action_profiles[2][i] == action_profiles[1][i]
        end
    end

end


max_stars_repo_name: tkelman/docker-power-suitesparse

max_issues_repo_name: tkelman/docker-power-suitesparse

max_forks_repo_name: tkelman/docker-power-suitesparse


text:
module LuxorDiagrams

using Librsvg_jll
using Luxor
using MathTeXEngine
using LaTeXStrings
using Parameters

end


text:
#=

# 203: Various coordinate systems
([source code](SOURCE_URL))
=#

module Example203_CoordinateSystems

using VoronoiFVM
using LinearAlgebra
using ExtendableGrids
using GridVisualize

function main(;nref=0,r1=0.0, r2=5.0, dim=2,Plotter=nothing)
    h=0.1*2.0^(-nref)
    R=collect(r1:h:r2)
    Z=collect(0:h:2)
    grid=VoronoiFVM.Grid(R)
    if dim==2
        grid=VoronoiFVM.Grid(R,Z)
    end
    circular_symmetric!(grid)

    function symlapcyl(r)
        if r1≈0.0
            return 0.25*(r2*r2-r*r);
        else
            return (log(r)-log(r2))/(log(r1)-log(r2));
        end
    end

    function flux!(f,u0,edge)
        u=unknowns(edge,u0)
        f[1]=u[1,1]-u[1,2]
    end

    function source!(f,node)
        if r1≈0.0
            f[1]=1.0
        else
            f[1]=0.0
        end
    end

    # Create a physics structure
    physics=VoronoiFVM.Physics(num_species=1,flux=flux!,source=source!)
    sys=VoronoiFVM.System(grid,physics,unknown_storage=:dense)
    ispec=1
    enable_species!(sys,ispec,[1])
    ileft=1
    if dim==2
        ileft=4
    end
    if !(r1≈0.0)
        boundary_dirichlet!(sys,ispec,ileft,1.0)
    end
    boundary_dirichlet!(sys,ispec,2,0.0)

    inival=unknowns(sys)
    solution=unknowns(sys)
    inival.=0
    solution.=0

    # Solve stationary problem
    solve!(solution,inival,sys)

    scalarplot(grid,solution[1,:],Plotter=Plotter)

    exact=symlapcyl.(coordinates(grid)[1,:])
    err=norm(solution[1,:]-exact,Inf)
end

#
# Called by unit test
#
function test()
    main() <1.0e-14
end

end


text:
using Test
using ClimaCoreVTK
using IntervalSets
import ClimaCore:
    Geometry, Domains, Meshes, Topologies, Spaces, Fields, Operators

OUTPUT_DIR = mkpath(get(ENV, "CI_OUTPUT_DIR", tempname()))
mkpath(joinpath(OUTPUT_DIR, "series"))

@testset "sphere" begin
    R = 6.37122e6

    domain = Domains.SphereDomain(R)
    mesh = Meshes.EquiangularCubedSphere(domain, 4)
    grid_topology = Topologies.Topology2D(mesh)
    quad = Spaces.Quadratures.GLL{5}()
    space = Spaces.SpectralElementSpace2D(grid_topology, quad)
    coords = Fields.coordinate_field(space)

    u = map(coords) do coord
        u0 = 20.0
        α0 = 45.0
        ϕ = coord.lat
        λ = coord.long

        uu = u0 * (cosd(α0) * cosd(ϕ) + sind(α0) * cosd(λ) * sind(ϕ))
        uv = -u0 * sind(α0) * sind(λ)
        Geometry.UVVector(uu, uv)
    end

    writevtk(
        joinpath(OUTPUT_DIR, "sphere"),
        (coords = coords, u = u);
        basis = :point,
    )
    @test isfile(joinpath(OUTPUT_DIR, "sphere.vtu"))

    times = 0:10:350
    A = [
        map(coords) do coord
            cosd(coord.lat) *
            (sind(coord.long) * sind(α) + cosd(coord.long) * cosd(α))
        end for α in times
    ]
    writevtk(
        joinpath(OUTPUT_DIR, "series", "sphere_scalar_series"),
        times,
        (A = A, B = A),
    )
    for (i, _) in enumerate(times)
        istr = lpad(i, 3, "0")
        @test isfile(
            joinpath(OUTPUT_DIR, "series", "sphere_scalar_series_$(istr).vtu"),
        )
    end
    @test isfile(joinpath(OUTPUT_DIR, "series", "sphere_scalar_series.pvd"))

    U = Array{Fields.Field}(undef, length(times))
    for t in 1:(div(350, 10) + 1)
        u = map(coords) do coord
            u0 = 20.0
            α0 = 45.0
            ϕ = coord.lat
            λ = coord.long

            uu = u0 * (cosd(α0) * cosd(ϕ) + sind(α0) * cosd(λ) * sind(ϕ))
            uv = -u0 * sind(α0) * sind(λ)
            Geometry.UVVector(uu, uv)
        end
        U[t] = u
    end
    writevtk(
        joinpath(OUTPUT_DIR, "series", "sphere_vector_series"),
        times,
        (U = U,),
    )
    for (i, _) in enumerate(times)
        istr = lpad(i, 3, "0")
        @test isfile(
            joinpath(OUTPUT_DIR, "series", "sphere_vector_series_$(istr).vtu"),
        )
    end
    @test isfile(joinpath(OUTPUT_DIR, "series", "sphere_vector_series.pvd"))
end

@testset "rectangle" begin

    domain = Domains.RectangleDomain(
        Geometry.XPoint(0) .. Geometry.XPoint(2π),
        Geometry.YPoint(0) .. Geometry.YPoint(2π),
        x1periodic = true,
        x2periodic = true,
    )

    n1, n2 = 4, 4
    Nq = 4
    mesh = Meshes.RectilinearMesh(domain, n1, n2)
    grid_topology = Topologies.Topology2D(mesh)
    quad = Spaces.Quadratures.GLL{Nq}()
    space = Spaces.SpectralElementSpace2D(grid_topology, quad)

    coords = Fields.coordinate_field(space)
    sinxy = map(coords) do coord
        sin(coord.x + coord.y)
    end
    u = map(coords) do coord
        Geometry.UVVector(-coord.y, coord.x)
    end

    writevtk(joinpath(OUTPUT_DIR, "plane"), (sinxy = sinxy, u = u))
    @test isfile(joinpath(OUTPUT_DIR, "plane.vtu"))

end

@testset "hybrid 2d" begin
    hdomain = Domains.IntervalDomain(
        Geometry.XPoint(0) .. Geometry.XPoint(10.0),
        periodic = true,
    )
    hmesh = Meshes.IntervalMesh(hdomain, nelems = 10)
    htopology = Topologies.IntervalTopology(hmesh)
    quad = Spaces.Quadratures.GLL{4}()
    hspace = Spaces.SpectralElementSpace1D(htopology, quad)

    vdomain = Domains.IntervalDomain(
        Geometry.ZPoint(0) .. Geometry.ZPoint(20.0),
        boundary_names = (:bottom, :top),
    )
    vmesh = Meshes.IntervalMesh(vdomain, nelems = 20)
    vtopology = Topologies.IntervalTopology(vmesh)
    vspace = Spaces.FaceFiniteDifferenceSpace(vtopology)

    fspace = Spaces.ExtrudedFiniteDifferenceSpace(hspace, vspace)
    cspace = Spaces.CenterExtrudedFiniteDifferenceSpace(fspace)
    writevtk(
        joinpath(OUTPUT_DIR, "hybrid2d_point"),
        Fields.coordinate_field(fspace);
        basis = :point,
    )
    @test isfile(joinpath(OUTPUT_DIR, "hybrid2d_point.vtu"))
    writevtk(
        joinpath(OUTPUT_DIR, "hybrid2d_cell"),
        Fields.coordinate_field(cspace);
        basis = :cell,
    )
    @test isfile(joinpath(OUTPUT_DIR, "hybrid2d_cell.vtu"))
end

@testset "hybrid 3d" begin

    hdomain = Domains.RectangleDomain(
        Geometry.XPoint(0) .. Geometry.XPoint(2π),
        Geometry.YPoint(0) .. Geometry.YPoint(2π),
        x1periodic = true,
        x2periodic = true,
    )

    hmesh = Meshes.RectilinearMesh(hdomain, 4, 4)
    htopology = Topologies.Topology2D(hmesh)
    quad = Spaces.Quadratures.GLL{4}()
    hspace = Spaces.SpectralElementSpace2D(htopology, quad)

    vdomain = Domains.IntervalDomain(
        Geometry.ZPoint(0) .. Geometry.ZPoint(20.0),
        boundary_names = (:bottom, :top),
    )
    vmesh = Meshes.IntervalMesh(vdomain, nelems = 20)
    vtopology = Topologies.IntervalTopology(vmesh)
    vspace = Spaces.FaceFiniteDifferenceSpace(vtopology)

    fspace = Spaces.ExtrudedFiniteDifferenceSpace(hspace, vspace)
    cspace = Spaces.CenterExtrudedFiniteDifferenceSpace(fspace)
    writevtk(
        joinpath(OUTPUT_DIR, "hybrid3d_point"),
        Fields.coordinate_field(fspace);
        basis = :point,
    )
    @test isfile(joinpath(OUTPUT_DIR, "hybrid3d_point.vtu"))
    writevtk(
        joinpath(OUTPUT_DIR, "hybrid3d_cell"),
        Fields.coordinate_field(cspace);
        basis = :cell,
    )
    @test isfile(joinpath(OUTPUT_DIR, "hybrid3d_cell.vtu"))
end

@testset "hybrid 3d sphere" begin
    R = 100.0

    hdomain = Domains.SphereDomain(R)
    hmesh = Meshes.EquiangularCubedSphere(hdomain, 4)
    htopology = Topologies.Topology2D(hmesh)
    quad = Spaces.Quadratures.GLL{5}()
    hspace = Spaces.SpectralElementSpace2D(htopology, quad)

    vdomain = Domains.IntervalDomain(
        Geometry.ZPoint(0),
        Geometry.ZPoint(20.0),
        boundary_names = (:bottom, :top),
    )
    vmesh = Meshes.IntervalMesh(vdomain, nelems = 20)
    vtopology = Topologies.IntervalTopology(vmesh)
    vspace = Spaces.FaceFiniteDifferenceSpace(vtopology)

    fspace = Spaces.ExtrudedFiniteDifferenceSpace(hspace, vspace)
    cspace = Spaces.CenterExtrudedFiniteDifferenceSpace(fspace)
    writevtk(
        joinpath(OUTPUT_DIR, "hybrid_sphere_point"),
        Fields.coordinate_field(fspace);
        basis = :point,
    )
end


text:
struct Action
    id::Integer
    status::String
    actiontype::String
    started_at::DateTime
    completed_at::Union{Nothing, DateTime}
    resourceid::Union{Nothing, Integer}
    resourcetype::String
    region::Union{Nothing, Region}
    regionslug::Union{Nothing, String}

    function Action(data::Dict{String})
        data["started_at"] = DateTime(data["started_at"][1:end-1])
        if data["completed_at"] != nothing
            data["completed_at"] = DateTime(data["completed_at"][1:end-1])
        end
        if data["region"] != nothing
            data["region"] = Region(data["region"])
        end

        new(
            data["id"],
            data["status"],
            data["type"],
            data["started_at"],
            data["completed_at"],
            data["resource_id"],
            data["resource_type"],
            data["region"],
            data["region_slug"]
        )
    end
end

function show(io::IO, a::Action)
    print(io, "Action ($(a.actiontype), $(a.started_at))")
end

# List all actions
function getallactions!(client::AbstractClient)
    uri = joinpath(ENDPOINT, "actions?per_page=$MAXOBJECTS")
    getalldata!(client, uri, Action)
end

# Retrieve an existing action
function getaction!(client::AbstractClient, action_id::Integer)
    uri = joinpath(ENDPOINT, "actions", "$(action_id)")
    Action(getdata!(client, uri))
end

function getaction!(client::AbstractClient, action::Action)
    getaction!(client, action.name)
end


text:
using Test
using DFTK
import DFTK: solve_ΩplusK, apply_Ω, apply_K
import DFTK: filled_occupation, compute_projected_gradient

include("testcases.jl")

@testset "Newton" begin
    Ecut = 3
    fft_size = [9, 9, 9]
    Si = ElementPsp(silicon.atnum, psp=load_psp(silicon.psp))
    model = model_DFT(silicon.lattice, [Si => silicon.positions], [:lda_xc_teter93])
    basis = PlaneWaveBasis(model, Ecut, silicon.kcoords, silicon.ksymops; fft_size=fft_size)
    scfres_start = self_consistent_field(basis, maxiter=1)

    ψ = DFTK.select_occupied_orbitals(basis, scfres_start.ψ)

    occupation = scfres_start.occupation
    filled_occ = filled_occupation(model)
    n_spin = model.n_spin_components
    n_bands = div(model.n_electrons, n_spin * filled_occ, RoundUp)
    @assert n_bands == size(ψ[1], 2)
    # number of kpoints and occupation
    Nk = length(basis.kpoints)
    occupation = [filled_occ * ones(n_bands) for ik = 1:Nk]

    ρ = compute_density(basis, ψ, occupation)

    rhs = compute_projected_gradient(basis, ψ, occupation)
    ϕ = rhs + ψ

    @testset "self-adjointness of solve_ΩplusK" begin
        @test isapprox(
            real(dot(ϕ, solve_ΩplusK(basis, ψ, rhs, occupation))),
            real(dot(solve_ΩplusK(basis, ψ, ϕ, occupation), rhs)),
            atol=1e-7
        )
    end

    @testset "self-adjointness of apply_Ω" begin
        _, H = energy_hamiltonian(basis, ψ, occupation; ρ=ρ)
        # Rayleigh-coefficients
        Λ = [ψk'Hψk for (ψk, Hψk) in zip(ψ, H * ψ)]

        # Ω is complex-linear and so self-adjoint as a complex operator.
        @test isapprox(
            dot(ϕ, apply_Ω(rhs, ψ, H, Λ)),
            dot(apply_Ω(ϕ, ψ, H, Λ), rhs),
            atol=1e-7
        )
    end

    @testset "self-adjointness of apply_K" begin
        # K involves conjugates and is only a real-linear operator,
        # hence we test using the real dot product.
        @test isapprox(
            real(dot(ϕ, apply_K(basis, rhs, ψ, ρ, occupation))),
            real(dot(apply_K(basis, ϕ, ψ, ρ, occupation), rhs)),
            atol=1e-7
        )
    end
end


text:
# gof.jl

export lossL1, lossMissedClass, lossSqErr, meanForecLoss;

function lossL1(x::Vector{Float64}, y::Vector{Float64})
    @assert length(x)==length(y)
    sum( abs.( x .- y ) )
end

function lossMissedClass(x::Int, y::Vector{Float64})
    Int( x != findmax(y)[2] )
end

function lossSqErr(x::Float64, y::Vector{Float64})
    ( x - collect(Float64, 1:length(y))'y )^2
end

function meanForecLoss(S::Vector{Int}, tprime::Vector{Int},
    lossFn::Function, sims::Vector{Dict},
    TT::Int, L::Int, K::Int,
    simind::Array{Dict{Symbol,Any},1}; λ_indx::Union{Nothing, λindxMMTD}=nothing,
    modeltype::String)

    nsim = length(simind)
    nprime = length(tprime)
    lossMat = zeros(Float64, nsim, nprime)
    missClassMat = zeros(Int, nsim, nprime)
    negllMat = zeros(Float64, nsim, nprime)
    sqErrMat = zeros(Float64, nsim, nprime)

    forecMat = zeros(Float64, nsim, nprime, K)

    for i in 1:nsim
        ii = deepcopy( simind[i] )
        if modeltype == "MMTD"
            R = length(sims[1][:lλ])
            λ_now = [ exp.(sims[ii][:lλ][r]) for r in 1:R ]
            Q_now = [ exp.( sims[ii][:lQ][r] ) for r in 1:R ]
        end

        for j in 1:nprime
            tt = deepcopy( tprime[j] )
            S_now = deepcopy( S[tt] )
            Slagrev_now = deepcopy( S[range(tt-1, step=-1, length=L)] )

            if modeltype == "MTD"
                forec = forecDist(Slagrev_now, exp.(sims[ii][:lλ]),
                    exp.(sims[ii][:lQ]))
            elseif modeltype == "MTDg"
                forec = forecDist_MTDg(Slagrev_now, exp.(sims[ii][:lλ]),
                    exp.(sims[ii][:lQ0]), [ exp.(sims[ii][:lQ][ell]) for ell = 1:L ] )
            elseif modeltype == "MMTD"
                forec = forecDist_MMTD(Slagrev_now, exp.(sims[ii][:lΛ]), λ_now,
                                       exp.(sims[ii][:lQ0]), Q_now, λ_indx)
            end

            St_vec = float([ (k == S_now) for k in 1:K ])

            lossMat[i,j] = lossFn(St_vec, forec)
            missClassMat[i,j] = lossMissedClass(S_now, forec)
            negllMat[i,j] = -log(forec[S_now])
            sqErrMat[i,j] = lossSqErr(Float64(S_now), forec)

            forecMat[i,j,:] = deepcopy(forec)
        end
    end
    (lossMat, missClassMat, negllMat, sqErrMat, forecMat)
end
function meanForecLoss(y::Vector{Int}, X::Matrix{Int}, P::Matrix{Float64},
    nprime::Int, lossFn::Function,
    sims::Array{Dict{Symbol,Any},1},
    TT::Int, L::Int, K::Int,
    simind::Vector{Int}; λ_indx::Union{Nothing, λindxMMTD}=nothing,
    modeltype::String)

    nsim = length(simind)
    lossMat = zeros(Float64, nsim, nprime)
    missClassMat = zeros(Int, nsim, nprime)
    negllMat = zeros(Float64, nsim, nprime)
    sqErrMat = zeros(Float64, nsim, nprime)

    PlossMat = zeros(Float64, nsim, nprime)

    forecMat = zeros(Float64, nsim, nprime, K)

    for i in 1:nsim

        ii = deepcopy( simind[i] )

        if modeltype == "MMTD"
            R = length(sims[1][:lλ])
            λ_now = [ exp.(sims[ii][:lλ][r]) for r in 1:R ]
            Q_now = [ exp.( sims[ii][:lQ][r] ) for r in 1:R ]
        end

        for tt in 1:nprime
            S_now = deepcopy( y[tt] )
            Slagrev_now = deepcopy( X[tt,:] )

            if modeltype == "MTD"
                forec = forecDist(Slagrev_now, exp.(sims[ii][:lλ]),
                exp.(sims[ii][:lQ]))
            elseif modeltype == "MTDg"
                forec = forecDist_MTDg(Slagrev_now, exp.(sims[ii][:lλ]),
                exp.(sims[ii][:lQ0]), [ exp.(sims[ii][:lQ][ell]) for ell = 1:L ] )
            elseif modeltype == "MMTD"
                forec = forecDist_MMTD(Slagrev_now, exp.(sims[ii][:lΛ]), λ_now,
                exp.(sims[ii][:lQ0]), Q_now, λ_indx)
            end

            St_vec = float([ (k == S_now) for k in 1:K ])

            lossMat[i,tt] = lossFn(St_vec, forec)
            missClassMat[i,tt] = lossMissedClass(S_now, forec)
            negllMat[i,tt] = -log(forec[S_now])
            sqErrMat[i,tt] = lossSqErr(Float64(S_now), forec)

            PlossMat[i,tt] = lossFn(P[tt,:], forec) / float(K)

            forecMat[i,tt,:] = deepcopy(forec)
        end
    end
    (lossMat, missClassMat, negllMat, sqErrMat, PlossMat, forecMat)
end

max_stars_repo_name: mheiner/MTD

max_issues_repo_name: mheiner/MTD

max_forks_repo_name: mheiner/MTD


text:
using Flase

v0 = 5.0
Dϕ = 4.0
world = World(
    v0 = 1.,
    n_dogs = 100,
    boxsize = 250.0,
    freedom_rate = 10.0,
    business_rate = 10.0,
    meanSleepyness = 10.0,
    motion = BrownianMotion(
        noise = v0^4 / Dϕ,
        friction = Dϕ / v0^2
        ),
    sheeps = DenseSheeps(
        n_sheeps = 200,
        gridsize = 250,
        )
    )
simulation = FiniteSimulation(;
    dt = 0.1,
    end_time = 1.0,
    world = world,
    plotter = UnicodePlotter()
    )
    # TODO: get this smooth
# simulation = FiniteSimulation(;
#     dt = 0.1,
#     end_time = 20_000.0,
#     world = world,
#     plotter = UnicodePlotter( skip = 2_000 )
#     )
Juno.@profiler Flase.runsim( simulation )


text:
using Test, SeisAcoustic, LinearAlgebra, DSP

include("forward_modelling.jl")
include("adjoint_operator.jl")
include("born_approximation.jl")


text:
"""
    resetERA5Variables( allfiles :: Bool ) -> nothing

Reset the list of Single-Level and PressureCustom variables.

Arguments
=========

- `allfiles` : If false, only get rid of all the SingleCustom variables, but if true, then the SingleVariable list will be reset back to the default for ERA5Reanalysis.jl
"""
function resetERA5Variables(;allfiles=false)

    if allfiles

        @info "$(modulelog()) - Resetting both the master and custom lists of ERA5 variables back to the default"
        flist = [
            "singlevariable.txt","singlecustom.txt",
            "pressurevariable.txt","pressurecustom.txt"
        ]
    else
        @info "$(modulelog()) - Resetting the custom lists of ERA5 variables back to the default"
        flist = ["singlevariable.txt","singlecustom.txt","pressurecustom.txt"]
    end

    for fname in flist
        copyera5variables(fname,overwrite=true)
    end

    return

end

function addERA5Variables(fname::AbstractString)

    @info "$(modulelog()) - Importing user-defined GeoRegions from the file $fname directly into the custom lists"

    vtype  = evarfiletype(fname)
    varray = listera5variables(fname)
    vlist  = varray[:,1]; nvar = length(vlist)
    if vtype == "SingleLevel"
        for ivar in 1 : nvar
            if isSingle(vlist[ivar],throw=false)
                @warn "$(modulelog()) - The SingleVariable ID \"$varID\" is already in use and thus cannot be added to singlevariable.txt, please use another ID"
            else
                SingleVariable(
                    varID = varray[ivar,1], lname = varray[ivar,2],
                    vname = varray[ivar,3], units = varray[ivar,4],
                    iscustom = false
                )
            end
        end
    elseif vtype == "SingleCustom"
        for ivar in 1 : nvar
            if isSingle(vlist[ivar],throw=false)
                @warn "$(modulelog()) - The SingleVariable ID \"$varID\" is already in use and thus cannot be added to singlecustom.txt, please use another ID"
            else
                SingleVariable(
                    varID = varray[ivar,1], lname = varray[ivar,2],
                    vname = varray[ivar,3], units = varray[ivar,4]
                )
            end
        end
    elseif vtype == "PressureCustom"
        for ivar in 1 : nvar
            if isPressure(vlist[ivar],throw=false)
                @warn "$(modulelog()) - The PressureVariable ID \"$varID\" is already in use and thus cannot be added to pressurecustom.txt, please use another ID"
            else
                PressureVariable(
                    varID = varray[ivar,1], lname = varray[ivar,2],
                    vname = varray[ivar,3], units = varray[ivar,4],
                )
            end
        end
    elseif vtype == "PressureLevel"
        error("$(modulelog()) - All the downloadable pressure variables from the ERA5 Climate Data Store have been listed in ERA5Reanalysis.jl")
    else
        error("$(modulelog()) - Unable to identify ERA5 Variable type, please go back and check the file header again")
    end

    return nothing

end

function rmERA5Variable(
    evar :: ERA5Variable
)

    if typeof(evar) <: PressureVariable
        error("$(modulelog()) - Variables of the type PressureVariable cannot be removed")
    end

    if typeof(evar) <: SingleVariable
        fid = joinpath(DEPOT_PATH[1],"files","ERA5Reanalysis","singlevariable.txt")
    elseif typeof(evar) <: SingleCustom
        fid = joinpath(DEPOT_PATH[1],"files","ERA5Reanalysis","singlecustom.txt")
    elseif typeof(evar) <: PressureCustom
        fid = joinpath(DEPOT_PATH[1],"files","ERA5Reanalysis","pressurecustom.txt")
    end

    flines = readlines(fid)
    nlines = length(flines)

    open("tmp.txt","w") do io
        for iline = 1 : nlines
            if !occursin("$(evar.varID),",flines[iline])
                write(io,"$(flines[iline])\n")
            end
        end
    end

    mv("tmp.txt",fid,force=true)

    return nothing

end

function tableERA5Variables()

    jfol = joinpath(DEPOT_PATH[1],"files","ERA5Reanalysis"); mkpath(jfol);
    fvar = ["SingleVariable","SingleCustom","PressureVariable","PressureCustom"]
    fmat = []

    for fname in fvar
        fid  = joinpath(jfol,"$(lowercase(fname)).txt")
        try
            vmat = readdlm(fid,',',comments=true,comment_char='#')
            nvar = size(vmat,1); ff = fill(fname,nvar)
            vmat = cat(ff,vmat[:,[1,3,4,2]],dims=2)
            fmat = cat(fmat,vmat,dims=1)
        catch
        end
    end

    head = ["Variable Type","ID","Name","Units","ERA5 Long-Name"];

    pretty_table(
        fmat,head,
        alignment=[:c,:c,:l,:c,:l],
        crop = :none, tf = tf_compact
    );

end

## Backend Functions

function copyera5variables(
    fname::AbstractString;
    overwrite::Bool=false
)

    jfol = joinpath(DEPOT_PATH[1],"files","ERA5Reanalysis"); mkpath(jfol);
    ftem = joinpath(@__DIR__,"..","..","extra",fname)
    fvar = joinpath(jfol,fname)

    if !overwrite
        if !isfile(fvar)

            @debug "$(modulelog()) - Unable to find $fvar, copying data from $ftem ..."

            open(fvar,"w") do io
                open(ftem) do f
                    for line in readlines(f)
                        write(io,"$line\n")
                    end
                end
            end

        end
    else

        if isfile(fvar)
            @warn "$(modulelog()) - Overwriting $fvar with original file in $ftem ..."
            rm(fvar,force=true)
        end

        open(fvar,"w") do io
            open(ftem) do f
                for line in readlines(f)
                    write(io,"$line\n")
                end
            end
        end

    end

    return

end

function listera5variables(fname::AbstractString)

    try
        return readdlm(fname,',',comments=true,comment_char='#')[:,1]
    catch
        return []
    end

end

function evarfiletype(fname::AbstractString)

    vtype = readlines(fname)[1]
    vtype = replace(vtype,"#"=>"")
    vtype = replace(vtype," "=>"")

    return vtype

end


text:
using Pointy
using Documenter

DocMeta.setdocmeta!(Pointy, :DocTestSetup, :(using Pointy); recursive=true)

makedocs(;
    modules=[Pointy],
    authors="Gus Hart",
    repo="https://github.com/glwhart/Pointy.jl/blob/{commit}{path}#{line}",
    sitename="Pointy.jl",
    format=Documenter.HTML(;
        prettyurls=get(ENV, "CI", "false") == "true",
        canonical="https://glwhart.github.io/Pointy.jl",
        assets=String[],
    ),
    pages=[
        "Home" => "index.md",
    ],
)

deploydocs(;
    repo="github.com/glwhart/Pointy.jl",
)


text:
println("hi")
println("hi there poop")

max_stars_repo_licenses: MIT Unlicense

max_issues_repo_licenses: MIT Unlicense

max_forks_repo_licenses: MIT Unlicense


text:
# https://juliagraphics.github.io/Gtk.jl/latest/manual/layout/#Grid-1
using Gtk

# win = GtkWindow("A new window")
# g = GtkGrid()
# a = GtkEntry()  # a widget for entering text
# set_gtk_property!(a, :text, "This is Gtk!")
# b = GtkCheckButton("Check me!")
# c = GtkScale(false, 0:10)     # a slider

win = GtkWindow("A new window")
g = GtkGrid()
a = GtkEntry()  # a widget for entering text
set_gtk_property!(a, :text, "This is Gtk!")
b = GtkCheckButton("Check me!")
c = GtkScale(false, 0:10)

# # Now let's place these graphical elements into the Grid:
# g[1,1] = a    # Cartesian coordinates, g[x,y]
# g[2,1] = b
# g[1:2,2] = c  # spans both columns
# set_gtk_property!(g, :column_homogeneous, true)
# set_gtk_property!(g, :column_spacing, 15)  # introduce a 15-pixel gap between columns
# push!(win, g)
# showall(win)

g[1,1] = a
g[2,1] = b
g[1:2,2] = c
set_gtk_property!(g, :column_homogeneous, true)
set_gtk_property!(g, :column_spacing, 15)
push!(win,g)

showall(win)


text:
"""
    JSONWorkBook
Preserves XLSX WorkBook data structure
"""
mutable struct JSONWorkbook
    xlsxpath::AbstractString
    sheets::Vector{JSONWorksheet}
    sheetindex::Index
end

function JSONWorkbook(xf::XLSX.XLSXFile, v::Vector{JSONWorksheet})
    wsnames = sheetnames.(v)
    index = Index(wsnames)
    JSONWorkbook(xf.filepath, v, index)
end
# same kwargs for all sheets
function JSONWorkbook(xf::XLSX.XLSXFile, sheets = XLSX.sheetnames(xf); kwargs...)
    v = Array{JSONWorksheet, 1}(undef, length(sheets))
    @inbounds for (i, s) in enumerate(sheets)
        v[i] = JSONWorksheet(xf, s; kwargs...)
    end
    close(xf)

    JSONWorkbook(xf, v)
end
# Different Kwargs per sheet
function JSONWorkbook(xlsxpath::AbstractString, sheets, kwargs_per_sheet::Dict)
    xf = XLSX.readxlsx(xlsxpath)

    v = Array{JSONWorksheet, 1}(undef, length(sheets))
    @inbounds for (i, s) in enumerate(sheets)
        v[i] = JSONWorksheet(xf, s; kwargs_per_sheet[s]...)
    end
    close(xf)

    JSONWorkbook(xf, v)
end
function JSONWorkbook(xlsxpath, sheets; kwargs...)
    xf = XLSX.readxlsx(xlsxpath)
    JSONWorkbook(xf, sheets; kwargs...)
end
function JSONWorkbook(xlsxpath; kwargs...)
    xf = XLSX.readxlsx(xlsxpath)
    JSONWorkbook(xf; kwargs...)
end

# JSONWorkbook fallback functions
hassheet(jwb::JSONWorkbook, s::Symbol) = haskey(jwb.sheetindex, s)
sheetnames(jwb::JSONWorkbook) = names(jwb.sheetindex)
xlsxpath(jwb::JSONWorkbook) = jwb.xlsxpath

getsheet(jwb::JSONWorkbook, i) = jwb.sheets[i]
getsheet(jwb::JSONWorkbook, s::AbstractString) = getsheet(jwb, jwb.sheetindex[s])
getsheet(jwb::JSONWorkbook, s::Symbol) = getsheet(jwb, jwb.sheetindex[string(s)])
Base.getindex(jwb::JSONWorkbook, i::UnitRange) = getsheet(jwb, i)
Base.getindex(jwb::JSONWorkbook, i::Integer) = getsheet(jwb, i)
Base.getindex(jwb::JSONWorkbook, s::AbstractString) = getsheet(jwb, s)
Base.getindex(jwb::JSONWorkbook, s::Symbol) = getsheet(jwb, string(s))

Base.length(jwb::JSONWorkbook) = length(jwb.sheets)
Base.lastindex(jwb::JSONWorkbook) = length(jwb.sheets)

Base.setindex!(jwb::JSONWorkbook, jws::JSONWorksheet, i1::Int) = setindex!(jwb.sheets, jws, i1)
Base.setindex!(jwb::JSONWorkbook, jws::JSONWorksheet, s::Symbol) = setindex!(jwb, jws, string(s))
Base.setindex!(jwb::JSONWorkbook, jws::JSONWorksheet, s::AbstractString) = setindex!(jwb.sheets, jws, jwb.sheetindex[s])

function Base.deleteat!(jwb::JSONWorkbook, i::Integer)
    deleteat!(getfield(jwb, :sheets), i)
    s = sheetnames.(getfield(jwb, :sheets))
    setfield!(jwb, :sheetindex, Index(s))
    jwb
end
Base.deleteat!(jwb::JSONWorkbook, sheet::Symbol) = deleteat!(jwb, string(sheet))
function Base.deleteat!(jwb::JSONWorkbook, sheet::AbstractString)
    i = getfield(jwb, :sheetindex)[sheet]
    deleteat!(jwb, i)
end

Base.iterate(jwb::JSONWorkbook) = iterate(jwb, 1)
function Base.iterate(jwb::JSONWorkbook, st)
    st > length(jwb) && return nothing
    # TODO deprecate df
    return (jwb[st], st + 1)
end

## Display
function Base.summary(io::IO, jwb::JSONWorkbook)
    @printf(io, "JSONWorkbook(\"%s\") containing %i Worksheets\n",
                basename(xlsxpath(jwb)), length(jwb))
end
function Base.show(io::IO, jwb::JSONWorkbook)
    summary(io, jwb)
    @printf(io, "%6s %-15s\n", "index", "name")
    println(io, "-"^(6+1+15+1))

    index = sort(OrderedDict(jwb.sheetindex.lookup); byvalue = true)
    for el in index
        name = string(el[1])
        @printf(io, "%6s %-15s\n", el[2], string(el[1]))
    end
end


text: println("hello world")

max_stars_repo_name: kennethsequeira/Hello-world

max_issues_repo_name: kennethsequeira/Hello-world

max_forks_repo_name: kennethsequeira/Hello-world


text:
"""
Tableau for the stochastic symplectic Euler method
  Tableau for the stochastic symplectic Euler method
  Satisfies the conditions for Lagrange-d'Alembert integrators.
  Satisfies the conditions for strong convergence of order 1.0 for one Wiener process
  for special choices of the stochastic Hamiltonians and forces, e.g., h=h(q), f=0.
"""
function TableauStochasticSymplecticEuler()

    a_q = ones(Float64, 1, 1)
    b_q = [1.0]
    c_q = [1.0]

    a_p = zeros(Float64, 1, 1)
    b_p = [1.0]
    c_p = [0.0]

    TableauSIPRK(:StochasticSymplecticEuler, 1, a_q, b_q, c_q, 1, a_q, b_q, c_q, 1, a_p, b_p, c_p, 1, a_p, b_p, c_p)
end

"Tableau for the 2-stage stochastic LobattoIIA-IIB method (Stormer-Verlet)"
function TableauStochasticStoermerVerlet()

    TableauSIPRK(:StochasticStoermerVerlet,
                 CoefficientsLobattoIIIA(2), CoefficientsLobattoIIIA(2),
                 CoefficientsLobattoIIIB(2), CoefficientsLobattoIIIB(2))
end


text:
module PolygonInbounds

export inpoly2, Demo

include("generic.jl")
include("utilities.jl")
include("inpoly2.jl")
include("polydemo.jl")

end # module


text:
using Documenter, LRMoE

makedocs(
	doctest = false,
    modules = [LRMoE],
	sitename = "LRMoE.jl",
	pages = [
		"Overview" => "index.md",
		"Modelling Framework" => "framework.md",
		"Expert Functions" => "experts.md",
		"Model Initialization" => "init.md",
		"Fitting Function" => "fit.md",
		"Predictive Functions" => "predictive.md",
		"Examples & Tutorials" => Any[
			"Data Simulation" => "examples/sim_data/simulate_data.md",
			"Data Formatting" => "data_format.md",
			"Fitting Function" => "examples/sim_fit/simulate_fit.md",
			"Adding Customized Expert Functions" => "customize.md"
		],

		# "Tutorials" => Any[
		# 	"Tutorials Intro" => "tutorials/tutorial_main.md",
		# 	"1 Install Mimi" => "tutorials/tutorial_1.md",
		# 	"2 Run an Existing Model" => "tutorials/tutorial_2.md",
		# 	"3 Modify an Existing Model" => "tutorials/tutorial_3.md",
		# 	"4 Create a Model" => "tutorials/tutorial_4.md",
		# 	"5 Monte Carlo + Sensitivity Analysis" => "tutorials/tutorial_5.md"
		# ],
		# "How-to Guides" => Any[
		# 	"How-to Guides Intro" => "howto/howto_main.md",
		# 	"1 Construct + Run a Model" => "howto/howto_1.md",
		# 	"2 Explore Results" => "howto/howto_2.md",
		# 	"3 Monte Carlo + SA" => "howto/howto_3.md",
		# 	"4 Timesteps, Params, and Vars" => "howto/howto_4.md",
		# 	"5 Port to v0.5.0" => "howto/howto_5.md",
		# 	"6 Port to v1.0.0" => "howto/howto_6.md"
		# ],
		# "Advanced How-to Guides" => Any[
		# 	"Advanced How-to Guides Intro" => "howto_advanced/howto_adv_main.md",
		# 	"Build and Init Functions" => "howto_advanced/howto_adv_buildinit.md",
		# 	"Using Datum References" => "howto_advanced/howto_adv_datumrefs.md"
		# ],
		# "Reference Guides" => Any[
		# 	"Reference Guides Intro" => "ref/ref_main.md",
		# 	"Mimi API" => "ref/ref_API.md",
		# 	"Structures" => "ref/ref_structures.md",
		# 	"Structures: Definitions" => "ref/ref_structures_definitions.md",
		# 	"Structures: Instances" => "ref/ref_structures_instances.md"
		# ],
		# "FAQ" => "faq.md",
	],
	format = Documenter.HTML(prettyurls = get(ENV, "JULIA_NO_LOCAL_PRETTY_URLS", nothing) === nothing)
)

deploydocs(
	repo = "github.com/UofTActuarial/LRMoE.jl.git",
	branch = "gh-pages",
	devbranch = "main",
	versions = ["stable" => "v^", "v#.#.#", "dev" => "dev"],
)

max_stars_repo_name: sparktseung/JuliaLRMoE

max_issues_repo_name: sparktseung/JuliaLRMoE

max_forks_repo_name: sparktseung/JuliaLRMoE


text:
module Pasteee

import HTTP
import JSON

"""
    Section(contents; name = "", syntax = "")

Creates a paste `Section` with `contents`.
The `syntax` argument determines syntax highlight.
"""
struct Section
    name::String
    syntax::String
    contents::String
    Section(contents::String; name::String = "", syntax::String = "") = new(name, syntax, contents)
end

dict(sec::Section) = Dict("name" => sec.name, "syntax" => sec.syntax, "contents" => sec.contents)

"""
    paste(appkey, sections; description = "", expiration = "never")

Submit a paste to Paste.ee and return its `id`.
The `sections` argument is either a `Vector` of [`Section`](@ref) objects,
or a single `Section` object.

The `expiration` setting can be set to `"never"` (default), or to a number of seconds given
as a `String` (e.g., `"3600"` for one hour).

See <https://pastee.github.io/docs/>.
"""
function paste(
    appkey::AbstractString,
    sections::AbstractVector{Section};
    description::AbstractString = "",
    #encrypted::Bool = false, # Not sure how this works
    expiration::AbstractString = "never"
)
    headers = ["X-Auth-Token" => appkey, "Content-Type" => "application/json"]
    data = Dict{String,Any}(
        "sections" => dict.(sections),
        "expiration" => expiration,
        "description" => description
    )
    # if encrypted
    #     data["encrypted"] = true
    # end
    response = HTTP.post("https://api.paste.ee/v1/pastes", headers, JSON.json(data))
    return JSON.parse(String(response.body))["id"]
end

paste(appkey::AbstractString, section::Section; kw...) = paste(appkey, [section]; kw...)

"""
    paste(appkey, contents::String; name = "", syntax = "", ...)

Paste a single section with `contents`, `name` and `syntax` given.
"""
function paste(
    appkey::AbstractString, contents::String;
    name::String = "",
    syntax::String = "",
    kwargs...
)
    paste(appkey, Section(contents; name, syntax); kwargs...)
end

"""
    delete(appkey, id)

Deletes paste `id` from Paste.ee.
"""
function delete(appkey::AbstractString, id::AbstractString)
    headers = ["X-Auth-Token" => appkey]
    HTTP.request("DELETE", "https://api.paste.ee/v1/pastes/$id", headers)
    return nothing
end

"""
    get(appkey, id)

Fetch paste `id` from Paste.ee.
"""
function get(appkey::AbstractString, id::AbstractString)
    headers = ["X-Auth-Token" => appkey]
    response = HTTP.get("https://api.paste.ee/v1/pastes/$id", headers)
    return JSON.parse(String(response.body))["paste"]
end

"""
    pastes(appkey; perpage = 25, page = 1)

Retrieve all pastes, organized in pages containing `perpage` entries.
Returns entries in page number `page`.
"""
function pastes(appkey::AbstractString; perpage::Int = 25, page::Int = 1)
    headers = ["X-Auth-Token" => appkey, "Content-Type" => "application/json"]
    data = Dict("perpage" => perpage, "page" => page)
    response = HTTP.get("https://api.paste.ee/v1/pastes", headers, JSON.json(data))
    return JSON.parse(String(response.body))
end

end # module


text:
"""
    dual(h::Hypergraph)

Return the dual of the hypergraph `h`.

NOTE
`h` needs to have at least one dimension greater than 0.
"""

function dual(h::Hypergraph)
    @assert(nhv(h)>0 || nhe(h)>0)

    T = nhv(h) > 0 ? eltype(values(h.v2he[1])) : eltype(values(h.he2v[1]))
    V = isa(eltype(h.he_meta), Union) ? eltype(h.he_meta).b : Nothing
    E = isa(eltype(h.v_meta), Union) ? eltype(h.v_meta).b : Nothing

    mx = Matrix{Union{Nothing,T}}(nothing, nhe(h), nhv(h))

    for v=1:nhv(h)
        for he in keys(h.v2he[v])
            mx[he, v] = h.v2he[v][he]
        end
    end

    Hypergraph{T, V, E}(mx; v_meta=h.he_meta, he_meta=h.v_meta)
end