Decision Transformer for Discrete action
I’m studying the decision transformer referring toTrain your first Decision Transformer.In the post, the example is for “halfcheetah” ( action space is continuous) andthe following model code is used.I’m trying to apply this to the discrete action space.I added the logit layer for the discrete action and changed the loss function as below.( red color: removed , blue color: added )Is this the right approach?image576×618 16.7 KB
2024-04-26T04:31:30Z
Hey,I’m also trying out something similar recently. Based on my implementation, your implementation looks almost similar to mine, except for the fact that I did not use an additional linear layer and directly used the outputs as the logit. Also, I had encode my actions into one-hot-encoding, so I had to do some reshaping with the action targets, but otherwise, I think this is pretty much spot on.I usually just print out the shapes and the intermediate variables at least once for a sanity check to make sure everything looks right and nothing it broadcasted incorrectly.
2024-12-04T17:20:19Z
Hey!and in the original_forward, why did you define the action_targets? Because I understand that the logits are the action_preds, no?
2024-12-04T19:52:03Z
Yeah, it does seem like the action_targets are technically not needed in the original_forward() function which is used during test mode, and I don’t see it being used in the code snippet either.
2024-12-07T11:36:57Z
Thank you for your quick response!I’m using DT for my bachelor degree final thesis and I’m a bit lost. Seeing that you have been using it for a longer time, I hope you don’t mind if I ask you some questions.I have a case very similar to the one you mentioned, with discrete actions and using one-hot encoding. So, does this code look good in your opinion?image1050×661 34.5 KBI’m also at a bit of a loss as to which activation function best fits this type of problem and certain model configuration parameters. How do you see these?image312×201 4.02 KBThanks in advanced!!!
2024-12-07T19:20:23Z
Thanks for sharing. It helps me a lot.