Smart Containers With Bidding Capacity: A Policy Gradient Algorithm for
  Semi-Cooperative Learning
  Smart modular freight containers -- as propagated in the Physical Internet
paradigm -- are equipped with sensors, data storage capability and intelligence
that enable them to route themselves from origin to destination without manual
intervention or central governance. In this self-organizing setting, containers
can autonomously place bids on transport services in a spot market setting.
However, for individual containers it may be difficult to learn good bidding
policies due to limited observations. By sharing information and costs between
one another, smart containers can jointly learn bidding policies, even though
simultaneously competing for the same transport capacity. We replicate this
behavior by learning stochastic bidding policies in a semi-cooperative multi
agent setting. To this end, we develop a reinforcement learning algorithm based
on the policy gradient framework. Numerical experiments show that sharing
solely bids and acceptance decisions leads to stable bidding policies.
Additional system information only marginally improves performance; individual
job properties suffice to place appropriate bids. Furthermore, we find that
carriers may have incentives not to share information with the smart
containers. The experiments give rise to several directions for follow-up
research, in particular the interaction between smart containers and transport
services in self-organizing logistics.
