{'fraction_non_alphanumeric': 0.03867753623188406, 'fraction_numerical': 0.015126811594202899, 'mean_word_length': 4.380279402209227, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Do BERTs Learn to Use Browser User Interface? Exploring Multi-Step Tasks with Unified Vision-and-Language BERTs

Pre-trained Transformers are good foundations for unified multi-task models owing to their task-agnostic representation. Pre-trained Transformers are often combined with text-to-text framework to execute multiple tasks by a single model. Performing a task through a graphical user interface (GUI) is another candidate to accommodate various tasks, including multi-step tasks with vision and language inputs. However, few papers combine pre-trained Transformers with performing through GUI. To fill this gap, we explore a framework in which a model performs a task by manipulating the GUI implemented with web pages in multiple steps. We develop task pages with and without page transitions and propose a BERT extension for the framework. We jointly trained our BERT extension with those task pages, and made the following observations. (1) The model learned to use both task pages with and without page transition. (2) In four out of five tasks without page transitions, the model performs greater than 75% of the performance of the original BERT, which does not use browsers. (3) The model did not generalize effectively on unseen tasks. These results suggest that we can fine-tune BERTs to multi-step tasks through GUIs, and there is room for improvement in their generalizability. Code will be available online.


Introduction
Prior studies have attempted to unify models for processing natural language to facilitate the transfer of learned knowledge by reducing task-specific structures. For example, Radford et al. (2018); Devlin et al. (2019) suggest that language models with a generic structure, Transformer (Vaswani et al., 2017), are effective. Raffel et al. (2020) proposed a text-to-text framework which converts tasks into a problem where a model receives and generates  (a) (b) UI step Figure 1: Framework comparison. (a) Typical frameworks assume single-step tasks in which a model takes a sequence of text and images to generate an output. (b) In our WoB-like framework, we make a task as web pages, which allow structured contents, hyperlinks and scripts. The page design decides how to submit an answer (e.g., choose a button or input text). A model completes a task in multiple steps using the browser user interface (BUI). The model take a screenshot to output an action for each step. (e.g., click or keystroke).
text. Cho et al. (2021) extended the input of the text-to-text framework to accommodate images. However, existing research on unified models remains limited. First, although the models proposed by Cho et al. (2021) use a linear sequence of text and several images as input, their models are not designed to handle input with a layout. Second, existing unified models assume singlestep tasks. Task-specific design still must be completed when applying these models to compound tasks, such as reading a single document and subsequently searching for missing information. The latter challenge is more difficult to address because methods for using a transformer in multiple-step tasks, have not yet been fully established. Although transformer-based models have been successful in many language-related tasks such as language understanding (Wang et al., 2019), question answering (Rajpurkar et al., 2016), visual question answering (Antol et al., 2015), and referring expression comprehension (Kazemzadeh et al., 2014), nevertheless, these are single-step tasks.
In this study, we investigate the following question to explore the possibility of performing tasks: Can language models complete tasks through graphical user interfaces expressed as visual input? For graphical user interface manipulation, Shi et al. (2017) introduced the World of Bits (WoB) platform, in which tasks are expressed as web pages, and models complete those tasks via browser UIs. We follow their work and apply language models to a similar task framework (Figure 1).
We formulate the interaction between a browser and a model ( § 3), and create task pages based on the existing datasets, including GLUE (Wang et al., 2019), SQuAD (Rajpurkar et al., 2016) and VQA (Antol et al., 2015) ( § 4). Our tasks include not only single-page but also multi-page tasks that require page jumps to diversify the goal of actions. We introduce a BERT (Devlin et al., 2019) extension with a simple memory mechanism and pretraining for actions ( § 5). In our experiments, we train our model in a multi-task setting. We validate whether our model can learn in the framework and compare it with other models based on the same BERT. We show that our pre-training and memory mechanisms are effective and analyze the models' ability to solve unseen tasks ( § 6). Our contributions: • We construct a task set to evaluate how well a language model is transferred to the GUI manipulation retaining its linguistic knowledge. • We introduced a BERT extension and demonstrate its ability to learn diverse tasks (GLUE, SQuAD, VQA, and multi-page tasks) jointly.
2 Related Work

Execution Style of Unified Models
Unified Models aim to reduce task-specific structures to promote learning different tasks jointly such that learned knowledge can be shared between tasks 2 . After the success of transformer-based language models (LMs) (Devlin et al., 2019;Radford et al., 2019) and their visual extensions (Lu et al., 2019;Li et al., 2019a;Tan and Bansal, 2019;Chen et al., 2020;Su et al., 2020), unified models with transformers have received significant attention. We can categorize unified transformers in terms of task execution: task-specific head and text gen-eration styles. The task-specific head style shares most model weights between tasks and provides a head for each task. ViLBERT-MT (Lu et al., 2020) and UniT (Hu and Singh, 2021) use this style. The text generation style employs text generation to bridge the differences in output between tasks. GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020) show that large pre-trained models can multitask in the text region by changing the prompt. T5 (Raffel et al., 2020) and VL-T5 (Cho et al., 2021), which extends T5 to vision, also employ this style. The text generation style can be applied to, in principle, all tasks that can be expressed in a text-to-text format. Our WoB-like framework is also highly flexible. A model manipulates web pages that define a task via general actions. As a result, it extends the tasks to what can be rendered in a browser screen while keeping the model structure. We refer to this style as the BUI action style.

Vision-and-Language Tasks
World of Bit platform (Shi et al., 2017) is a reinforcement learning environment for web tasks, which is most relevant to our work. The MiniWoB benchmark and its extension MiniWoB++ (Liu et al., 2018) on the platform are widely used to study models for web tasks. They consist of simple tasks such as clicking a button, typing text or more complex tasks such as booking a flight. Although prior work on those benchmarks spans architecture/policy (Liu et al., 2018;Gur et al., 2018;Jia et al., 2019;Humphreys et al., 2022), parsing (Srivastava et al., 2020, and environment generation (Gur et al., 2021), Few studies have investigated the use of LM's linguistic knowledge in the task procedures. Document AI is a technique for automatically reading, understanding, and analyzing documents (Cui et al., 2021). Our work relates to studies on HTML documents. Tanaka et al. (2021);Chen et al. (2021) proposed reading comprehension datasets on web pages. Wu et al. (2021); Li et al. (2021a) proposed pretrained models for HTML documents. Although documents are processed differently (such as using screenshots or incorporating hierarchy of the elements), prior studies were concerned with a visually rich layout. Our focus is on the interaction between models and the documents. UI modeling is an emerging topic, and Bai et al. (2021);He et al. (2021) have pre-trained UI models for mobile devices to obtain better representations for the UI in terms of understanding tasks, such as predicting the application type or retrieving similar UI components. Li et al. (2021b) proposed a multitask UI model that can answer questions about the UI. While the questions include commands e.g., 'Go to the next screen', they are limited to singlestep commands. By contrast, our models use UIs by recurrently generating actions. Vision-and-language navigation (VLN) (Anderson et al., 2018;Das et al., 2018;Shridhar et al., 2020) studies models that follow instructions in a physical space, such as room. VLN tasks have progressed in action generation with V&L models. Recent studies used pre-trained LMs to encode instructions (Li et al., 2019b;Majumdar et al., 2020;Hong et al., 2021;Qi et al., 2021). However, the visual input rarely contains long text because the target is a physical space. Combination of views with a long text and actions remains a challenge.

Task Formulation with Browser UI
In this study, the term browser refers to software for accessing web pages. A browser renders web pages, navigates to a new page when a hyperlink is clicked, and executes the scripts on a page internally.
Our formulation focuses on browsers that run on personal computers 3 . We assume that the browser input devices are a mouse and keyboard, and that the browser provides a screenshot. At each step, the model partially observes the state of web pages from a screenshot to output an action. The cursor position is drawn as a dot in the screenshot. We apply the action to the browser and waited for a period of time (∼ 500ms) 4 for the browser to complete internal computation (e.g., rendering, navigating). Subsequently, we take the next screenshot. In conclusion, suppose a screenshot of the visible area of a page s i and model's action a i at step i, then the model predicts a i from s i :

Task Pages
This section describes the web pages we created for tasks. Although there are no restrictions on the layout of the pages, we used layouts that have an instruction, a main content, and an answer form for simplicity. We assumed that a task example has a single answer. Figure 2 summarizes task pages we made. This section describes the types of task page and how to obtain gold actions for training.

Types of Task Page
(a) Pre-Training for Actions (PTA). Prior knowledge of interface usage, such as the use of clickable buttons, could assist more efficient learning of tasks in the BUI by avoiding situations where models learn such knowledge and reasoning (e.g., reading comprehension) simultaneously. We introduced pre-training for actions: a set of small tasks that focus on moving the cursor, clicking a button, moving the visible area, and inputting text. As shown in Figure 2, in PTA tasks, the instructions are written at the top of the screen, and the model succeeds if it follows the instruction. We generated task instances using templates (in Appendix C.2).
To evaluate to what extent models can solve traditional tasks in BUI, we created tasks of this type based on existing datasets. We used GLUE (Wang et al., 2019) and SQuAD (Rajpurkar et al., 2016(Rajpurkar et al., , 2018 for natural language understanding and VQA (Antol et al., 2015) for visual grounding. Task pages of this type involve scrolling pages and submitting answers. We chose answer forms that matched the format of those datasets. We used buttons for GLUE (classification), and a text box for SQuAD and VQA (question answering). The condition for success is to submit the correct answer of the original datasets.
(c) Multi-page tasks. This type introduces page transitions to focus more on procedural tasks that BUI enables. We designed Search and Answer (SA) task ( Figure 3). For the task, we made databases on question answering tasks by sampling the contexts (paragraphs or images) and questions from SQuAD and VQA. We assigned unique ids to the contexts and questions. Task pages of SA are linked to one of those databases that can be queried with the search UI. The goal of the tasks is to answer a question about the database using the search UI. We prepared four groups to verify whether the models can handle different questions: • SA-H: How many questions are related to CID? requires querying a given Context ID (CID) and answering the number of Hits. • SA-Q: What is the question of QID?
requires identification of the question corresponding to a given Question ID (QID). • SA-QID: What is the QID of QUESTION?
requires identification of the QID corresponding to a given question. • SA-A: Answer the question of QID.
requires answering the question corresponding to a given QID. While SA-H, -Q, and -QID can be answered directly from the search results, SA-A requires models to display a detailed page to produce the answers. Appendix C.3 provides further detail.

Creating Gold Sequences
Supervised learning was used to train the proposed BUI model because the probability of completing a task by randomly acting on a page appears small. To record gold sequences of actionscreenshot pairs, we manually created rules for each task and manipulated task pages loaded in a browser following the rules. We designed the rules to identify the contents on a task page once and to take actions to submit answers.
Individual rules are listed as follows. Note that each rule breaks down further into the actions. • GLUE. Scroll down until the buttons appear (to view all contents). Click the correct button. • SQuAD and VQA. Scroll down until the submit form appears. If the question is unanswerable, check unanswerable. Otherwise, type the answer in the text box. Click the submit button. • SA-H. Query the CID in an instruction. Submit the number of hit records. • SA-Q. Query the QID in an instruction. Scroll down until the record related to the query appears. Submit the corresponding question.
• SA-QID. We swapped the question and QID in the SA-Q rule. We used only the first three tokens of the question to reduce the number of actions. • SA-A. Query QID in an instruction. Click the 'show' link to display the context. View the entire context to produce the answer for the question corresponding to the QID. Submit the answer.

BUI-BERT
This section describes how to extend the pre-trained BERT small to manipulate the browser UI. A small language model was used instead of pre-trained professional models with standard size (e.g., Lay-outLM) owing to the multiple long sequences required by the BUI setup. As illustrated in Figure 4, vision, language, memory, and some auxiliary tokens are fused with fusion BERT to obtain the next action. We initialized the weight of fusion BERT based on the weight of the pre-trained BERT and pre-trained the model using the PTA tasks.

Vision Input
We used grid features from a pre-trained convolutional neural network similar to Huang et al. (2020), considering the speed and amount of data. We encoded the screenshot at each step with a frozen pre-trained ResNet (He et al., 2016) 5 followed by a trainable fully connected layer.

Language Input
To treat words as a separate modality, we detected words from a screenshot and broke down the words into sub-words using a BERT uncased tokenizer 6 . We emulated a word-based OCR by inserting span tags between words in the HTML pages. We present a detection example in Appendix D.1. While this emulation works in text content and labels on the buttons, it does not capture text in text boxes. Despite this limitation, we use this emulation to reduce the computation time and avoid the necessity to consider detection errors. Note that models can recognize their input through the last action tokens. Location embedding. We added location embedding to each sub-word embeddings to indicate the location on a screenshot. The word bounding box 7

Fusion BERT
[ACT] Linear Linear Linear Linear was encoded by a trainable MLP. Sub-words in a word share the location embedding of the word.

Memory Mechanism
Our model uses memory to handle previous inputs. Suppose that K is a hyper-parameter for the memory token length (We used 64 in this paper). Our memory mechanism used 2 × K embeddings. The first half K was copied from the previous memory output, and the second half was filled with the [M] embedding, a trainable one-hot vector. After fusing inputs, we retained the K encoded embeddings corresponding to the second half for the next step. During training, we inputted the memory embeddings recurrently while the number of steps did not exceed the maximum (50 in our study).

Auxiliary Inputs
Last action. The last action is represented with the embeddings of the action name, the cursor position, and the sub-word 8 . We used trainable one-hot vectors for the action name and the sub-word embeddings. We encoded the cursor position using the same MLP as the word location by setting width and height to zero. Next action. We appended trainable one-hot vectors for [ACT], [TOK], [X], and [Y] tokens and inputted these tokens to predict the next action.

Next Action Prediction and Loss
We predicted the next action from the embeddings that the fusion BERT encoded the [ACT], [TOK], [X], and [Y] tokens. Suppose the encoded embeddigns are e act , e Tok , e x , and e y . We first classified the action name from e act . We then classified the token id in Fusion BERT's vocabulary from e tok for TOKEN and the pixel coordinate 9 from e x and e y for MOVETO. All embeddings are projected to the class distributions with trainable linear layers. During training, we used the Softmax cross-entropy loss for the action name, token, x, and y. These were evenly added in a mini-batch: where · denotes average for non-pad labels. Note that although our loss function does not explicitly refer to memory usage, our results in 9 x ∈ {1, ..., screen width} and y ∈ {1, ..., screen height}.  Section 6.1 and 6.3 show that the models learn to use memory with this loss.
6 Experiments 10 First, we trained our BUI models on the PTA tasks to pre-train the models. Second, we trained the BUI models in the multi-task setting; thereafter, we compared the BUI models to the models with different task styles. Finally, we analyzed the models.

Pre-Traing for Actions
We trained small and medium sized BUI-BERTs 11 on PTA tasks jointly with 60k training examples.
Setup. The memory length of both models was 64.
We set the screen at 640px×448px and resized the screenshots by half before inputting to ResNet18. The maximum epoch was 50. We tracked the validation loss at the end of each epoch and used the model with the smallest validation loss for the evaluation with the actual browser. During evaluation, the trial was stopped and considered a failure if a model did not submit an answer within 1.5 times the number of steps in the gold sequences. We used the ADAM optimizer (Kingma and Ba, 2014) with a fixed learning rate of 5e-5 and accumulated the gradient such that the mini-batch size was 128.
Results. Table 2 presents the results of the PTA tasks. Our models performed well in the cursor, button, and text tasks, showing that we can finetune language models on such simple actions. The model scores are above the chance level on the area task, which requires models to remember the instruction to choose a correct button. This suggests the effectiveness of our memory mechanism. BUT-BERT medium performs worse than BUT-BERT small in the button and area tasks despite its larger parameter size. The reason is not yet clear. Considering that both scored at the same level on the text task, a possible interpretation may be that the medium model did not learn the location embedding of subwords well and could misplaced the sub-words on the view.

Main Tasks
We trained the pre-trained BUI-BERTs on CoLA ( Compared models. Table 3 shows the summary. BERT small /+V: To estimate the upper bound of performance, we fine-tuned BERT small to each task independently, except SA, with task-specific heads. BERT small -s2s+V: For comparison with text generation models, we prepared an encoder-decoder model whose encoder and decoder weights were initialized based on the weights of BERT small . We trained that model on all the tasks except for SA jointly. The input sequences were generated such that they provided the most complete information required to solve a task, for example, task description, question, and class labels, using templates (in Appendix B.1). The models with the suffix +V use an image input for VQA. We obtained the grid features using ResNet18 in a manner similar to BUI-BERTs. We inserted the features into the head of the input embeddings. Appendix B.2 provides further details. Setup. We trained all models in 10 epochs. We tracked the validation loss at the end of each epoch to select the best model. The other conditions for BUI-BERTs were the same as those for the PTA training. We optimized the hyper-parameters for the compared models (see Appendix B.3).
Results. Table 4 summarizes the results. For the non-action or single-page tasks, we estimate the upper bound on the pre-trained BERTs small performance by taking the higher of BERT small (singletask setting) and BERT small -s2s+V (multi-task setting) performance. As we can see, BUI-BERT small achieves about 76%-90% of that estimation except for CoLA in our task framework. This suggests that pre-trained language models can learn to use their linguistic knowledge in an action sequence, although not perfectly. Note that the small CoLA scores in the multi-task setting can be due to   the relatively small training data. We observed no trend that apply to all tasks in the relationship between model performance and size (BUI-BERT small/meduim ). The medium model performs better on the single-page tasks, which is consistent with the size of the language models. However, their performance is reversed on the multi-page tasks. Further investigation is required to see if performance scales with model size in our tasks.

Analysis
Ablation study. We added two models to validate PTA and the memory mechanism. For BUI-BERT small w/o PTA, we initialized its weight with BERT small and directly trained it on the multitask training. For BUI-BERT small w/o mem, we omitted the memory sequence. This model was re-initialized and trained on PTA. Table 5 shows the results. Ablated models were lower than BUI-BERT small on almost all the tasks. This shows that PTA and the memory mechanism is effective. Especially, the significant degradation of BUI-BERT small w/o mem on the SA tasks suggests that memory plays an important role. SA tasks. We present the result of SA tasks in detail in Table 5. As we can see, all models were successful in submitting an answer in most cases except for BUI-BERT small w/o mem. However, the SA-A task scores are low. This low scores reflect the difference of the tasks: while the SA-QID, -Q, and -H tasks can be solved by copying displayed results, the SA-A task requires inferring the answer for a question. Bridging this gap is a challenge for the future. Note that the scores on SA-QID, -Q and -H also reflect the difficulty: SA-H is the easiest because the answer (the hit count) is always displayed at the first line; SA-Q is relatively easy because the hit count is always; SA-QID is more difficult because the hit count is more than one, and models are sometimes required to find the answer by scrolling. There is still room for improvement regarding the SA-Q and -QID tasks. Unseen tasks. To see how the models understood the instruction, we evaluated them on unseen tasks. We used three GLUE tasks: WNLI (Levesque et al., 2012), MRPC (Dolan and Brockett, 2005), and SST-2 (Socher et al., 2013). Those were twochoice tasks, and their similarity to the learned tasks was differed. WNLI and MNLI were textual entailment tasks. MRPC and STS-B were equivalence and similarity tasks. SST-2 is a sentiment prediction, which was new to the models. We report the number of submissions (#sub) and correct submissions (#cor) in Table 6. We observed that BUI-BERT smallmedium failed to submit an answer in many cases although the submission rate of those models is above a chance level estimated from the percentage of the area occupied by the buttons on the screen. Based on the percentage of correct answers in the submissions, it does not seem that the models refrain from submitting their answers when they are not confident. This low submission rate suggests that the models likely memorize the entire procedures without aligning the actions in a procedure to the words in an instruction.

Discussion
Expanding the world that NLP can handle is one of our long-term goals (Bisk et al., 2020). Models that manipulate GUIs provide a unified basis for studies on dynamic interactions, e.g., dynamic grounding (Chandu et al., 2021), because such mod-   els can adapt to diverse software through GUIs. We converted existing benchmark tasks into GUI tasks in this paper. Such GUI tasks would help evaluate how well the performance of a language model is transferred to a GUI model. This evaluation viewpoint complements MiniWoB (Shi et al., 2017) and MiniWoB++ (Liu et al., 2018).
Although our experiments showed encouraging results that language models can learn GUI manipulation while retaining linguistic knowledge to some extent, our study has the following limitations. • The high computational cost precludes detailed experiments, including validation of scale by using larger LMs. We need to seek light-weight models, such as efficient transformers (Tay et al., 2020) or Perceiver (Jaegle et al., 2021). • Performance on unseen tasks is low. Although we expect that data augmentation (i.e., modifying the page layout) can improve the performance, reinforcement learning (Sutton and Barto, 2018) is likely to be essential for exploration. • Our models use the emulated OCR, which relies on a browser. To apply our models to GUIs other than the browser GUI, changing that function remains future work. While using an actual OCR is a straightforward solution, its processing time is not negligible for our purpose. Therefore, we might create LMs that directly read from pixels.

Conclusion
In this work, we demonstrated that BERT can be applied to a task framework that requires multiple actions to use a browser UI. In multi-task training, our BERT extension with a memory mechanism learned to solve six tasks according to the UI, including hyperlinks, provided by the task pages. Simultaneously, we observed the low ability to solve unseen tasks. Because the computational cost of current models limits in-depth experiments, we plan to create light-weighted models with existing efficient methods as the next step.

A Environment Detail
Browser. We used the following environment to render and execute task pages: Packages and libraries. We used Python 3.6 and PyTorch (1.10) to implement our BUI models. For sequence to sequence models, we used the Transformers library (4.12). To evaluate SQuAD and VQA, we used the public scripts 12 .
Training. We used a NVIDIA V100 GPU with 32 GB VRAM or a NVIDIA RTX 3090 GPU with 24 GM VRAM in each training run.

A.1 Required Time
We report the required time for our experiments in the Table 7.

B.1 Input Templates
We used templates to make the text inputs for the seq2seq models. Table 8 shows the templates and the rule to fill the sentences in the template for each dataset. We made the templates so that they provided task_type, instruction, content names and content values. In addition, image embeddings are added to the head of sequence for VQA. We made TASK_TYPE and INSTRUCTION with reference to the contents of the data set.

B.2 Image Input
First, we resize an given image 320px× 224px. If the aspect ratio does not match, we center the image and fill in the missing pixels with black pixel. Second, we input the image to the frozen pre-trained ResNet18 model to obtain the last feature map (C4; 10x7). We flat the feature map in one dimension and input each feature to one fully-connected linear layer, which is trainable, to align the dimension with the hidden dimension of the LM. Finally, we 12 SQuAD : https://rajpurkar.github. io/SQuAD-explorer/ and VQA : https: //github.com/GT-Vision-Lab/VQA. concatenate those features and language embeddings before adding positional and segment type embeddings.

B.3 Hyper-parameters
We used the ADAM optimizer without scheduling of learning rate (LR), and enabled Automatic Mixed Precision (AMP). Every training was 10 epoch. In a preliminary experiment, we observed that optimization of pre-trained ResNet18 had little impact on the VQA performance, so we only used the frozen setting above.
BERT small / +V. We fixed the max token length for the GLUE tasks and VQA 300, and for SQuAd 512. We tried six hyper-parameter combination: the mini-batch size from {64, 128, 256}, the LR from {1e-4, 5e-5}. We adopted the hyperparameter set whose smallest validation loss was the smallest.

B.4 Classification with Seq2Seq models
For classification tasks, we considered the model failed to submit an answer when the generated text did not exactly match any class labels specified in the instruction.

C Tasks in the BUI setup C.1 Instructions for Answer Forms
Here, we shows the instructions and answer forms as images. Figure 5 shows the SQuAD and VQA pages. Figure 6 shows the task pages for the GLUE tasks. Figure 7 and Figure 8 show the task pages for the SA tasks. Instructions are basically the same as the counter parts for the seq2seq models shown in Table 8 except for that word choices are changed so that they fit to the screen.

C.2 Templates for Pre-Training for Actions
Vocabulary. We made a vocabulary from the training split of the Wikitext103 (Merity et al., 2016)       only alphabets and numbers. We lower-cased the words. Sets of words in the instructions are expanded to make the variation. We sampled one uniformly from the instructions for a task instance.

C.2.1 Cursor Instructions:
• Move the cursor in the box.
• Point to the box with the cursor.
The coordinates of the box was sampled from a window uniformly. WORD was sampled from the vocabulary.

C.2.3 Area Instructions:
• Scroll down until the buttons appear and click the button labelled WORD. • Scroll down until the buttons appear and click the WORD button.
WORD was sampled from the vocabulary.

C.2.4 Text
Instructions: • {Type, Enter, Input} the string to the left of it in each text box. Click the submit button at last.
Each string was made by jointing two words, sampled from the vocabulary, with a space.

C.3 Detail of Search and Answer Tasks
For Search and Answer Tasks, we sampled 100 contexts (paragraphs or images) from each of SQuAD and VQA to create a database. The database contains ∼2k questions because each context has approximately 10 questions. We chose this database size to make it difficult to enter the whole data into the model. We assigned unique labels to each context and question in the database, CID, and QID, and created four tasks. A database yields 200 SA-H tasks and ∼2k SA-QID, -Q, -A tasks. Finally, we sampled 500 tasks from those generated tasks. In total, we created 100 databases (50000 tasks) for the training split, and 10 databases (5000 tasks) for the validation split. The contexts do not overlap between databases.
The search UI uses partial matching on the entries C.4 Distribution of the Gold Sequence Length Figure 9 shows the distributions of the length of gold action sequences. Almost all of the examples fall within the upper limit of 50 steps that we set during our training. Tasks that require entering answers into text boxes tend to have a longer number of steps.

D.1 OCR Emulation
We used OCR emulation, where we surrounds each word in HTML sources using span tags, instead of real OCR in this work. Figure 10 shows an example. The Emulation do not capture the text in text boxes owing to technical reason. Words are sorted in a top faster and left faster manner. Sorting preserves natural orders basically, but it sometimes breaks the order as shown in the figure.
D.2 Mini-Batching Strategy Figure 11 illustrates mini-batching we used for training. We packed multiple trajectories in a line of mini-batches to increase the filling rate. We input memory and last actions recurrently for a trajectory and reset them at each head of trajectories. Figure 12 shows the learning curves of the BUI models. In the PTA training, three models were roughly converged. In the multi-task training, all models except BUI-BERT small w/o PTA were roughly converged in 10 epoch. However, the loss of BUI-BERT small w/o PTA began to reduce drastically around 5k update and it could become smaller after 10 epoch. This indicate that PTA speeds up the convergence of the loss at least, but it may not affect the final performance achieved after longer time.

D.4 Cases of Task Execution
We show the cases of task execution using BUI-BERT small in Figure 13 as an aid to understanding.  Since the number of actions in the document classification task is basically two, we showed MNLI as a typical example.