{'fraction_non_alphanumeric': 0.025373817290802545, 'fraction_numerical': 0.018259111104550755, 'mean_word_length': 4.793893782604977, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Does Collaborative Editing Help Mitigate Security Vulnerabilities in Crowd-Shared IoT Code Examples?

Background: With the proliferation of crowd-sourced developer forums, software developers are increasingly sharing more coding solutions to programming problems with others in forums. The decentralized nature of knowledge sharing on sites has raised the concern of sharing security vulnerable code, which then can be reused into mission critical software systems - making those systems vulnerable in the process. Collaborative editing has been introduced in forums like Stack Overflow to improve the quality of the shared contents. Aim: In this paper, we investigate whether code editing can mitigate shared vulnerable code examples by analyzing IoT code snippets and their revisions in three Stack Exchange sites: Stack Overflow, Arduino, and Raspberry Pi. Method:We analyze the vulnerabilities present in shared IoT C/C++ code snippets, as C/C++ is one of the most widely used languages in mission-critical devices and low-powered IoT devices. We further analyse the revisions made to these code snippets, and their effects. Results: We find several vulnerabilities such as CWE 788 - Access of Memory Location After End of Buffer, in 740 code snippets . However, we find the vast majority of posts are not revised, or revisions are not made to the code snippets themselves (598 out of 740). We also find that revisions are most likely to result in no change to the number of vulnerabilities in a code snippet rather than deteriorating or improving the snippet. Conclusions: We conclude that the current collaborative editing system in the forums may be insufficient to help mitigate vulnerabilities in the shared code.


INTRODUCTION
Crowd-sourced developer forums like Stack Overflow (SO) are popular among developers. The Stack Exchange network of sites that host Stack Overflow had 9+ billion page views from 100+ million users in 2019 alone. Stack Overflow now hosts more than 50 million posts and is visited by 11 million users per day. Questions related to coding challenges often receive code examples as solutions. Stack Overflow contains code snippets in 75% of their answers [41]. The quality of these examples and their direct reuse without modifications is a concern. Previous studies have found that 9.8% of 7,444 Stack Overflow accepted answers contained at least one instance of a poor coding practice [28], and Android code examples shared in SO are reused in millions of popular Android app [11].
Collaborative editing is introduced in forums like SO to allow users to suggest ways to improve the shared content. Previous studies offer valuable insight on the security of SO C/C++ code examples and the effect of post revisions [40][41][42]. However, what happens when a vulnerable code example is revised?
In this paper, we attempt to answer this question by analyzing vulnerable crowd-shared C/C++ IoT code examples. C/C++ is widely used in mission-critical systems and resource-constrained IoT devices. The Internet of Things (IoT) is an internet connected system of physical objects ("things") [17]. Rapid developments in IoT have made it so that an estimated 27.1 billion IoT devices will be connected by 2025 [30]. Increased demand for IoT in various use cases consequently increases the importance of understanding the unique challenges of IoT security [15].
The prevalence of IoT devices in our everyday life and the ease of access to such computing resources make development using IoT devices widespread. As such, recent research reports a growing number of IoT related posts in forums like SO [33,39]. In addition, certain Stack Exchange sites are more specialized to certain fields or technologies. For example, the Arduino and Raspberry Pi sites are designed for discussion regarding development using these two tools, which are popular in the field of IoT.
We study all IoT C/C++ code snippets shared in the SO, Arduino, and Raspberry Pi Stack Exchange sites. We apply a static code vulnerability analysis tool (cppcheck) to check each code example for vulnerabilities. For each identified vulnerable code example, we collect its revision history from the three sites. We then check whether the vulnerability was introduced pre or post-revision of a code example, and whether certain revision types (e.g., code improvement) introduced or fixed vulnerabilities.
We find several severe vulnerabilities in the shared code snippets like CWE 788 -Access of Memory Location After End of Buffer. We also find that most snippets are not revised, and the majority of vulnerabilities are introduced pre code revisions. When revisions

Data Collection
We study IoT code examples shared in the following three Stack Exchange sites: SO, Arduino, Raspberry Pi. SO is the most popular Q&A site for software developers of all kinds. The other two sites are specifically setup to foster IoT-based discussions. We download the January 2022 data dump of each site, and then obtain all code examples present in answers from Arduino and Raspberry Pi. For SO, we obtain code examples from answers that belong to questions labeled as the 75 IoT-related tags from Uddin et al. [39]. For SO, the question tags were checked to see if they contained the keywords C or C++, and if they contained IoT related keywords. We observed however that Arduino and Raspberry Pi questions did not have programming languages present in their tags in most cases. In order to determine the language of the code examples on these sites, we instead used the language detection tool Guesslang which has an accuracy of 90% according to it's documentation [14]. Then, we used a similar approach used in previous studies to reject code snippets that only contained pseudo code by ignoring snippets that contained less than the median SO line count of 5 lines [5,18,42]. We also collected the entire version history of each of these code snippets. In total, we obtain 13,170 code snippets from 10,248 posts with 19,018 post versions. A breakdown of the collected snippets as well as their versions is shown in Table 1.

Data Preprocessing
The collected snippets were analyzed for weakness on the CWE (Common Weakness Enumeration) List. To do this we used cppcheck version 2.4.1 released in March 2021, which is a static code analyzer that supports various types of code checks in C and C++ code. It also allows for specific weaknesses to be suppressed. According to Zhang et al, cppcheck is able to identify 59 out of the 90 code weaknesses that are related to C and C++ [42]. Previous studies have found that cppcheck had just a 0.78 false positive rate against a test suite of 650 common C/C++ bugs [3]. Zhang et al. found that 85 out of 100 CWE instances detected by cppcheck were labelled as accurate with a strong agreement among the study's authors (Cohen's Kappa of 0.68) [42]. While we analyzed the initial results of running the obtained code snippets through cppcheck, we observed many instances of syntax errors. These errors are likely to be automatically detected by code editors and removed by the programmer, so we proceeded to ignore such errors in our analysis, similarly to Zhang et al., who ignored 129,395 instances of syntax errors in their initial observation of 154,198 CWE instances. Other reported errors we noticed to be unfair to deem as a weakness are CWE types such as CWE 563 -Assignment to Variable without Use. Such errors are not important as users of Q/A sites often intend to answer specific questions in code examples with direct answers, not to provide complete solutions. We therefore suppress errors in cppcheck of this nature, which are summarized in Table 2.

STUDY RESULTS
Our empirical study answers for research questions (RQ) to offer insights into the relationship between the vulnerability of IoT code snippets and their revisions in three Stack Exchange sites: (1) Were the vulnerabilities introduced through post revisions?
(Section 4.1) We then tracked the revisions of each code snippet, and identified when the vulnerability was introduced. We find that that the vast majority of vulnerabilities are introduced before edits are made to code snippets, i.e., when the code snippet was first shared. As shown in Figure 3, 713 out of the 740 vulnerable code snippets were either not changed, or contained CWE instances before revisions were made. Table Table 3 shows the distribution of the vulnerable code snippets by the stage when the vulnerability was first found in the corresponding code snippet (pre or post-revisions). We see that ESEM '22, September 19-23, 2022, Helsinki, Finland Selvaraj and Uddin 2lightgray!30 To understand when a user may introduce a vulnerability, we study the evolution and version history of the collected code snippets. To determine the stage in which a vulnerability was introduced in a code example, we first collected the entire version history of each post. This version history was then manually analyzed for edits, and was restricted to the code examples themselves. Therefore edits made to the text portion of the answer were not counted as revisions. We utilized this method as minor text changes in the post, such as typo fixes, will have no affect on the vulnerabilities present in the code segment. After obtaining all code snippet revision history, we analyzed snippets that contained multiple versions for vulnerabilities in order to determine which version the error was introduced. Using Cppcheck to detect CWE types, we compared the results from the revised snippets with the initial ones.

Results.
Out of our studied 13,170 code examples across the three Stack Exchange sites, we found total 740 code snippets flagged as vulnerable by the cppcheck tool. We then tracked the revisions of each of the 740 code snippets and identified when the vulnerability was introduced in the code snippet. We find that that the vast majority of vulnerabilities are introduced before edits are made to code snippets, i.e., when the code snippet was first shared. As shown in Figure 3, 713 out of the 740 vulnerable code snippets were either not changed, or contained CWE instances before revisions were made.

Post Revisions 27 VCS
Pre Revisions 713 VCS Figure 3: Distribution of vulnerable code snippets (VCS) by the stage when their vulnerabilities were introduced (pre vs post revision of a code snippet) Table Table 3 shows the distribution of the vulnerable code snippets by the stage when the vulnerability was first found in the corresponding code snippet (pre or post-revisions). We see that only a few of the vulnerable code snippets in each site were introduced post-edit, i.e., most of the snippets contained a vulnerability when the code examples were first shared. Out of the three sites, 95.8% of the vulnerable code snippets in SO had the vulnerabilities introduced pre-edit. The site Arduino had 96.6% of the vulnerable code snippets introduced during pre-edit and 3.4% introduced during post-edit. Finally, Raspberry Pi had no vulnerabilities introduced after revisions. Listing 4 shows an example vulnerability that was not present in original version but was introduced once the user made a revision in an attempt to improve their code. In this case, an instance of CWE 398 -7PK Code Quality in line 13 was introduced after the inclusion of a for loop that uses a variable (loop) with the same name as another function. An example of a vulnerability that was introduced before revisions and was never fixed during revisions can be found in SO post (27918518), where an instance of CWE 401 -Missing Release of Memory after Effective Lifetime is present in line 5. Figure 4 shows the evolution of the number of CWE introduced pre and post revisions from 2009 to 2021. We find that although the number of CWEs introduced pre revisions dramatically increased between 2012 and 2016, it has been on a slight decline since. Furthermore, the number of new CWEs introduced after a revision has been made has remained relatively consistent and small.
We also find the IoT devices affected by the CWE types detected in the code snippets by analyzing the CVE descriptions of the CWE types. As shown in Fig 5, Snapdragon Consumer IoT and Snapdragon Industrial IoT are the most common affected IoT devices, followed by other devices such as GoPro cameras.

Approach.
To understand when a user may introduce a vulnerability, we study the evolution and version history of the collected code snippets. To determine the stage in which a vulnerability was introduced in a code example, we first collected the entire version history of each post. This version history was then manually analyzed for edits, and was restricted to the code examples themselves. Therefore edits made to the text portion of the answer were not counted as revisions. We utilized this method as minor text changes in the post, such as typo fixes, will have no affect on the vulnerabilities present in the code segment. After obtaining all code snippet revision history, we analyzed snippets that contained multiple versions for vulnerabilities in order to determine which version the error was introduced. Using Cppcheck to detect CWE types, we compared the results from the revised snippets with the initial ones.

Results.
Out of our studied 13,170 code examples across the three Stack Exchange sites, we found total 740 code snippets flagged as vulnerable by the cppcheck tool. We then tracked the revisions of each of the 740 code snippets and identified when the vulnerability was introduced in the code snippet. We find that that the vast majority of vulnerabilities are introduced before edits are made to code snippets, i.e., when the code snippet was first shared. As shown in Figure 3, 713 out of the 740 vulnerable code snippets were either not changed, or contained CWE instances before revisions were made.

Post Revisions 27 VCS
Pre Revisions 713 VCS Figure 3: Distribution of vulnerable code snippets (VCS) by the stage when their vulnerabilities were introduced (pre vs post revision of a code snippet) Table Table 3 shows the distribution of the vulnerable code snippets by the stage when the vulnerability was first found in the corresponding code snippet (pre or post-revisions). We see that only a few of the vulnerable code snippets in each site were introduced post-edit, i.e., most of the snippets contained a vulnerability when the code examples were first shared. Out of the three sites, 95.8% of the vulnerable code snippets in SO had the vulnerabilities introduced pre-edit. The site Arduino had 96.6% of the vulnerable code snippets introduced during pre-edit and 3.4% introduced during post-edit. Finally, Raspberry Pi had no vulnerabilities introduced after revisions. Listing 4 shows an example vulnerability that was not present in original version but was introduced once the user made a revision in an attempt to improve their code. In this case, an instance of CWE 398 -7PK Code Quality in line 13 was introduced after the inclusion of a for loop that uses a variable (loop) with the same name as another function. An example of a vulnerability that was introduced before revisions and was never fixed during revisions can be found in SO post (27918518), where an instance of CWE 401 -Missing Release of Memory after Effective Lifetime is present in line 5. Figure 4 shows the evolution of the number of CWE introduced pre and post revisions from 2009 to 2021. We find that although the number of CWEs introduced pre revisions dramatically increased between 2012 and 2016, it has been on a slight decline since. Furthermore, the number of new CWEs introduced after a revision has been made has remained relatively consistent and small.
We also find the IoT devices affected by the CWE types detected in the code snippets by analyzing the CVE descriptions of the CWE types. As shown in Fig 5, Snapdragon Consumer IoT and Snapdragon Industrial IoT are the most common affected IoT devices, followed by other devices such as GoPro cameras.
Listing 4 shows a vulnerability that was not present in original version but was introduced once the user made a revision in an attempt to improve their code. In this case, an instance of CWE 398 -7PK Code Quality in line 13 was introduced after the inclusion of a for loop that uses a variable (loop) with the same name as another function. An example of a vulnerability that was introduced before revisions and was never fixed during revisions can be found in SO post ( 27918518 ), where an instance of CWE 401 -Missing Release of Memory after Effective Lifetime is present in line 5. Figure 4 shows the evolution of the number of CWE introduced pre and post revisions from 2009 to 2021. We find that although the number of CWEs introduced pre revisions dramatically increased between 2012 and 2016, it has been on a slight decline since. Furthermore, the number of new CWEs introduced after a revision has been made has remained relatively consistent and small.
We also find the IoT devices affected by the CWE types detected in the code snippets by analyzing the CVE descriptions of the CWE types. As shown in    Summary of RQ1: Out of the 740 vulnerable code snippets, we find that the vast majority had vulnerabilities introduce pre code revisions (713 out of 740), which is also the case when looking at each stack exchange site individually. We analyze the vulnerabilities and CWE types present in the Stack Exchange code snippets to better understand their overall security. We also aim to determine if revisions add or remove certain types of vulnerabilities in order to gain more insight on the effect of code revisions.

Approach.
We use the output generated by cppcheck to determine types of CWEs present and the CWE database for their descriptions and characteristics. Using these characteristics, we try to understand the root cause behind the weaknesses in order to determine a categorization. This is done by both of the paper's authors. We then check how revisions impacted the distribution of vulnerable code snippets under each category.

4.2.3
Results. Across the three stack exchange sites, we observe a total of 31 CWE types present in the 740 vulnerable code snippets (total of 1221 CWE instances), with many code snippets containing more than one CWE type. The most frequently occurring CWE type we find is CWE 398 -7PK Code Quality, which was present in 520 code snippets. This CWE type however is not a distinct vulnerability, but instead indicates that the code snippet is of poor quality. We categorize the remaining 30 distinct CWE types based on their characteristics into the following 8 weakness categories: Evaluation -incorrect logic and comparisons, Memory -insufficient memory management, Function -improperly called and designed functions , Initialization -improperly initialized variables, Reachability -unreachable or undefined code, Resource -mismanagement of a program's resources, Conversion -incorrect conversions of variables to different types, and Calculation -improper or incorrect calculations. In Figure 6 we observe that code snippets are more likely to contain CWE types that are evaluation, memory, and initialization related. Among the 30 distinct CWE types shown in Table 4, we find that CWE 457 -Use of Uninitialized Variable is the most frequently occurring (detected in 49 out of 294 code snippets), followed by CWE 686 -Function Call With Incorrect Argument Type (32 out of 294). Table 5 displays the proportion of vulnerable code snippets in each weakness category that have weaknesses introduced pre or post-edit. We find that across all categories and types, vulnerabilities are introduced more frequently before revisions are made than after, and for most categories there are no vulnerable code snippets that

Approach.
We use the output generated by cppcheck to determine the types of CWEs present, and the CWE database for their descriptions and characteristics. Using these characteristics, we try to understand the root cause behind the weaknesses in order to determine a categorization. This is done by both of the paper's authors. We then check how revisions impacted the distribution of vulnerable code snippets under each category.

4.2.3
Results. Across the three stack exchange sites, we observe a total of 31 CWE types present in the 740 vulnerable code snippets (total of 1221 CWE instances), with many code snippets containing more than one CWE type. The most frequently occurring CWE type we find is CWE 398 -7PK Code Quality, which was present in 520 code snippets. This CWE type however is not a distinct vulnerability, but instead indicates that the code snippet is of poor quality. We categorize the remaining 30 distinct CWE types based on their characteristics into the following 8 weakness categories: Evaluation -incorrect logic and comparisons, Memory -insufficient memory management, Function -improperly called and designed functions , Initialization -improperly initialized variables, Reachability -unreachable or undefined code, Resource -mismanagement of a program's resources, Conversion -incorrect conversions of variables to different types, and Calculation -improper or incorrect calculations. In Figure 6 we observe that code snippets are more likely to contain CWE types that are evaluation, memory, and initialization related.
Among the 30 distinct CWE types shown in Table 4, we find that CWE 457 -Use of Uninitialized Variable is the most frequently occurring (detected in 49 out of 294 code snippets), followed by CWE 686 -Function Call With Incorrect Argument Type (32 out of 294). Table 5 displays the proportion of vulnerable code snippets in each weakness category that have weaknesses introduced pre or post-edit. We find that across all categories and types, vulnerabilities are introduced more frequently before revisions are made than after, and for most categories there are no vulnerable code snippets that contain weaknesses introduced post revisions. We also observe that memory related CWE instances are more likely to be introduced by code revisions compared to evaluation related weaknesses or instances of CWE 398.
When looking at code snippets that contained vulnerabilities before revisions, but then saw a reduction in CWE instances after, we find that only 3 weakness categories (evaluation, memory, and initialization) and CWE 398 contained such snippets.   contain weaknesses introduced post revisions. We also observe that memory related CWE instances are more likely to be introduced by code revisions compared to evaluation related weaknesses or instances of CWE 398. When looking code snippets that contained vulnerabilities before revisions, but then saw a reduction in CWE instances after, we find that only 3 weakness categories (evaluation, memory, and initialization) and CWE 398 contained such snippets.

Summary of RQ2:
We detect a total of 31 CWE types in 740 vulnerable code snippets (1221 CWE instances in total). These CWE types were generalized into 8 categories, and we find that evaluation, memory, and initialization weaknesses are the most common. Across all categories, vulnerabilities are more frequently introduced before revisions are made to the snippet.

RQ 3 Does the type of vulnerabilities differ
depending on revision types?
Motivation. Users revise their answers for a variety of reasons. We aim to better understand the behaviour of these users by analyzing the types of revisions that are more commonly made when editing code snippets.

Approach.
We manually analyze code snippets that have at least one code revision by looking at the nature of the edit. Then following work by Wang et al. [41], we label each revision with four revision types: code correction, code formatting, code improvement, and code removal/addition.

4.3.3
Results. An overview of the 4 revision types we label the revised code snippets as is shown in Table 6. Overall we find that 142 out of the 740 vulnerable code snippets are revised, as shown in Figure 7. Out of the 598 vulnerable code snippets that did not have any code revisions, 327 originate from posts that were never revised, meaning the user did not make any changes to the original version of the post. The remaining 271 code snippets were not revised themselves, but belong to posts that were only textually edited. For example, users edit posts to correct typos, clarify their work, or to include references and links to other resources. For the 142 code snippets that were revised, we observe that users more frequently make revisions with the intention to improve the functionality of the example to better answer the question (86 out of 142 snippets, 60.6%), and to make corrections in the code such as fixing syntax errors and typos (31 out of 142, 21.8%). As shown in Figure 8, the least common type of edit we observe in the code snippets with revisions is formatting changes, such has adding white space through indentation (9 out of 142 code snippets).

19%
Not Revised 598 81% We then further analyze the relationship between the CWE type present in a revised code snippet and it's revision type. Table 7 shows that across the 6/8 weakness categories that contain vulnerable code snippets that were revised, the most common revision type is code improvement. We also find that code correction revisions are more common in snippets with evaluation and initialization weaknesses (3 and 4 snippets respectively), and code removal/addition   contain weaknesses introduced post revisions. We also observe that memory related CWE instances are more likely to be introduced by code revisions compared to evaluation related weaknesses or instances of CWE 398. When looking code snippets that contained vulnerabilities before revisions, but then saw a reduction in CWE instances after, we find that only 3 weakness categories (evaluation, memory, and initialization) and CWE 398 contained such snippets.

Summary of RQ2:
We detect a total of 31 CWE types in 740 vulnerable code snippets (1221 CWE instances in total). These CWE types were generalized into 8 categories, and we find that evaluation, memory, and initialization weaknesses are the most common. Across all categories, vulnerabilities are more frequently introduced before revisions are made to the snippet.

RQ 3 Does the type of vulnerabilities differ
depending on revision types? 4.3.1 Motivation. Users revise their answers for a variety of reasons. We aim to better understand the behaviour of these users by analyzing the types of revisions that are more commonly made when editing code snippets.

Approach.
We manually analyze code snippets that have at least one code revision by looking at the nature of the edit. Then following work by Wang et al. [41], we label each revision with four revision types: code correction, code formatting, code improvement, and code removal/addition.

4.3.3
Results. An overview of the 4 revision types we label the revised code snippets as is shown in Table 6. Overall we find that 142 out of the 740 vulnerable code snippets are revised, as shown in Figure 7. Out of the 598 vulnerable code snippets that did not have any code revisions, 327 originate from posts that were never revised, meaning the user did not make any changes to the original version of the post. The remaining 271 code snippets were not revised themselves, but belong to posts that were only textually edited. For example, users edit posts to correct typos, clarify their work, or to include references and links to other resources. For the 142 code snippets that were revised, we observe that users more frequently make revisions with the intention to improve the functionality of the example to better answer the question (86 out of 142 snippets, 60.6%), and to make corrections in the code such as fixing syntax errors and typos (31 out of 142, 21.8%). As shown in Figure 8, the least common type of edit we observe in the code snippets with revisions is formatting changes, such has adding white space through indentation (9 out of 142 code snippets).

19%
Not Revised 598 81% We then further analyze the relationship between the CWE type present in a revised code snippet and it's revision type. Table 7 shows that across the 6/8 weakness categories that contain vulnerable code snippets that were revised, the most common revision type is code improvement. We also find that code correction revisions are more common in snippets with evaluation and initialization weaknesses (3 and 4 snippets respectively), and code removal/addition Summary of RQ2: We detect a total of 31 CWE types in 740 vulnerable code snippets (1221 CWE instances in total). These CWE types were generalized into 8 categories, and we find that evaluation, memory, and initialization weaknesses are the most common. Across all categories, vulnerabilities are more frequently introduced before revisions are made to the snippet.

RQ 3 Does the type of vulnerabilities differ
depending on revision types? 4.3.1 Motivation. Users revise their answers for a variety of reasons. We aim to better understand the behaviour of these users by analyzing the types of revisions that are more commonly made when editing code snippets.

Approach.
We manually analyze code snippets that have at least one code revision by looking at the nature of the edit. Then following work by Wang et al. [41], we label each revision with four revision types: code correction, code formatting, code improvement, and code removal/addition.

4.3.3
Results. An overview of the 4 revision types we label the revised code snippets as is shown in Table 6. Overall we find that 142 out of the 740 vulnerable code snippets are revised, as shown in Figure 7. Out of the 598 vulnerable code snippets that did not have any code revisions, 327 originate from posts that were never revised, meaning the user did not make any changes to the original version of the post. The remaining 271 code snippets were not revised, but rather were part of posts that were only textually edited. For example, users edit posts to correct typos, clarify their work, or to include references and links to other resources. For the 142 code snippets that were revised, we observe that users more frequently make revisions with the intention of improving the functionality of the example to better answer the question (86 out of 142 snippets, 60.6%), and to make corrections in the code such as fixing syntax errors and typos (31 out of 142, 21.8%). As shown in Figure 8, the least common type of edit we observe in the code snippets with revisions is formatting changes, such has adding white space through indentation (9 out of 142 code snippets).   contain weaknesses introduced post revisions. We also observe that memory related CWE instances are more likely to be introduced by code revisions compared to evaluation related weaknesses or instances of CWE 398. When looking code snippets that contained vulnerabilities before revisions, but then saw a reduction in CWE instances after, we find that only 3 weakness categories (evaluation, memory, and initialization) and CWE 398 contained such snippets.

Summary of RQ2:
We detect a total of 31 CWE types in 740 vulnerable code snippets (1221 CWE instances in total). These CWE types were generalized into 8 categories, and we find that evaluation, memory, and initialization weaknesses are the most common. Across all categories, vulnerabilities are more frequently introduced before revisions are made to the snippet.

RQ 3 Does the type of vulnerabilities differ depending on revision types?
4.3.1 Motivation. Users revise their answers for a variety of reasons. We aim to better understand the behaviour of these users by analyzing the types of revisions that are more commonly made when editing code snippets.

Approach.
We manually analyze code snippets that have at least one code revision by looking at the nature of the edit. Then following work by Wang et al. [41], we label each revision with four revision types: code correction, code formatting, code improvement, and code removal/addition.

4.3.3
Results. An overview of the 4 revision types we label the revised code snippets as is shown in Table 6. Overall we find that 142 out of the 740 vulnerable code snippets are revised, as shown in Figure 7. Out of the 598 vulnerable code snippets that did not have any code revisions, 327 originate from posts that were never revised, meaning the user did not make any changes to the original version of the post. The remaining 271 code snippets were not revised themselves, but belong to posts that were only textually edited. For example, users edit posts to correct typos, clarify their work, or to include references and links to other resources. For the 142 code snippets that were revised, we observe that users more frequently make revisions with the intention to improve the functionality of the example to better answer the question (86 out of 142 snippets, 60.6%), and to make corrections in the code such as fixing syntax errors and typos (31 out of 142, 21.8%). As shown in Figure 8, the least common type of edit we observe in the code snippets with revisions is formatting changes, such has adding white space through indentation (9 out of 142 code snippets).

Figure 7: Number of code snippets revised and not revised
We then further analyze the relationship between the CWE type present in a revised code snippet and it's revision type. Table 7 shows that across the 6/8 weakness categories that contain vulnerable code snippets that were revised, the most common revision type is code improvement. We also find that code correction revisions are more common in snippets with evaluation and initialization weaknesses (3 and 4 snippets respectively), and code removal/addition We then further analyze the relationship between the CWE type present in a revised code snippet and its revision type. Table 7 shows Code Improve-ment IP Functionality or performance changes to the code, includes changes to logical expressions, calculations, and the types of variables or return values.
Code Removal / Addition RA Removing or adding code segments, for example adding a new function or class.

Summary of RQ3:
We observe that out of the 142 code snippets that are revised, the most common types of revisions made are code improvements and code corrections. When looking at the relationship between CWE categories and revision types, we find the for the majority of CWE categories (6/8), code improvement revisions are the most common.

Approach.
Using the version history of the obtained code snippets, we first determine the number of code revisions for each code snippet. For the snippets that have at least one revision, we run all versions through cppcheck to determine if the number of vulnerabilities stayed the same, decreased, or increased.

Results
. We look at whether certain revision types are correlated more closely with particular revision outcomes, such as reducing the number of vulnerabilities in the snippet or introducing new ones. As shown in Fig 9, for the 40 revised code snippets that contained correction and formatting revisions, all did not experience any change in vulnerabilities. We further find that all 9 code snippets that were improved due to revisions were revised with the intent to improve the code, such as Listing 3. However, a large number of code snippets with improvement related revisions also experienced deterioration or no change in the number of vulnerabilities (23 and 54 code snippets respectively). Finally, we notice that code snippets with revisions related to the removal or addition of large segments of code were more likely to deteriorate (12 out of 16 snippets). This is likely due to the fact that the addition of large code segments increases the chance a weakness will be introduced. For example, in the following Stack Overflow answer (52560467), the user revised the snippet by adding the check_relay function and the large switch statement in the enable_relay function. Although these changes may have been made with the intention to better answer the question, they ultimately introduced an instance of CWE 398 in line 35 due to the scope of the variable state. Overall, only a small number of code snippets experienced an improvement due to revisions (9 in total). As shown in Figure 10, those with initialization related weaknesses and instances of CWE 398 contained the most (2 out of 59, and 5 out of 520 respectively).   Figure 6 Code Improve-ment IP Functionality or performance changes to the code, includes changes to logical expressions, calculations, and the types of variables or return values.
Code Removal / Addition RA Removing or adding code segments, for example adding a new function or class.

Summary of RQ3:
We observe that out of the 142 code snippets that are revised, the most common types of revisions made are code improvements and code corrections. When looking at the relationship between CWE categories and revision types, we find the for the majority of CWE categories (6/8), code improvement revisions are the most common.

Approach.
Using the version history of the obtained code snippets, we first determine the number of code revisions for each code snippet. For the snippets that have at least one revision, we run all versions through cppcheck to determine if the number of vulnerabilities stayed the same, decreased, or increased.

Results
. We look at whether certain revision types are correlated more closely with particular revision outcomes, such as reducing the number of vulnerabilities in the snippet or introducing new ones. As shown in Fig 9, for the 40 revised code snippets that contained correction and formatting revisions, all did not experience any change in vulnerabilities. We further find that all 9 code snippets that were improved due to revisions were revised with the intent to improve the code, such as Listing 3. However, a large number of code snippets with improvement related revisions also experienced deterioration or no change in the number of vulnerabilities (23 and 54 code snippets respectively). Finally, we notice that code snippets with revisions related to the removal or addition of large segments of code were more likely to deteriorate (12 out of 16 snippets). This is likely due to the fact that the addition of large code segments increases the chance a weakness will be introduced. For example, in the following Stack Overflow answer (52560467), the user revised the snippet by adding the check_relay function and the large switch statement in the enable_relay function. Although these changes may have been made with the intention to better answer the question, they ultimately introduced an instance of CWE 398 in line 35 due to the scope of the variable state. Overall, only a small number of code snippets experienced an improvement due to revisions (9 in total). As shown in Figure 10, those with initialization related weaknesses and instances of CWE 398 contained the most (2 out of 59, and 5 out of 520 respectively). that across the 6/8 weakness categories that contain vulnerable code snippets that were revised, the most common revision type is code improvement. We also find that code correction revisions are more common in snippets with evaluation and initialization weaknesses (3 and 4 snippets respectively), and code removal/addition revisions are more common in snippets with memory weaknesses (5 snippets). Overall, there is a statistically significant difference between the number of vulnerable code snippets and the number of revised vulnerable code snippets across the six categories (Mann Whitney U test = 0.02).

Code Correc-tion CR
Changes to the syntax, and typo changes in comments and variable/function names.
Code Formatting FM Adding white space or newlines, improving the formatting/readability of the code.
Code Improvement IP Functionality or performance changes to the code, includes changes to logical expressions, calculations, and the types of variables or return values.
Code Removal / Addition RA Removing or adding code segments, for example adding a new function or class.

Summary of RQ3:
We observe that out of the 142 code snippets that are revised, the most common types of revisions made are code improvements and code corrections. When looking at the relationship between CWE categories and revision types, we find the for the majority of CWE categories (6/8), code improvement revisions are the most common.
improve code snippets by fixing errors and reducing vulnerabilities, or they can further deteriorate an already vulnerable snippet by adding new weaknesses. Revisions may also leave vulnerabilities unchanged. It is thus important to better understand the impact of code revisions on reducing vulnerabilities.

Approach.
Using the version history of the obtained code snippets, we first determine the number of code revisions for each code snippet. For the snippets that have at least one revision, we run all versions through cppcheck to determine if the number of vulnerabilities stayed the same, decreased, or increased.

Results
. We look at whether certain revision types are correlated more closely with particular revision outcomes, such as reducing the number of vulnerabilities in the snippet or introducing new ones. As shown in Fig 9, for the 40 revised code snippets that contained correction and formatting revisions, all did not experience any change in vulnerabilities. We further find that all 9 code snippets that were improved due to revisions were revised with the intent to improve the code, such as Listing 3. However, a large number of code snippets with improvement related revisions also experienced deterioration or no change in the number of vulnerabilities (23 and 54 code snippets respectively). Finally, we notice that code snippets with revisions related to the removal or addition of large segments of code were more likely to deteriorate (12 out of 16 snippets). This is likely due to the fact that the addition of large code segments increases the chance a weakness will be introduced. For example, in the following Stack Overflow answer (52560467), the user revised the snippet by adding the check_relay function and the large switch statement in the enable_relay function. Although these changes may have been made with the intention to better answer the question, they ultimately introduced an instance of CWE 398 in line 35 due to the scope of the variable state. Overall, only a small number of code snippets experienced an improvement due to revisions (9 in total). As shown in Figure 10, those with initialization related weaknesses and instances of CWE 398 contained the most (2 out of 59, and 5 out of 520 respectively).

Results
. We look at whether certain revision types are correlated more closely with particular revision outcomes, such as reducing the number of vulnerabilities in the snippet or introducing new ones. As shown in Fig 9, for the 40 revised code snippets that contained correction and formatting revisions, all did not experience any change in vulnerabilities. We further find that all 9 code snippets that were improved due to revisions were revised with the intent to improve the code, such as Listing 3. However, a large number of code snippets with improvement related revisions also experienced deterioration or no change in the number of vulnerabilities (23 and 54 code snippets respectively). Finally, we notice that code snippets with revisions related to the removal or addition of large segments of code were more likely to deteriorate (12 out of 16 snippets). This is likely due to the fact that the addition of large code segments increases the chance of introducing a weakness. For example, in the following Stack Overflow answer ( 52560467 ), the user revised the snippet by adding the check_relay function and the large switch statement in the enable_relay function. Although these changes may have been made with the intention of better answering the question, they ultimately introduced an instance of CWE 398 in line 35 due to the scope of the variable state. Overall, only a small number of code snippets experienced an improvement due to revisions (9 in total). As shown in Figure 10, those with initialization related weaknesses and instances of CWE 398 contained the most (2 out of 59, and 5 out of 520 respectively).
In total, 115 code snippets with pre-existing vulnerabilities experienced one or more revisions. In Figure 11, we observe that the majority of all revised code snippets (in total 142) were revised just once. However, we also observe that the effect of revisions on these code snippets is minimal. Figure 12 shows the vast majority of these code snippets with pre-existing vulnerabilities that were revised did not experience a decrease or increase in vulnerabilities (98 out of 115). We also find that in some code snippets (9 out of 115), revisions removed vulnerabilities that existed in previous versions. However, in a similar number of code snippets (8 out of 115), vulnerabilities were introduced by revisions.  Figure 10: Distribution of code snippets that were improved due to revisions by CWE type or CWE category In total, 115 code snippets with pre existing vulnerabilities experienced one or more revisions. In Figure 11, we observe that the majority of all revised code snippets (in total 142) most were revised just once. However, we also observe that the effect of revisions on these code snippets is minimal. Figure 12 shows the vast majority of these code snippets with pre-existing vulnerabilities that were revised did not experience a decrease or increase in vulnerabilities (98 out of 115). We also find that in some code snippets (9 out of 115), revisions removed vulnerabilities that existed in previous versions. However, in a similar number of code snippets (8 out of 115), vulnerabilities were introduced by revisions. Stack Overflow answer 25827521 is an example of a snippet that was revised once but did not improve in terms of removing it's weakness. Comparing the final version of the code to the original, we observe that the user likely intended to improve the functionality and correctness of the code. The revisions they made were changing the function parameters to void instead of leaving it empty to ensure no arguments can be passed, and also attempting error handling by first checking if read_AT_string did not return NULL. However, they did not address the instance of CWE 562 -Return of Stack Variable Address detected by cppheck in line 36 where the return value was a pointer to a local variable, which is invalid in c++. This is an example of a revision that went beyond non-functional changes such as adding comments or fixing typos, but was still not effective at removing the weakness in the snippet. We analyze the 31 deteriorated code snippets for similarities in the changes made to the code, and the subsequent CWE instances that were introduced. In total we find 4 distinct deterioration types : (1) New component added + introduced new weakness, (2) Modified existing component + introduced new weakness, (3) New component added + introduced more existing weaknesses, and (4) Modified existing component + introduced more existing weaknesses. We further find that a few deteriorations occurred due the a user fixing a syntax error in the code, which allowed for more weaknesses to be detected by cppcheck (5). The distribution of these 5 types is shown in Figure 13, where we see that the most common type of deterioration was caused by users modifying an existing component in the code (such as a function, variable, or class), and introduced new CWE instances that were not present before.
Summary of RQ4: The vast majority of revised snippets remained unchanged in terms of the number of CWE instances present in the code. When looking at the effect of each revision type, we find that all revised related to code correction and formatting left the code snippet unchanged, while a large proportion of code removal/addition type revisions deteriorated.

DISCUSSION
Our study findings can guide the following stakeholders: IoT developers, IoT vendors, Forum designers, and IoT researchers and educators. We discuss the implications below.
IoT Developers: In Figure 14, we show the number of CVEs reported in the NVD database per the six vulnerability types we observed in our IoT code snippets (in pre + post revisions). The reported CVEs are grouped under eight types in the cvedetails.com site: (1) Denial of service, (2) Overflow, (3) Code execution, (4) Memory corruption, (5) Cross-site scripting, (6) Gain privilege, (7) Gain information, and (8) Bypass. This Figure 14 shows that the six vulnerability types in the IoT code snippets can cause all the eight types of critical security issues in the IoT devices, if the vulnerable code snippets are reused from the Stack Exchange sites. Therefore, IoT developers need to stay aware of the potential security problems in such shared code snippets in the developer forums. Our study findings like the catalog of CWEs and the associated vulnerable code snippets can be used by the IoT developers to learn about Figure 12: Distribution of revision effect in snippets with pre-existing vulnerabilities Stack Overflow answer 25827521 is an example of a snippet that was revised once but did not improve in terms of removing its weakness. By comparing the final version of the code to the original, we observe that the user likely intended to improve the functionality and correctness of the code. We see they changed the function parameters to void instead of leaving it empty to ensure no arguments could be passed, and they also attempted error handling by first checking if read_AT_string did not return NULL. However, they did not address the instance of CWE 562 -Return of Stack Variable Address detected by cppheck in line 36 where the return value was a pointer to a local variable, which is invalid in C++. This is an example of a revision that went beyond non-functional changes such as adding comments or fixing typos, but was still not effective at removing the weakness in the snippet.
We analyze the 31 deteriorated code snippets for similarities in the changes made to the code, and the subsequent CWE instances that were introduced. In total, we find 4 distinct deterioration types : (1) New component added + introduced new weakness, (2) Modified existing component + introduced new weakness, (3) New component added + introduced more existing weaknesses, and (4) Modified existing component + introduced more existing weaknesses. We further find that a few deteriorations occurred due to the user fixing a syntax error in the code, which allowed for more weaknesses to be detected by cppcheck (5). The distribution of these 5 types is shown in Figure 13, where we see that the most common type of deterioration was caused by users modifying an existing  Figure 10: Distribution of code snippets that were improved due to revisions by CWE type or CWE category In total, 115 code snippets with pre existing vulnerabilities experienced one or more revisions. In Figure 11, we observe that the majority of all revised code snippets (in total 142) most were revised just once. However, we also observe that the effect of revisions on these code snippets is minimal. Figure 12 shows the vast majority of these code snippets with pre-existing vulnerabilities that were revised did not experience a decrease or increase in vulnerabilities (98 out of 115). We also find that in some code snippets (9 out of 115), revisions removed vulnerabilities that existed in previous versions. However, in a similar number of code snippets (8 out of 115), vulnerabilities were introduced by revisions. Stack Overflow answer 25827521 is an example of a snippet that was revised once but did not improve in terms of removing it's weakness. Comparing the final version of the code to the original, we observe that the user likely intended to improve the functionality and correctness of the code. The revisions they made were changing the function parameters to void instead of leaving it empty to ensure no arguments can be passed, and also attempting error handling by first checking if read_AT_string did not return NULL. However, they did not address the instance of CWE 562 -Return of Stack Variable Address detected by cppheck in line 36 where the return value was a pointer to a local variable, which is invalid in c++. This is an example of a revision that went beyond non-functional changes such as adding comments or fixing typos, but was still not effective at removing the weakness in the snippet. We analyze the 31 deteriorated code snippets for similarities in the changes made to the code, and the subsequent CWE instances that were introduced. In total we find 4 distinct deterioration types : (1) New component added + introduced new weakness, (2) Modified existing component + introduced new weakness, (3) New component added + introduced more existing weaknesses, and (4) Modified existing component + introduced more existing weaknesses. We further find that a few deteriorations occurred due the a user fixing a syntax error in the code, which allowed for more weaknesses to be detected by cppcheck (5). The distribution of these 5 types is shown in Figure 13, where we see that the most common type of deterioration was caused by users modifying an existing component in the code (such as a function, variable, or class), and introduced new CWE instances that were not present before.
Summary of RQ4: The vast majority of revised snippets remained unchanged in terms of the number of CWE instances present in the code. When looking at the effect of each revision type, we find that all revised related to code correction and formatting left the code snippet unchanged, while a large proportion of code removal/addition type revisions deteriorated.

DISCUSSION
Our study findings can guide the following stakeholders: IoT developers, IoT vendors, Forum designers, and IoT researchers and educators. We discuss the implications below.
IoT Developers: In Figure 14, we show the number of CVEs reported in the NVD database per the six vulnerability types we observed in our IoT code snippets (in pre + post revisions). The reported CVEs are grouped under eight types in the cvedetails.com site: (1) Denial of service, (2) Overflow, (3) Code execution, (4) Memory corruption, (5) Cross-site scripting, (6) Gain privilege, (7) Gain information, and (8) Bypass. This Figure 14 shows that the six vulnerability types in the IoT code snippets can cause all the eight types of critical security issues in the IoT devices, if the vulnerable code snippets are reused from the Stack Exchange sites. Therefore, IoT developers need to stay aware of the potential security problems in such shared code snippets in the developer forums. Our study findings like the catalog of CWEs and the associated vulnerable code snippets can be used by the IoT developers to learn about   Figure 10: Distribution of code snippets that were improved due to revisions by CWE type or CWE category In total, 115 code snippets with pre existing vulnerabilities experienced one or more revisions. In Figure 11, we observe that the majority of all revised code snippets (in total 142) most were revised just once. However, we also observe that the effect of revisions on these code snippets is minimal. Figure 12 shows the vast majority of these code snippets with pre-existing vulnerabilities that were revised did not experience a decrease or increase in vulnerabilities (98 out of 115). We also find that in some code snippets (9 out of 115), revisions removed vulnerabilities that existed in previous versions. However, in a similar number of code snippets (8 out of 115), vulnerabilities were introduced by revisions. Stack Overflow answer 25827521 is an example of a snippet that was revised once but did not improve in terms of removing it's weakness. Comparing the final version of the code to the original, we observe that the user likely intended to improve the functionality and correctness of the code. The revisions they made were changing the function parameters to void instead of leaving it empty to ensure no arguments can be passed, and also attempting error handling by first checking if read_AT_string did not return NULL. However, they did not address the instance of CWE 562 -Return of Stack Variable Address detected by cppheck in line 36 where the return value was a pointer to a local variable, which is invalid in c++. This is an example of a revision that went beyond non-functional changes such as adding comments or fixing typos, but was still not effective at removing the weakness in the snippet. We analyze the 31 deteriorated code snippets for similarities in the changes made to the code, and the subsequent CWE instances that were introduced. In total we find 4 distinct deterioration types : (1) New component added + introduced new weakness, (2) Modified existing component + introduced new weakness, (3) New component added + introduced more existing weaknesses, and (4) Modified existing component + introduced more existing weaknesses. We further find that a few deteriorations occurred due the a user fixing a syntax error in the code, which allowed for more weaknesses to be detected by cppcheck (5). The distribution of these 5 types is shown in Figure 13, where we see that the most common type of deterioration was caused by users modifying an existing component in the code (such as a function, variable, or class), and introduced new CWE instances that were not present before.
Summary of RQ4: The vast majority of revised snippets remained unchanged in terms of the number of CWE instances present in the code. When looking at the effect of each revision type, we find that all revised related to code correction and formatting left the code snippet unchanged, while a large proportion of code removal/addition type revisions deteriorated.

DISCUSSION
Our study findings can guide the following stakeholders: IoT developers, IoT vendors, Forum designers, and IoT researchers and educators. We discuss the implications below.
IoT Developers: In Figure 14, we show the number of CVEs reported in the NVD database per the six vulnerability types we observed in our IoT code snippets (in pre + post revisions). The reported CVEs are grouped under eight types in the cvedetails.com site: (1) Denial of service, (2) Overflow, (3) Code execution, (4) Memory corruption, (5) Cross-site scripting, (6) Gain privilege, (7) Gain information, and (8) Bypass. This Figure 14 shows that the six vulnerability types in the IoT code snippets can cause all the eight types of critical security issues in the IoT devices, if the vulnerable code snippets are reused from the Stack Exchange sites. Therefore, IoT developers need to stay aware of the potential security problems in such shared code snippets in the developer forums. Our study findings like the catalog of CWEs and the associated vulnerable code snippets can be used by the IoT developers to learn about

DISCUSSION
Our study findings can guide the following stakeholders: IoT developers, IoT vendors, Forum designers, and IoT researchers and educators. We discuss the implications below. IoT Developers: In Figure 14, we show the number of CVEs reported in the NVD database per the six vulnerability types we observed in our IoT code snippets (in pre + post revisions). The reported CVEs are grouped under eight types in the cvedetails.com site: (1) Denial of service, (2) Overflow, (3) Code execution, (4) Memory corruption, (5) Cross-site scripting, (6) Gain privilege, (7) Gain information, and (8) Bypass. This Figure 14 shows that the six vulnerability types in the IoT code snippets can cause all the eight types of critical security issues in the IoT devices, if the vulnerable code snippets are reused from the Stack Exchange sites. Therefore, IoT developers need to stay aware of the potential security problems in such shared code snippets in the developer forums. Our study findings like the catalog of CWEs and the associated vulnerable code snippets can be used by the IoT developers to learn about such pitfalls. Such knowledge can be useful for the developers in multiple phases. First, when they share any IoT code examples in the forums. Second, when they revise a shared code example. Third, when they reuse a shared code example. In all phases, they can check the code for vulnerability by consulting our catalog of CWEs. Given that revisions from others rarely fix such a vulnerable code snippets, it is important for the IoT developers to practice such quality assurance of the code snippets, even after the code snippet is revised by other users in the forums. IoT Vendors: In Figure 14, we showed that the six vulnerability type we observed in the shared IoT code snippets can be mapped to eight CVE types in NVD database. In the NVD database, each reported CVE is categorized into the severity types: Critical (C), High (H), Medium (M), and Low (L). In Table 8, we show the distribution of the CVEs by four severity types. Overall, we see that all six vulnerability categories in our observed IoT code snippets can contribute to many critical and highly severe security vulnerabilities in the IoT devices. In Figure 5, we showed that such affected IoT devices can belong to many vendors (e.g., Cisco, Snapdragon, etc.). Therefore, IoT device and SDK vendors can work together to make the devices more resilient. One solution would be to incorporate automated security testing tool into the IoT SDKs and devices. The IoT vendors cannot rely much on the collaborative editing system in the crowd-sourced developer forums to improve the code snippets. . Width Es.     Table 9 can inform forum designers of the fact that the current mechanisms (e.g., the four revision types) to support collaborative editing in the online forums are insufficient to help mitigate security vulnerabilities in the shared code. In Figure 15, we reinforce this observation by showing a bubble chart as a tradeoff between the number of revisions made to our studied vulnerable code snippets and the number of distinct CVEs types that were introduced during the revisions. The size of each bubble corresponds to the number of total CVEs reported against the CWE IDs found per CWE category. Figure 15 shows that the most number of CVEs reported for the 'Memory' category weaknesses, and code snippets with such weaknesses in our dataset were revised considerably  that correction to a code should have fixed more vulnerability. The statistics from Table 9 can inform forum designers of the fact that the current mechanisms (e.g., the four revision types) to support collaborative editing in the online forums are insufficient to help mitigate security vulnerabilities in the shared code. In Figure 15, we reinforce this observation by showing a bubble chart as a tradeoff between the number of revisions made to our studied vulnerable code snippets and the number of distinct CVEs types that were introduced during the revisions. The size of each bubble corresponds to the number of total CVEs reported against the CWE IDs found per CWE category. Figure 15 shows that the most number of CVEs reported for the 'Memory' category weaknesses, and code snippets with such weaknesses in our dataset were revised considerably with little or no success in fixing. On the other hand, code snippets belonging to 'Conversion' type weaknesses were revised almost never. Overall, there is a positive correlation between the number of revisions made to a code snippet and the number of vulnerabilities found in the code snippet. The findings call for a redesign in the collaborative editing process in the forums by the designers of the sites, e.g., to facilitate the incorporation of security guidelines into the editing process by providing automated recommendations.

opers in
IoT Researchers and Educators: Our findings offer a grim picture on the effectiveness of collaborative editing to help mitigate security vulnerabilities in developer forums. IoT researchers can join hands with both the forum designers and the IoT vendors to conduct research on the better design of the collaborative editing and to incorporate security validation framework into the IoT devices. Our findings reinforce the recent worries on software supply chain attacks [27] by showing that security vulnerabilities are prevalent in online developer forums and they are mostly left unaddressed during revisions. One way to help IoT developers during the sharing of code snippets in online forums is to educate them with on-demand documentation about the security issues by consulting security patterns and the vulnerabilities reported in the NVD database. The IoT security educators can join hands with the forum designers to produce such documentation, which can also offer new directions to the current approaches that utilize online developer forums to create and/or improve software documentation [8,[34][35][36][37][38]. This is important given studies that IoT developers do indeed consult about security issues in online developer forums [33], but they also face difficulty to get answers to their questions [39].

THREATS TO VALIDITY
Internal validity threats relate to author bias in deciding which weaknesses to ignore. We noticed that some claimed weaknesses in cppcheck were not accurate. We addressed this issue by suppressing certain CWE types in Cppcheck like CWE 563 (Assignment to Variable without Use) and by following a previous study [42] to suppress syntax errors and to ignore code snippets with less than 5 lines. In addition, the categorization of the 28 distinct CWE types into 8 weakness categories was done by both authors. Construct validity threats relate to errors that may have occurred during the data collection. To determine if a SO post was related to IoT, we used tags from existing studies [33,39]. For Arduino and Raspberry, we made the assumption that all posts would be related to IoT. Another threat is our use of the language detection tool guesslang to identify C/C++ code snippets. Guesslang has been used in previous studies to specifically detect C/C++ code and has a validity rate of 90% [14]. External validity threats relate to how our findings can be generalized to the nature of IoT posts on online Q/A sites as a whole. We focused on vulnerabilities in Stack Exchange answers. This is because code snippets found in answers are meant to be 'solutions' and are more likely to be copied and used by developers.
We also focus our study on strictly C and C++ code snippets due to their popularity in IoT development.

RELATED WORK
To the best of our understanding, the C/C++ code examples shared in Stack Overflow were subject to two empirical studies recently, first by Verdi et al. [40] and then by Zhang et al. [42]. Our study differs from the two studies as follows.

CONCLUSION
We analyzed code examples from the Stack Overflow, Arduino, and Raspberry Stack Exchange sites. We focused on analyzing weaknesses by analyzing their revision history. We found a total of 31 CWE types present in 740 code snippets. We observed that the vast majority of vulnerabilities are introduced pre code revisions (713 out of 740). When snippets are revised, the number of vulnerabilities that are present in that snippets are likely to not change.
Our results indicate the collaborative editing in the forums do not help mitigate the code vulnerabilities. Our future work will focus developing techniques to incorporate security recommendations into the collaborative editing process.