{'fraction_non_alphanumeric': 0.027052957737617768, 'fraction_numerical': 0.005111750735305683, 'mean_word_length': 5.091788828207339, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 1, 'zh_pornsignals.json': 0}}
Challenging software developers: dialectic as a foundation for security assurance techniques

Development teams are increasingly expected to deliver secure code, but how can they best achieve this? Traditional security practice, which emphasizes ‘telling developers what to do’ using checklists, processes and errors to avoid, has proved difﬁcult to introduce. From analysis of industry interviews with a dozen experts in app development security, we ﬁnd that secure development requires ‘dialectic’: a challenging dialog between the developers and a range of counterparties, continued throughout the development cycle. Analysing a further survey of 16 industry developer security advocates, we identify the six assurance techniques that are most effective at achieving this dialectic in existing development teams, and conclude that the introduction of these techniques is best driven by the developers themselves. Concentrating on these six assurance techniques, and the dialectical interactions they involve, has the potential to increase the security of development activities and thus improve software security for everyone.


Introduction
Software security and privacy are becoming major issues: almost every week we hear that yet another organization's software systems have been compromised [1]. While there are many aspects to an organization's security and privacy, the software used clearly has a significant impact on whether such breaches happen. Therefore, the effectiveness of developers at creating secure software is vital. 1 Unfortunately, there is evidence that the developers are not delivering this security. A recent report from Veracode [2] concluded that 'more than 85 percent of all applications have at least one vulnerability in them; more than 13 percent of applications have at least one very high severity flaw'. A report from Microsoft [3] analysed storage and collaboration Software as a Service applications, and found a wide range of issues, including 28% of storage apps not supporting data encryption. The Ponemon Institute carried out a IBM-funded survey in 2015 of 640 individuals from organizations developing mobile apps in the USA [4], and found that 77% believed that securing mobile apps was 'very hard', and that 73% believed that developer lack of understanding of security issues was a major contributor to the problem.
All these surveys suggest that many errors were avoidable; developers could have made choices that would have prevented the issues. These studies demonstrate that existing industry practices are insufficient to provide the application security and privacy we need.
To address this problem, we could have looked for improvements to environments and APIs; we could have looked at tools to automate security improvements; or we could look at ways in which we can help programmers themselves to improve security given existing constraints. All are valuable approaches; we chose the third option. In this article, therefore, we investigate programmers' security practices, using expert surveys to find out what works in practice.

First phase: App security expert interviews
Faced with a broad range of diverse kinds of developer, we initially limited our scope to mobile app development. The initial research question we formulated during the work was therefore: What techniques and ideas lead to the development of better secure app software?
To answer this question, we needed an understanding of what works in the real world. We therefore conducted a Constructivist Grounded Theory [5] study, involving face-to-face interviews with a dozen experts whose cumulative experience totalled well over 100 years of secure app development. Our early analysis of the interviews [6,7] found a wide range of difference between interviewees, and concluded that there was little consensus in the industry.
Further analysis identified that the most important and successful secure development techniques share a quality we call 'dialectic', meaning learning by challenging. Dialectic techniques use dialog with a range of counterparties to achieve security in an effective and economical way. The increase in security comes from the developers' continued interaction with the resulting challenges, not from passive learning. This suggests that we need developers to actively seek out arguments and challenges in order to improve software security.

Second phase: Intervention expert interviews
Given the suggestion that dialectic techniques increase security, we wanted to find ways to introduce such techniques to development teams who were not yet using them. We observed that none of the techniques, nor the dialectic concept, appeared specific to mobile app development. Indeed, though we had chosen the first survey's participants for their experience with app security, in discussing the questions most had cited experience with a far wider range of software domains. We therefore conducted a second survey to find ways to induce teams to adopt dialectic techniques, extending our scope to cover professional software development in general.
Our research question for this next step, therefore, was: What interventions can change the environment for members of a development team to achieve good security?
In addressing this question, we considered motivational factors, choice of tools, supporting processes and potential blockers, culture, awareness, training and skills. We observed that previous research in software security has concentrated on the negative: identifying vulnerabilities and making checklists of issues to avoid (see the 'Resources to help programmers' section). Yet developers are motivated by positive aspects of their work such as building and helping others [8], so we chose instead to concentrate on positive approaches to improving security.
Using the same Grounded Theory approach as the first phase, we accordingly interviewed sixteen specialists in developer software security, asking them about these topics. These specialists ranged from senior experts in the major multinational online service providers to solo consultants.
From our interviews, we identified a consistent perception of the developer as an active agent in their own right, whose decisions could be influenced by security experts but not controlled by them. More specifically, we identified six assurance techniques preferred by the experts, and a range of counterparties with whom the dialectic takes part. We analyse these in detail in this article, highlighting the dialectic effect on security.

Contribution and overview
The contribution of this work is a new perspective upon the introduction and improvement of security in software development teams. This article provides: • A theory, 'Dialectical Security,' to provide a basis for analysis of developer security techniques; • Identification of six assurance techniques preferred by industry experts; • Analysis of each of these assurance techniques in terms of its dialectic counterparties; and • Examples and approaches for introducing the techniques.
In the rest of this article, 'Background and related work' explores existing work; 'Methodology' explains the methodology in more detail and describes the research participants; 'Analysis' describes the results of the surveys, introducing dialectic and analysing how it applies in each of six assurance techniques; 'Discussion' examines the research as a whole; 'Threats to validity' explores validity issues; and 'Conclusion' summarises the article and suggests approaches for future work.

Background and related work
We looked for related work in three areas: • How and why programmers learn security, • Resources to help programmers improve security, and • Techniques to help teams improve

How programmers learn security
There is relatively little literature on how programmers learn, whether about novices or professional programmers. Johnson and Senges [9] studied how programmers learned to function in a complicated organization, Google. They concluded that the majority of programmer learning there was peer learning, facilitated by strong corporate standards and culture; obviously, the results are limited to that organization. Enes and Conradi [10] used interviews to discover how professionals, including a few programmers, acquired their expert knowledge. It concludes that the preferred learning mechanisms are all informal ones: especially on-the job training and personal interaction. Murphy-Hill et al. explored how developers find new software tools [11]; the research was more wide-ranging and included both an initial survey and a larger scale, 79 participant, diary-based survey. They concluded the event is relatively rare, and happens more through joint working than recommendations.
Balebako et al. surveyed and interviewed over 200 app developers, and concluded that most approached security issues using web search, or by consulting peers [12]. A survey by Acar et al. reached similar conclusions; and they also determined experimentally the surprising result that programmers using digital books achieved better security than those using web search [13]. Yskout et al. tested experimentally the effect of using security patterns in server design; the results suggested a benefit but were statistically inconclusive [14].
Xie et al. [15], interviewed programmers to investigate why they believed they made security errors; they found a consistent tendency to treat security as 'someone else's problem'. Though the results are limited in scope to highly experienced developers in US companies, it seems reasonable to conclude that the conclusions may apply more widely.
Votipka et al. [16] compared the approaches of professional penetration testers with those of developer testers, concluding that their approaches are similar; the experience of the pen testers leads to better security findings; but the closeness of developer testers to programmers can lead to better outcomes. They therefore recommended that penetration testers work directly with programmers.

Resources to help programmers
Turning to resources, one might expect that an effective contribution to app security would be books explaining how to do secure development. The recently-published book, Bell et al.'s Agile Application Security [17] does this, explaining security techniques to application development, and agile and programming concepts to security experts, and providing a good deal of practical information, though it is not yet widely read. 2 Few other resources seem to be helpful to developers at a higher level than code. There are many good works describing the theory and practice of software security, such as Gollman's 'Computer Security' [18], Schneier's 'Secrets and Lies' [19] and Anderson's 'Security Engineering' [20]; these are particularly valuable for introducing the concepts of 'whole system security', but all work at a level that isn't helpful as anything but background reference for a software developer.
Instead, the most useful learning books for software developers tend to be those that convey information in a relatively terse and readable form, and in manageable chunks. A good example is Howard et al.'s '24 Deadly Sins of Software Security' [21]. An online version is the community-written OWASP Top Ten Risks sites [22], a widely accessed 3 resource detailing specific programming issues and how to avoid them, which has versions for a variety of platforms. Their authority and availability make them effective, though they consider few 'whole system security' issues. Acar et al. [23] surveyed a range of web-based software security resources available for developers, concluding that there are distinct gaps in the coverage, and that not much of it was suitable for developer learning.
Some of the most popular 4 security books are the ones with a platform-specific 'Black Hat' (attacker) approach. For example the Android Hacker's Handbook [29], and its corresponding versions for iOS and web apps, contain a good deal about exploits against the operating system, a certain amount about analysing existing software, but little about how to guard against exploits as a developer.
Most programmers tend to use web search and discussion sites such as Stack Overflow as their primary source of information on security [13]. Unfortunately, these lack overview discussions [30], making them valuable in helping programmers sort out problems they know they have, but does not point out problems that they do not know they may have; most security problems are likely to be of this second type.
Discussion sites have a second problem related to security: their answers on security matters tend to be of questionable accuracy especially when they quote code. Acar et al. [13] analysed answers on Stack Overflow to app security questions, worryingly finding around 50% of solutions to a set of security questions to contain insecure code snippets.

Techniques to help teams improve
One approach to helping teams with security is providing code analysis tools: seven research groups [31][32][33][34][35][36][37] have created security defect detection tools to help developers improve code, using feedback via IDEs or elsewhere. Xie et al. [38] explored the impact of their IDE-based security analysis tool for web applications on a sample of 21 students and 6 professionals and found: Even when creating secure code is relatively easy . . . users still need to be motivated to make the needed changes.
One might expect the most effective approach to be a prescriptive set of instructions telling programmers what to do, a 'Secure Software Development Lifecycle' (SSDL) such as those promoted by Microsoft [39] and others. However, Conradi and Dyba [40], among others, identified that programmers have difficulty with, and resist learning from, formal written routines, and this appears to have been the experience with SSDLs in practice [41]. So, since about 2010 several of the SSDLs have been replaced by 'Security Capability Maturity Models' [42,43], to allow management influence on software security at a corporate level based on measurements using checklists of processes used to improve security. These are effective [44] at defining what development teams should achieve, but provide little help to developers on how best to achieve it.
Some research teams have explored the impact of training and external involvement on teams' delivery of secure software. Tü rpe et al. [45] explored ethnographically the effect of a penetration testing session and workshop on 37 members of a large geographically dispersed project, concluding that it failed mainly because the workshop consultant highlighted problems without offering solutions. Poller et al. [46] followed an attempt to improve long-term security practices in an agile development team of about 15 people, which failed because pressure to add functionality meant that attention was not given to security issues, and because their normal work procedures did not support goals such as security improvement. The authors concluded that successful interventions would need 'to investigate the potential business value of security, thus making it a more tangible development goal'; and that security is best promoted as a team, not individual, effort.
Work by Ashenden and Lawrence [47] took a different approach. They used an Action Research method to investigate and improve the relationships between security professionals and software developers in a single company, and found the approach effective in improving communication. Lopez et al. [48] used discussion workshops based on ethnographic studies to encourage similar communication. No evidence is yet available of longer-term impact from either intervention.
Another approach is to investigate the economics of different software assurance techniques. Such et al. [49] surveyed 150 security specialists to analyse the economics of different software assurance techniques related to security. The survey asked about twenty predefined assurance techniques, finding Public Review and (tool-based) 2 Based on Amazon.com rankings as at January 2018 3 Based on Google rankings in February 2016 4 Based on Amazon.com rankings as at January 2016 Static Analysis to be the most cost-effective, and Formal Verification and Cryptographic Validation to be the least. Interestingly, the researchers also identified frequently-used combinations of techniques, finding that the combination Architectural Review, Configuration Review, (manual) Penetration Test and (automated, web-based) Vulnerability Scan was seen as most cost-effective.

Limitations of existing literature
Consistent in all this literature, is a lack of knowledge and theory as to how successful industry development teams achieve software security. This article addresses that lack.

Methodology
This section explains our choice of methodology, Grounded Theory, and introduces the two sets of research participants.

Choice of methodology
Since there was little existing theory on developers and app security, we had no basis for an experimental approach; although we had a target-improving developer security-we had no theory to test. Written or email surveys could be useful to find out current practice from a list of options but are unsuitable for the kinds of open question that generates theory.
Our purpose in the research was to generate knowledge about good approaches to achieve secure development. Two perceptions drove the choice of research approach: • We had found few resources discussing how to tackle app development security. • Existing literature tended to be negative in approach, listing things the developer must not do; this contrasts with the kinds of books preferred by developers, which we had observed to be positive in outlook.
Since our aim was to generate a novel theory, we chose Grounded Theory as our primary research method. Our studies used semistructured interviews with industry experts. We chose interviewees opportunistically; our connections in industry provided introductions to a number of successful, and mostly senior, practitioners with considerable experience of helping teams achieve software security.
The first 'App Security Expert' phase of interviews took 6 months, and involved 12 such experts; the second, 'Intervention Expert' phase took a similar period, and involved 15 interviews with 16 experts. The Appendix contains the protocols used for each set of interviews.
Both studies were separately approved by the Lancaster University Faculty of Science and Technology Research Ethics committee.

Grounded theory
Grounded Theory (GT) [50] uses textual analysis of unstructured text to make theory generation into a dependable process. It requires line-by-line analysis of everything relevant that is available to the researcher: interview transcripts, relevant research literature, field notes from observation and anything else that can reduce to text form. The process is iterative, with analysis of initial findings from interviews or similar leading to changes in the research thrust and direction.
We used the Constructivist GT variant [5], acknowledging the effects of the researcher on the results. We followed the principles for software engineering GT described by Stol et al. [51]. The interviews consulted the participants as experts rather than as subjects, addressing what each had found to be most successful in their experience in secure software development. To encourage positivity, we used elements of Appreciative Inquiry [52] in our questioning: the 'Discovery' of best practice and the 'Dream' of ideal practice. Questions avoided discussion on what does not work and concentrated on the most effective practice known to each interviewee.
We recorded interviews and transcribed them manually; organization of the data used the commercial tool NVivo; coding, memo creation, and sorting were all by the lead author.
The Grounded Theory analysis involved scanning each text lineby-line, highlighting points of interest. One or more codes are then assigned to each point of interest, with new codes created as required to identify the ideas and concepts involved. Codes were reused across documents, so that a given code represents the same idea throughout the research. During that process, the coder also 'memoed' ideas and thoughts about how the terms may be interrelated. Gradually, as codes were assigned, the coder merged codes with similar meaning, and organized the resulting codes into groups called 'categories'.
The coding process for both surveys generated a total of 2140 coding entries to 340 distinct codes, grouped into 12 high-level categories.
The Grounded Theory methodology derives from social science, and in contrast to many computer science methodologies, dual coding is not a GT practice [5]. Therefore, inter-rater agreement calculations are not applicable. Instead researcher interaction takes the form of discussion of the codes, findings and outcomes, and in this case was based around the writing of papers.
In GT analysis, the aim is to find a single overarching Core Category: the category which covers the most interesting features in the data. Data gathering is considered complete when further data received does not lead to significant new concepts, a situation termed 'theoretical saturation'. The Core Category then forms a basis from which to construct theory. The 'Analysis' section presents the theory of the core category we found: Dialectical Security.
Finally, we sent a detailed report describing the findings from each survey to each of the interviewees in that survey.

Research participants
Interviewees were recruited opportunistically; connections in industry and academia provided introductions to successful, and mostly senior, practitioners with considerable experience of helping teams achieve software security. Figure 1 introduces the participants: those in the first 'App Expert' survey have IDs starting P; those in the second 'Intervention Expert' survey have IDs starting with Q. The horizontal axis indicates their organization size: Small for consultants and up to 10 people; Medium up to 1000 people, else Large or Government. The vertical axis provides a subjective estimate of the 'secure software capability maturity' [53] of the software teams they work with, as follows: To show further the range of participants involved, Fig. 2 summarizes the recruitment process. We approached various people; the Low Little or no awareness or activity related to software security Medium Aware of and addressing security issues, typically including some developers with good security knowledge. High Experts at software security, within an organizational culture that assigns it a high priority.
horizontal axis indicates how long we had known the people approached. Some we interviewed directly, and these are shown closest to the axis. Professional and personal contacts are shown above the axis, and contacts from academia-mostly encountered at industry-academic workshops-below. In other cases, the contacts we approached referred us to others, and the resulting interviewees are shown further from the axis. Most of the participants were based in the UK other than P11, Q5 based in the USA; Q7 in Sweden; and Q10, Q13 in Germany. All UK interviews were face-to-face; others used video conferencing.
Only one person was interviewed in both surveys (P6 Q2, shown together); the requirement in the second survey for a wider remit than app development discouraged other repeat interviews. Each participant was in a different organization, other than P3/P11, Q14/ Q15, and Q12/Q16-the latter two who were interviewed together. Table 1 in the 'Assurance techniques' section has the participant job roles and the company descriptions.
We chose experts in secure software development; many were therefore predominantly developers first and security experts second. As Table 1 shows, only 14 of the 27 had job titles or company missions specifically related to security. Their qualification as experts was based on their reputations, either directly from the researchers' knowledge; or as validated by the people who referred them to us.
In the following sections, quotations from the interviewees are in italics. We have edited them to protect confidentiality and indicate context: square brackets show additions and replacements; ellipses show removals.

Introducing dialectic
The Core Category that emerged from the Grounded Theory analysis of the first 'App Security' phase of interviews was the nature of the developer's interaction with external parties: a friendly adversarial interaction. The word 'dialectic' had surfaced in our earlier work [6], and we realized it was the common theme to all these interactions. 'Dialectic' is the finding out of knowledge, especially logical inconsistencies, through one person questioning another 5 .
I think part of the good practice guide is embedding questioning. How, why, what, where, when. It is almost asking developers to ask the questions that begin with those words and thinking of a set of questions that begins with those words. What if we did that? How would we do that? When could we do this? (P5) If there is something in the application that is a particularly sensitive area. . . you probably want a security guy to come and sit for a few days and have a look at it: someone who can work with the develop team but who hasn't been part of that development team, in a way. So, somebody who can ask the stupid question -I think that is really important, to bring in an external person. (Q11) Dialectic surfaced as the Grounded Theory 'Core Category' with the observation that many of the techniques used for security depended on someone or something 'surprising' and 'questioning' developers: There is a separate Security Review system for, so if you are doing code that impacts security in your judgement, it goes to people who are security experts who will do the security review and they find stuff. (P3) Every production change has to go through the Change Advisory Board, . . . Normally the way the [Penetration Testing] report would be dealt with is we would sit down in a sprint planning meeting and we would go through the report: us, the client, sometimes the tester. . . and we would discuss the findings and produce work that would be added to the next sprint to address what had been found. (Q1) Dialectic first appeared as the technique used by the Greek philosopher Socrates [54]. Dialectic addresses a particular weakness in much existing secure development research, namely that emphasis 5 Although the 'Dialectic' concept came from the first phase analysis, the interviews in the second phase confirmed it, and some of the quotations in this section come from those interviews.
on code level security often misses possible exploits based on the functionality and system design -or may prove unnecessarily costly: Instead, developers need to think about the approaches an attacker might use to gain benefit from the system they are producing, and then to decide how to thwart those approaches.
Yes, the question is 'who is the attacker, who is the bad guy, who is the threat model you are dealing with?' (P3) This is a significantly different approach from 'normal programming', as it requires developers to think in unfamiliar ways.
They are very devious. There are exploits that they have realised which are, well, you wouldn't really think like that if you were an engineer (P2) It was not difficult to work out why our experts should view dialectic as a solution. Unlike other forms of software quality such as performance or reliability, security involves the idea of an attacker who will use remarkably different thinking. To deal with such threats, a developer needs to think 'outside the box;' the easiest way to achieve that is with challenges from others.
The dialectic continues throughout the development cycle; and in each case, it is always two-way: the experts state that the increase in security comes from active interaction with the challenges, not from a passive understanding of the challenge.
Our theory does not suggest necessarily that dialectic provides the best techniques for achieving security; it does suggest, however, that dialectic is associated with effective software security.

Assurance techniques
From the second 'Intervention Expert' phase of interviews, we derived further theory identifying six software assurance techniques as the most effective. 6 These exclude training and motivation activities, and are as follows: For consistency with other literature, the definitions above are taken from Such et al. [49], except for Stakeholder Negotiation, which was not discussed in that paper. Note that the coder was not aware of that paper at the time of coding, and the assurance technique names derived from the Grounded Theory analysis were different.
Participant discussion of assurance techniques Table 1 shows the participants' discussion of these techniques in both phases. The numbers indicate the percentage of participant words coded to each assurance technique: the share of the discussion devoted to that technique within that interview. Cells are highlighted based on their values. The most discussed techniques are on the left; less discussed ones towards the right. 7 The table shows the substantial difference in results we saw according to the questions asked of experts. Some of the difference between the groups can be attributed to the difference in coding: in the second phase we were explicitly coding for 'interventions', which include the assessment techniques. However, we observe a surprising lack of reference to forms of Vulnerability Scan and Stakeholder Negotiation in the first survey.

Dialectic counterparties
The theory also identifies specific counterparty roles for this dialectic interaction: • Members of the developer's team • Stakeholders, such as product managers • Automated code analysis tools • Security experts • Penetration test experts • Other development teams • End users and operations staff.
Note that these interactions are not limited to human forms of questioning; tools for automated code analysis also provide dialectic challenge.
The sections 'Threat assessment' to 'Penetration testing' explore in detail how dialectic applies in each assurance technique, discussing the counterparties for each. Each section discusses the context of the problem, and outlines how the technique solves it, exploring how the interviewees use the technique, backing the discussion with quotations from interview participants

Threat assessment
Threat Assessment is the activity of identifying and ranking the threats to computer software, a software component, or an IT system as a whole [49].

Context
Any system can be broken with sufficient determination, ingenuity, and resources.
Every security system can be broken. Period. There are even ways of getting the certificates off a phone, by freezing the phone and reading the memory. There is nothing you can do to stop a truly determined person to getting in, short of dropping it into a nuclear furnace. The best you can do is make it difficult enough for them, that they will lose interest -that it's not worth the trouble. (P7) I quickly realised that no system is ever unbreakable (P9) As a result, secure development is not a matter of making a completely secure system. Instead, it becomes a question of which defences to implement: where one should spend the time and effort defending the system to deter the largest and most damaging potential exploits. Making those choices requires an understanding of the potential attackers: Several interviewees indicated that developers must drive the threat modelling process.
You need developers to do threat models, but you need developers to understand how the attackers brain works, what is the methodology, the way I know how to do that is to make them do the steps, make them ask the questions. (Q11) Threat Assessment Identifying and ranking the threats to computer software, a component, or an IT system. Stakeholder Negotiation

Discussion and negotiation with stakeholders, such as product managers, on security choices Configuration Review
A review of the way a system or its software has been configured to see if this leads to known vulnerabilities, using manual checking software versions or automated build review scanners. Vulnerability Scan The process of using an automated scanner on a web application or network to identify vulnerabilities.

Source Code Review
The manual examination of source code to discover faults that were introduced during the software development process. Penetration Test A simulated attack on a component or system, carried out by a security expert using similar techniques to that of a real-world malicious attacker.
7 As previously discussed, the asterisked P6 and Q2 were different interviews of the same expert.

Execution and counterparties
The counterparties discussed for Threat Assessment included other developers: Threat modelling: what I see as the big benefit here is, is that it is not that you are coming up with a list of issues that product of the team has, it is more putting the team into the perspective, to think about the functionality from a different aspect, from a different point of view, that that brings up a list of issues, no doubt about that, but it also fosters the understanding to see the coding from a different side, to also to think about it when you create code: what else can there be? (Q10) Some experts emphasized the importance of including more senior stakeholders:

If [the developers] don't know what a threat model is, then tell
them what a threat model is, and the simplest explanation in the world is 'who is going to attack you, and where is the gold?' 'Where's the gold' answers need to come from managers, and company owners. Not from developers because their answers will be totally wrong. And actually that is a big learning that you can give to any company when you go in. Whatever the CEO's threat model is, explain it to all of the devs and go 'if you don't know what the worst nightmare is for your CEO, then how can you be expected not to make that cock up' (Q9)

And the Security Experts themselves:
I was involved in a lot of conversations about trying to think about doing really evil things, so I think in order to protect people from harm we have to think about how harm can be done. So, brain-storming bad intent is part of the life, really. (P5)

And Penetration Test Experts:
One of the things I like to do with the [penetration testing] guys is if you sit down and say, "what are all the different ways you could subvert this system." It is quite common to come up with 20, 30, 40, 50 in five or ten minutes of brainstorming. I bet you, you wouldn't think of half of them. (P2) Particularly with development teams using agile approaches, this Threat Assessment process continues informally throughout the initial development project, and into the subsequent deployment and later lifetime of the product. The most security-capable teams incorporated attacks and motivations found throughout the software product lifecycle.

Stakeholder negotiation
Stakeholder Negotiation is the activity of discussion and negotiation with stakeholders, such as product managers, on security choices.

Context
Merely identifying the possible attackers and exploits does not itself deliver software security. The need is to prevent them from causing damage to users, stakeholders, or others.
For businesses it is a risk-based approach which they need to understand and neither [management nor programmers] should be caring about actual nitty gritty details of coding which is just an artefact of the whole thing. (P6) Given the Threat Assessment, a development team can take the list of possible attacks, and work out possible mitigations for each. These mitigations will each have costs in development time, commitment, finance, and sometimes usability. The team can estimate financial and other costs for each. How, though, do they make the decision which to implement?
Similarly, there is a major challenge for products after the end of the initial development project: Like many things that get delivered in a project, the project ends, and interest dies with it. Unfortunately. And I think you lead into a significant challenge in securing things on an operational basis. (P8) The decision of what aspects of security to implement is a commercial one. Implied in every decision about software security is a trade-off of the cost of the security against the benefit received. Every security enhancement needs to be weighed against other uses of the investment-financial, time, usability-required.

Solution
The solution is for development teams and security experts to express risk and costs to stakeholders (project managers, senior management, customers) in terms they can understand and use to question about security concerns against other organization and project needs.

Execution and counterparties
The primary stakeholders are those who prioritize development tasks, such as product managers. Evaluating and expressing the threat in business terms requires discussion and experience, so other Team Members and Security Experts are also important: The teams take a risk management approach, full stop. For every risk, we try and weight it on two axis -one is the severity, if it happens. So, ignore the likelihood, how bad is it if it happens? And the other is the likelihood, how likely is it to happen. . . so, the project teams will plan their work, and come up with an end date, and the project manager will make sure that they add enough capacity to the project for the weighted risks to happen. (Q14) Many of our interviewees made the point that 'security is not an absolute,' but that security is what the users and stakeholders need for a particular situation at a particular time. For such stakeholders to make a good decision on what security to implement requires particularly effective communication. The stakeholders will be making cost benefit trade-offs comparing various business risks.
You've got to put a weighting on the threat. You've got a level of threat, and you've got to put the appropriate level of security against that. (P4) There are techniques available to give objective assessment of security risks, such as work by ben Othmane et al. [55]. Vitally-and several interviewees stressed this-the cost-benefit trade-offs mean that perfect security, even if possible, would rarely be a good business decision: And actually the way this works, in practice is you have to do less than a perfect job, in order to have a measurable degree of failure or fraud or whatever, so that you can adjust your investment and say 'I am managing this to an economically viable level' because if it is zero, you have invested too much. (P6) For simpler projects and systems, there may not be sufficient engagement from stakeholders to be able to do this kind of trade-off; in that case, it becomes the responsibility of the development team: Given that each mitigation now has a cost and benefit, the decision on whether to do it becomes part of standard project management process. It is outside the scope of this technique-and indeed of the topic of software security-to explore how to make these decisions in general; the balancing of risk cost and reward is a wellunderstood aspect of business life.
And it has to be a bit of a trade off as well in terms of business. You've got to make the trade off as to what's good for getting a solution available now, and having one available in a year's time, which no one will buy, because everyone's gone with one which doesn't even consider security at all. (P12)

Configuration review
Configuration Review is a review of the way a system or its software has been configured to see if this leads to known vulnerabilities, using manual checking software versions or automated build review scanners [49].

Context
Using an insecure component automatically makes a developed system insecure, regardless of the quality of the code developed by the team: WordPress plug-ins are an enormous liability. Anyone can write one, most of them are rubbish. Anyone can get them put up on the plug-in directly, which gives them this air of authenticity, and quality that they don't deserve, frankly. (Q1) So, a 'low hanging fruit' for development is to use only components (including frameworks and toolchains) that are well written and securely implemented. This is non-trivial, given the wide range of components available.

Solution
Configuration Review is the activity of reviewing the codebase to evaluate the components used for security: using public repositories of security vulnerabilities to assess well-known components; avoiding little-known components where possible; and using Source Code Review and Penetration Test to evaluate such components if they must be used.

Execution and counterparties
For some development systems there are corresponding websites with security reviews of plug-ins; cross referencing with these sites is a powerful security technique. This may use automated analysis tools: Where such sites are unavailable, or for new plug-ins, there is a value to Source Code Reviews (see 'Source code review' ) to establish the likely security attributes of a given plug-in. This does however have a significant cost to development teams since it takes effort, however much automation may be involved, and restricts the plug-ins that that developers can use.
And so that is one of the things you end up sitting down with developers going "I'm sorry, but I know this is actually going to slow you down". And we are desperately normally trying to avoid that, I'm trying to make your lives as easy as possible. . . . But you have to say "well, no, you can't just add [components]you have to review them. You don't have to do the most detailed review in the world, but if you think it looks worrying, then . . .

don't put it in your code". (Q9)
A second issue is that, since plug-ins are widely shared, any weakness in a plug-in becomes known to attackers, and therefore it is important to keep plug-ins upgraded to the latest versions in which defects have been corrected.
Since Penetration Testing tests a whole system, it will also find vulnerabilities in components, so testing is another way to assess components:

Source code review
Source Code Review is the manual examination of source code to discover faults that may have been introduced during the software development process [49].

Context
It is notoriously difficult to spot one's own errors. This is especially true when the errors are faults in complex reasoning or are due to misunderstandings. Thus, a programmer working solo is likely to create avoidable security problems, just because they can naturally have only one point of view.
So, it is very easy when you are trying to deliver something yourself, as a developer, to pass over the bit that you are not doing (P5) This problem extends to programming teams. A team will always suffer to some extent from 'groupthink'; the need to generate a shared understanding brings with it the danger that that understanding may include misunderstandings and blind spots. This is particularly important with software security, since such blind spots often lead to vulnerabilities in the developed software.

Solution
In Source Code Review, development teams to provide a counterpart who reviews the security and privacy aspect of assumptions, decisions, and code. Several interviewees stressed the importance of this being a dialectic, questioning, process for security reviews: What is our most successful technique for secure software? In terms of what I have seen, certainly talking amongst the developers, the code reviews have been very useful. (Q16)

Execution and counterparties
The reviewer is typically another developer from the same team.
Code review is what we do endlessly. We certainly do not let any form of code out the door, without an independent review and that is eyeballs on the code and that is discussion about the code.
If a Security Expert is available, however, they can often provide more security-relevant questioning than development team members: The most successful technique has to be review by [a security] expert-you can't really beat that-an actual conversational review by an expert, because someone who is an expert in security might not be an expert in the domain. (P3) [We have] 'software pen testing,' where you have some Subject Matter Experts who take a piece of code and review that in detail, and work with the developers in tandem, looking at the code and saying, "we think this is really high risk, we really want to look at this." And then somebody goes through that code with the mindset of "how do I exploit the code" (Q5) The 'Configuration review' section discusses Source Code Reviews of components developed externally; this might not be a detailed review, but a lightweight check, possibly backed up by forms of Vulnerability Scan ( see that section) We have a tool that we wrote . . . for checking . . . plug-ins, which is intended to make code reviews of plug-ins more focused. It looks for things that are indications of badness and you go and review that list of things, rather than sitting down and reviewing the entire plug-in. (Q1) Another approach to Source Code Review is providing dialectic questioning to a developer via pair programming, where the code is written by two developers sharing one computer: Two heads are better than one, more eyes on the problem. (P7) Cheaper possible alternatives are 'Rubber Duck Debugging' [56] and 'Cardboard Consultant' [57], where the developer explains their thought processes to an anthropomorphized object.
And the heavyweight approach is a formal review process, with separate review meetings delivering lists of defects for a developer to fix.
It is in the culture. We do reviews; we always have to do reviews. We set things up -and it is not regarded as a second class. (Q6)

Vulnerability scan
Vulnerability Scan is the process of using an automated scanner on an application or network to identify vulnerabilities [49].

Context
It is a poor use of expensive resources to find problems that are cheap to find in other ways. Source Code Review, Configuration Review, and Penetration Testing are all expensive in terms of human effort.
[There are] things that big organisations seem to not want to do, like . . . automated testing, code inspection -things that were always considered too expensive (P5) Indeed experts' assessment of cost-cheap, moderate, or expensive-for these three techniques, taking the modal choice in Such et al.'s survey, are that Source Code Review and Penetration Testing are expensive; Configuration Review is moderate [49].

Solution
Vulnerability Scan uses automated tools to look for possible security flaws in the written code. Tools to do this are sometimes called 'lint' checkers, after a UNIX tool that does extra checking for C code.
[The most successful technique I have found is] to use various types of Lint checkers. (P7) The experts' assessment was that Vulnerability Scan is cheap [49]. Indeed, automated tools can be used as an extension to the compilation process of the code, and many interviewees saw them as valuable in automating the removal of certain classes of security bugs.
We use excellent tooling from the Alassian stack, the Crucible Tool . . . We do a lot of static analysis. We review for security, we certainly do, but the point is, is that we will try to automate the removal of whole classes of problem from that. (Q3)

Execution and counterparties
In this the dialectic counterparty is the tool or tool suite itself. There are now many such tools, some produced by commercial companies, supporting different languages, and purposes. They work similarly to compilers, and generally use analysis techniques developed for compilers. Since codebases vary enormously in their style, requirements, and ways of using code, such tools often require significant work to configure: We use something called Sonar which is a code inspection tool. We'd written templates and guides for our coding standards and certain patterns we are looking for in a code and we are looking for changes in the code that are greater than a certain percentage, and there are specific bits of the code we are looking for any change that should never happen. (P5) Such tools typically generate large numbers of 'false positive' warnings. So, developers need to use them as questions ('is this piece of code OK') rather than as notifications of changes to be made. Hence in this article we regard them as a dialectic counterparty, rather than merely an error detector.
Pay attention to the warnings, pay attention to the Link errors.
[So it is not just the automated checks. It is the attitude towards those automated checks, taking them really seriously] Use them, don't forget them. (P7) Some interviewees, however, had found the time cost of analysing the responses to be too great: If you mean static analysis type tools, the answer is no [we do not use them]. Two teams I have worked with have used them. One gave up entirely, because they looked at the amount of time that they were spending on it, versus the reward that they were getting, and then looked at what happened in the first pen test that they had, where they just got completely pwned. (Q9)

Penetration testing
A Penetration Test is a simulated attack on a component or system, carried out by a security expert using similar techniques to that of a real-world malicious attacker [49].

Context
All the assurance techniques discussed so far (sections 'Threat assessment' to 'Vulnerability scan') need access to the development team or source code. Yet, the delivered system is what the attackers see and where the privacy issues occur.

Solution
In Penetration Testing, an external 'white hat' security team simulates what an attacker would do to attempt to gain access or disable the service. The team then feeds back any 'successful' exploits they have found to the development and operations teams.

Execution and counterparties
The single dialectic counterparty is a Penetration Test Expert. Few developers have more than basic skill at Penetration Testing, so it would be unusual for a development team member to take the role. The developers take the results of the Penetration Test and use them to plan future security enhancements.
Normally the way the report would be dealt with, we would sit down in a sprint planning meeting and we would go through the report: us, the client, sometimes the tester, . . . and we would discuss the findings and produce work that would be added to the next sprint to address what had been found. (Q1) At the operating system level, one can also Penetration Test a mobile operating system using software on the device: I think the one [approach to testing our phones] that has been, arguably, most useful has been using specialist external consultancy around security. Not for training, but "can you just come in and penetration test this device?" (P2) Some interviewees have success extending the Penetration Testing to involve direct discussion (dialectic) with the developers: So, the idea is the penetration tester is testing the web application, and the developer is sitting with the penetration tester. That is very effective. A developer will not surely not know more about the application and will see -I am not such a good pen tester, but I do not claim to be able to find all the vulnerabilities in the application -there may be something though that is too arcane for me in 5 days or 3 days to understand from a black box perspectives, but if you are sitting with a developer they are part of the process and they tell you 'ah, you know that request you have just intercepted in Burp -I'll change it to that!' (Q11) Table 2 summarizes the Dialectical Security theory, showing which counterparties provide the dialectic questioning for each of the assurance techniques discussed in sections 'Threat assessment' to 'Penetration testing'. Note that end users and operations staff were not mentioned in the context of any of the Assurance Techniques; they are omitted from the table.

Differences between the two surveys
The 'Participant discussion of assurance techniques' section shows remarkable differences between the two surveys. Even P6/Q2 (one person) discussed Vulnerability Scan in the second interview and not in the first. We suggest these are attributable mainly to the different questions asked (see Appendix). It appears that when experts review developer security in general, they tend to focus on Threat Assessment and Configuration Review; when they come to consider aspects of development security in more detail they then focus more on other techniques, especially Vulnerability Scan.
It is interesting that many of the second survey's participants emphasized Stakeholder Negotiation, even though the nearest that survey's interview protocol came to mentioning it was 'Security coordination with other teams' (where the survey designers had in mind security experts and other developers). That, and the lack of emphasis on it in the first survey, suggest that Stakeholder Negotiation is an important 'emergent requirement' for software development security when practitioners consider the practical detail of implementing interventions.

Choosing assurance techniques
We observe that the list of key assurance techniques derived in the 'Assurance techniques' section is nevertheless a small subset of the 20 techniques identified by Such et al. [49]: the explicit focus that experts placed on them is therefore important.
Based on Table 2 and the expert cost estimates from Such et al., we can further characterize the assurance techniques in terms of two important practical considerations: their cost and their need for security expertise. Figure 3 shows this characterization, using the modal estimate of cost for each technique, and assigning 'cheap' to Product Negotiation, since it in incorporated into activities already carried out by a development team. Source Code Review, Threat Assessment and to a lesser extent Product Negotiation, are more effective with the involvement of security experts, but can take place without them; Penetration Testing requires security expertise, whether in-house or outsourced.
Based on this, we deduce that the most widely applicable assurance techniques are Threat Assessment, Product Negotiation, Configuration Review, and Vulnerability Scan; and that teams with greater resources will benefit from adding Source Code Review and Penetration Testing.

Incorporating the assurance techniques into the development cycle
The ordering of the assurance techniques in the 'Assurance techniques' section is roughly chronological within a development cycle. Figure 4 illustrates how they might be incorporated into an iterative cycle. Threat Assessment and Product Negotiation affect what functionality is produced, so apply to the planning element; Configuration Review and Vulnerability Scan can be automated into a product build; Source Code Review needs a candidate complete implementation, so is typically done at the release stage; and Penetration Testing applies to an installed system, so comes in the test phase.
The 'Summary of assurance techniques and dialectic counterparties' section discusses the dialectic counterparties involved in each assurance technique, and observes that end users and operations staff were not mentioned. We speculate that this reflects the difficulty of involving them, or possibly that it represents an opportunity not yet realized by the security specialists we interviewed.

Improvements on existing practice
Considering the assurance techniques, past improvements to developer based security have often been based purely on the intervener proving the inadequacy of existing security using Penetration Testing, an approach that often proved ineffective in the longer term (see 'Techniques to help teams improve'). Dialectical Security suggests an alternative way of approaching security improvement: regarding it as an interactive dialog with the developers, where all the techniques, from Threat Assessment through to Penetration Testing, are regarded as generating challenges for the developers to debate, discuss and address. The surveys described in this article demonstrate that expert opinion considers this approach to be more successful.

Relationship to existing work
Comparing the existing work on developers and security, the section 'How programmers learn security' discussed research on current practice by developers such as that by Balebako et al. [12]; this work goes further by identifying approaches for better practice.
Considering the assurance techniques themselves, we suggest that all but Stakeholder Negotiation are reasonably well understood [49]. There has been, however, little work on the human interactions required to achieve the techniques, a gap filled by this article. The work of Such et al. [49] provides a taxonomy and cost basis for assurance techniques and identifies the set of Architectural Review, Configuration Review, Penetration Testing, and Vulnerability Scan as the most cost effective. By deeper and more intensive questioning of the subjects, this article reaches a similar but slightly different conclusion, that the subset Threat Assessment, Product Negotiation, Configuration Review, Vulnerability Scan, Source Code Review and Penetration Testing is likely to be the most effective.
As discussed in 'Techniques to help teams improve', there has been a good deal of work on 'Secure Software Development Lifecycle', but prescriptive SSDLs have not been successful with development teams. This work identifies a set of assurance techniques that has been successful. It also, by doing so, provides a basis for future work finding the most effective ways to introduce those techniques to teams that are not yet using them.
The work by Ashenden and Lawrence [47] provides valuable suggestions how to improve the effectiveness of the relationships between security professionals and software developers; Dialectical Security goes further by proposing ways to leverage such improved communication to improve the deliverables from software developers.

Possible future work on dialectic
The importance of dialectic in secure development justifies further research into ways best to use human and team interactions to improve security. Future research might ask research questions such as: • What quantitative evidence can we find that dialectic techniques lead to better security outcomes? • What are the most effective ways to generate understanding of attackers and potential exploits? • How best do we represent security questions in business terms? • What forms of cross-team interaction are most effective to ensure app security?
There are several possible approaches to this research. An ethnographic approach might follow the progress of a development team, identifying where the major security mitigations were identified and how the negotiations took place in practice. Alternatively, a survey approach might ask the questions of a variety of developers and stakeholders to produce a possible consensus.
A different and potentially rewarding area of research is in further dialectic techniques for software security. For example, De Bono [58] has defined a variety of 'Lateral Thinking' techniques to help teams and individuals to challenge their thinking; these might be effective in Threat Assessment. Such research might also investigate techniques to use other counterparties such as operations staff and end users, which are apparently not widely used currently (see 'Summary of assurance techniques and dialectic counterparties').

Threats to validity
How certain can we be that this analysis accurately reflects reality? We approach this question by analysing threats to validity.
Considering first Conclusion Validity, do the research data justify the conclusions? Grounded Theory's rigorous process of line-by-line coding, categorization, and core category generation leads to a theory that does reflect the interview data. The use of extensive quotations ensures that this can be at least partially checked.
In terms of Construct Validity, do the Dialectical Security theory and the descriptions of assurance techniques represent actual practice? Grounded Theory handles this primarily in terms of 'theoretical saturation', reached when new interviews do not add substantially to the theory. Guest [59] suggests that a dozen interviews are often sufficient for this; in this case as researchers we believe we have reached theoretical saturation with regard to the list of techniques, but not with regard to all the potential detail to be uncovered within each technique.
In terms of External Validity, can the results be generalized to a wider scope? Grounded Theory's conclusions are always limited to the specific scope studied [5]. We therefore selected interviewees from a wide range of industry roles (see Fig. 1 and Table 1), via a range of mechanisms (see Fig. 2), and used completely open questions (see Appendix). We conclude that the results are likely to be generalizable to development teams similar to those we interviewed, specifically to teams in the UK, Germany, and the USA.

Conclusion
From a rigorous study using interviews of experts in secure app development, this article derives a theory, Dialectical Security, to provide a basis for analysis of developer assurance techniques. Specifically, it concludes that: • Developer security depends on dialectic: friendly challenges to developer thinking by a variety of counterparties; and • Six assurance techniques, a small subset of the many in use in industry, are sufficient to provide this dialectic and to deliver software development security. This is important for modern software development ecosystems in two ways. Identifying the need for dialectic means that addressing security can move from being about formal processes, artefacts and reports, to being about the developers' interactions with counterparties (including each other) and how best to leverage those interactions. And the identification of a small number of preferred assurance techniques makes it realistic for teams to introduce these techniques with a minimum of support from security experts.
Based on the insights in these surveys, then, what needs to be done is to find practical ways to help development teams to adopt these assurance techniques, and to incorporate the techniques' dialectic into normal development. Initially, this can be through direct training or consultancy-style involvement between researchers and development teams, such as that described in Weir et al. [60]. Later, to gain a wider impact, there are approaches we have previously proposed [61] to encourage adoption by a large population of developers, such as educational online games for developers to play; storytelling, through blogs or even TV; and massively online courses or TED-style video.
Focussing on Dialectical Security's six assurance techniques and the interaction they promote will, we believe, enhance the security of software as it is developed, and lead to better privacy and safety for all of us.