Table of Links

• Abstract and I. Introduction
• II. Related Work
• III. Background and Motivation
• IV. FluentCrypto
• V. Study Setup
• VI. Study Result
• VII. Threats to Validity
• VIII. Conclusion and References

VI. STUDY RESULT

This study showed that FluentCrypto helped participants to complete more crypto tasks in a shorter time compared to when they used the Node.js crypto API. Figure 3 presents the mean time that participants spent on each task as well as the perceived level of difficulty when dealing with each task. Notably, participants spent on average 43% shorter time to complete each task when they relied on FluentCrypto as compared to Node.js cryptography API. We also found that the participants succeeded to complete five more tasks when they relied on FluentCrypto. The investigation of the solutions revealed that there was no security issue in the ones developed for the first task. Only three solutions were secure when participants used Node.js cryptography API to develop the second and third tasks, whereas every solution developed with FluentCrypto was secure.

We applied statistical tests to determine whether these findings are significant. We found that the p-values of all tests are smaller than 0.05 indicating that we can reject the null hypotheses. In other words, there is sufficient evidence with a high confidence to conclude that participants are more efficient to develop secure solutions when adopting FluentCrypto as compared to the Node.js crypto API.

In general, participants perceived a higher level of difficulty as the tasks became more complex. We noted that the subjects mostly relied on the built-in crypto operations in FluentCrypto which handle low-level complexities. This could explain why the gain of using FluentCrypto increased with more complex task, especially, for novices who lacked programming experience or had limited crypto knowledge. Unexpectedly though, when dealing with Task 2, a couple of experienced developers perceived a higher level of difficulty with FluentCrypto than when they used the Node.js standard crypto library. However, these developers often completed their tasks faster when they used FluentCrypto. Moreover, participants rated the third task, on average, easier and spent shorter time on this task than the second task when they used the FluentCrypto. Although these two tasks are different, FluentCrypto is designed in such a way that the solutions are similar to implement. Therefore, what the subjects experienced earlier, especially when dealing with the second task, may have contributed to their success in the third task.

In the following, we discuss the result of each task based on participant characteristics such as years of experience in programming, level of self-reported knowledge in cryptography, and prior experience with the Node.js crypto API.

A. Does adopting FluentCrypto increase developer efficiency when working on a cryptographic task?

We discuss our findings regarding the time that participants invested in each task as well as the perceived level of difficulty.

1) Task 1: All participants solved the first task with both the Node.js crypto API and FluentCrypto. Table IIa explains the results regarding task one. The difficulty level looks similar among the groups but the time needed for the task increases for less experienced participants and decreases for more experienced participants. Nevertheless, the one participant who had more than 10 years’ programming experience completed the task with the help of the standard Node.js API three times faster than using the FluentCrypto. Table IIb and Table IIc reveal similar results based on the perceived level of cryptographic knowledge and experience with the Node.js crypto API. Similarly to our observation from the programming experience perspective, developers who had solid knowledge in cryptography as well as those who were familiar with the Node.js crypto API, completed the task faster than when they used the FluentCrypto.

2) Task 2: The second task was more challenging for participants than the first task. The overall perceived difficulty was increased and the task took more time to solve (see Table IIIb and Table IIIc). Two participants could not finish the task with Node.js crypto API, whereas everyone completed the task when using FluentCrypto. It is evident that FluentCrypto decreased the perceived difficulty for less experienced participants. The time taken to finalize the task when using FluentCrypto reduced for all participant groups except for the group “5-10 years of experience” (Table IIIa).

3) Task 3: The third task had the highest difficulty for the participants when using the Node.js crypto API (Table IVa). It also took on average more time to complete by participants than the other two tasks. FluentCrypto decreased the difficulty and time required to complete the task for all groups, especially for the less experienced group (Table IVb). FluentCrypto improved difficulty and the required time to solve the task for both participants with and without knowledge of Node.js crypto API (Table IVc). One of the senior participants who had beyond 10 years’ experience opted out of the study at this phase. When we asked the reason, he said “I must deal with an urgent matter at work”. Of the remaining participants, three failed to finish the task with Node.js crypto API, but everyone succeeded in this task when they used FluentCrypto.

B. Is FluentCrypto effective in developing secure cryptographic solutions?

We manually assessed the security of each functioning solution against a check list of risks provided by a team of external security experts. We also checked any hindrances raised by the participants.

1) Task 1: Participants were satisfied by the current Node.js documentation, but had difficulties in finding the right part of documentation for FluentCrypto. All participants chose the SHA256 algorithm when they used Node.js cryptography API. In case of FluentCrypto, participants either relied on the default secure algorithm specified by CryRule, or they provided their own choices that were acceptable by FluentCrypto.

2) Task 2: The participants mostly complained about the Node.js documentation and mentioned that it is either incomplete or confusing, and consequently, they had to do additional research on the internet. When using FluentCrypto, one participant attempted to generate an IV for an unsupported cipher algorithm and was not satisfied with the thrown error message.

A majority of participants who completed this task misused Node.js crypto API. Only two participants used a key that was not hardcoded, and others used a hardcoded secret from which they generated the key. None of the participants hardcoded a secret when they used FluentCrypto. They mostly relied on the provided call to generate a key.

3) Task 3: In Node.js crypto API, participants were unhappy about a lack of useful examples in the documentation and therefore, had to explore other information sources. They mainly had difficulties in grasping how to generate a key pair or how to use the keys. Conversely, developers appreciated the level of details in FluentCrypto documentation. What’s more, they found the solution very similar to what they followed while dealing with the previous task. Three participants also acknowledged that FluentCrypto is expressive e.g., “It is great that the name of an API speaks about its functionality” – a participant mentioned.

Except one participant, the rest used hard-coded values to generate the key pairs when they used the Node.js cryptography API, which brings severe security risks. Two solutions also suffered from a small key size. In contrary, none of the FluentCrypto-based solutions suffered from these issues. The participants unanimously employed the secure default key-pair generation provided by FluentCrypto.