Empirical Study

a large-scale experiment

Empirical Study

a large-scale experiment

We present the first study that analyzes the usage of asynchronous libraries and new language constructs, async/await in a large experiment.

We analyzed:

1378 Windows Phone Apps


12M Source Lines of Code

produced by

3376 Developers

These are the Windows Phone applications that we studied: Application List.
We gathered these applications from Github and Codeplex by using our tool WPCollector.

(Mis)usage Examples of Asynchronous Idioms

Developers learn a new programming construct through both positive and negative examples. We provide hundreds of real-world examples of all asynchronous idioms. Because developers might need to inspect the whole source file or project to understand the example, we link to highlighted source files on GitHub.

Long-running CPU-Bound Operations
New Thread Link to Usage Examples
Background Worker Link to Usage Examples
Threadpool Link to Usage Examples
New Task Link to Usage Examples
Blocking I/O-Bound Operations
Callback-Based APM Idioms Link to Usage Examples
Task-Based TAP Idioms with async/await Link to Usage Examples
Misuse of Async/Await
Fire-Forget async Methods Link to Usage Examples
Unnecessary async Methods Link to Usage Examples
Long-running Operations Under async Methods Link to Usage Examples
Unnecessarily Capturing Context Under async Methods Link to Usage Examples
Check out our other empirical study for parallel libraries: LearnParallelism.NET


Asyncifier & AsyncFixer


Asyncifier & AsyncFixer

We developed (i) Asyncifier, an automated refactoring tool that converts callback-based asynchronous code to the new async/await;
(ii) AsyncFixer, a tool that finds and corrects common misuses of async/await.

Patches (Examples of Program Transformations)

We selected the most recently updated apps, applied Asyncifier and AsyncFixer ourselves, and offered the modifications to the original developers as a patch via a pull request. Developers accepted 314 patches generated by our tools.
ps: If the link to the patch or pull request is broken, it means that the application repository is removed from its hosting service.

Table 1: Asyncifier Transformations
App Name Status # Link
TheOldReaderWPACCEPTED5Link to PR or Patch
Diettr-GPLACCEPTED1Link to PR or Patch
OcellACCEPTED3Link to PR or Patch
phoneguitartabACCEPTED3Link to PR or Patch
HackerNewsACCEPTED4Link to PR or Patch
Windows-Phone-Starter-Kit-for-RSSACCEPTED1Link to PR or Patch
SyncFramework-ToolkitACCEPTED4Link to PR or Patch
2ch-Browser-WP7ACCEPTED3Link to PR or Patch
battlelogmobileACCEPTED4Link to PR or Patch
RuminateWAITING1Link to PR or Patch

Table 2 shows the applications which are selected to fix "Long-running operations under async methods". Table 3 shows the applications which are selected to fix "Unnecessary async/await methods". We did not especially select 10 apps to fix "Unnecessarily capturing context"; but AsyncFixer did fix this misuse in the selected apps, shown in Table 2 and Table 3.

Table 2: AsyncFixer Transformations - "Long-running operations under async methods"
App Name Status # Link
kido-winACCEPTED5Link to PR or Patch
inspireACCEPTED1Link to PR or Patch
azure-wp-restACCEPTED3Link to PR or Patch
Kinvey-WindowsPhone8ACCEPTED63Link to PR or Patch
indulged-flickrACCEPTED54Link to PR or Patch
battlelogmobileACCEPTED4Link to PR or Patch
WindowsPhone-CommonACCEPTED1Link to PR or Patch
kanboxwpACCEPTED4Link to PR or Patch
TheOldReaderWPACCEPTED1Link to PR or Patch
iRacer-MotionControlwas WAITING (now ACCEPTED)1Link to PR or Patch
Table 3: AsyncFixer Transformations - "Unnecessary async/await methods"
App Name Status # Link
kido-winACCEPTED47Link to PR or Patch
Cimbalino-Phone-ToolkitACCEPTED3Link to PR or Patch
restafariACCEPTED41Link to PR or Patch
adsclientACCEPTED24Link to PR or Patch
kanboxwpACCEPTED2Link to PR or Patch
trellowACCEPTED4Link to PR or Patch
BankrACCEPTED1Link to PR or Patch
32feetACCEPTED3Link to PR or Patch
wpnestACCEPTED17Link to PR or Patch
Kinvey-WindowsPhone8ACCEPTED7Link to PR or Patch

What people say

These are what people say about the paper or the transformations of the toolkit when they accept our patches.

I'm very impressed at your work in analyzing the problems empirically and providing tooling to help! Don Syme, Designer&Architect of the F# programming language

That was pretty good, thanks. I'll look forward to the release of that refactoring tool, it seems to be really useful. Guillermo, Developer of Ocell

You are doing very interesting stuff. That's fortunate, because I've been thinking about replacing all asynchronous calls to new async/await style code Vitaliy Soubbotskyy, Developer of phoneguitartab

This is awesome insight! Great work! Tiangong You, Developer of indulged-flickr

That's very interesting and useful. Peter Freeman Foot, Developer of 32feet

That you have pointed out is correct. This time, performance has been improved to 28 milliseconds from 49 milliseconds. Kenji Wada, Developer of Softbuild.Data

Totally agree, I normally try to take the same minimizing approach, though it seems I missed these. Thanks! Pedro Lamas, Developer of Cimbalino-Phone-Toolkit


This is a joint work with University of Illinois at Urbana-Champaign, Delft University of Technology, and Oregon State University.

Semih Okur

PhD Student
University of Illinois

David Hartveld

MSc Student
Delft University of Technology

Danny Dig

Assistant Professor
Oregon State University

Arie van Deursen

Delft University of Technology