Reflection

Thank you for all the suggestions. They were really helpful.

No code-marking:

I made some major changes in the experiment setup for this round of code-marking experiment. The new code-mark experiment can be seen at below links:

Session 0: http://projects.abhardwaj.org/redprint-experiment/codemark/ps0.php

Session 1: http://projects.abhardwaj.org/redprint-experiment/codemark/ps1.php

Session 1: http://projects.abhardwaj.org/redprint-experiment/codemark/ps2.php

I ran this experiment with 6 more subjects and after discussing the results (only 1 out of 6 code-marked) with Scott, I decided to focus on the other aspects of Redprint which looked more promising. As I had said earlier, “instant example display” was the feature people liked the most and so Scott suggested me to run some experiments which focuses on “instant-search through the editor window” vs. “search through search bar”.

Instant Example Display Experiment Setup:

Randomly, participants were given either instant-enabled or instant-disabled setup.

First session (Session 0) was a practice session (look at the URL below):

Session 0: http://projects.abhardwaj.org/redprint-experiment/instant-enabled/ps0.php

In this session, we asked them to first get familiar with the IDE by doing various tasks. Then we gave them a sample code, which they executed. The sample code was designed in such a way that it introduces them with the APIs that can be useful for the next session.

Second session (Session 1) was the actual coding:

Session 1: http://projects.abhardwaj.org/redprint-experiment/instant-enabled/ps1.php

We gave all the participants a starter code and they were asked to complete the code for computing mean, median and mode of a given dataset. We gave them the algorithms for computing mean, mode and median.

The experiment result seems to be very promising. Result summary can be seen at:

https://spreadsheets0.google.com/spreadsheet/viewanalytics?formkey=dHZob2k5TWkwTlRVdzBzUkpaUnRmRWc6MQ

In this blog, I will discuss results of the first controlled experiment I conducted for Redprint. I will also touch upon various issues I see with my experiment setup and some of the difficulties I have in simulating some scenarios.  In the end I will talk about a few other observations which I hadn’t thought of initially as part of my experiment but is worth considering.

Participants Background:

This controlled experiment involved 4 subjects over three different sessions. Over the next week, I would be conducting two more experiments with 8 more subjects. Out of the four subjects participated in the experiment, three are Stanford CS students (two MS students and one PhD student). The fourth participant is a recent graduate (completed his MS CS last year from NYU and is currently a s/w developer).

Experiment Tasks:

The participants did one of the two tasks below:

Task #1 (the 3 Stanford students worked on it)

Session 1: Create an array, add some values in in it. Print the array

Session 2: Implement STACK as an ADT (push() and pop())

Session 3: Implement Queue as an ADT (  enqueue() and dequeue() )

Task #2 (the NYU student worked on it)

Session 1: Write a function to create md5 checksum of a string

Session 2: Write a function that can take username and password as an argument and store it in a global array (password should be stored as md5 hash)

Session 3: Write an authentication model which can add user account (username ad password) and authenticate it for a given username and password.

For different participants, the time gap between sessions were different (mainly because of their schedule).

Observations:

Despite the fact that I had told them about code-mark feature, only one out of the four subjects really did code-mark in the first session. I deleted the first session code for all subjects and they were forced to start afresh. Before the start of the second session, I told them that we might delete any of their files and if they really want to reuse their old code, they must code-mark it.

And everybody did code-mark his code in session 2. They reused their code-marked snippet in session 3.

Question: Do you think, it was natural? Seems like, I forced it on them. What can be corrected for other subjects?

Most of the people code-marked the entire file. Only one of the four participants code-marked snippets/functions.  After looking at the diff of the code in session 2 and session 3, I think most of them just modified their code-marked snippet to get the things working in session 3 (that may be primarily because of the fact that the task in session 3 was very little different from the task in the session 2). The quality of the code varied among different individuals. One participant wrote a really high quality code while the other three wrote the average quality code.

I couldn’t measure the time for the first session. For the second session: mean time was 30 minutes with a standard deviation of 10 minutes, while for the third session, mean time was 11 minutes with a standard deviation of 4 minutes.

Post Experiment Survey:

All participants were asked to fill a post-experiment survey. Please see the below link for details:

https://spreadsheets.google.com/spreadsheet/ccc?key=0AlQqshOTBvVndE9FZGlJdFhHYWVReDd0SFMyNGFzbmc&hl=en_US&authkey=COCnm4YB

Usage of different component of the IDE:

I logged the number of requests made to different modules of Redprint.

Note:  every call for a search  (in Reminisc searchbar) calls the three methods – searchCodemark(), searchExamples() and searchHelp()

Also, this search only includes search requests made by users by typing a query in the search box. It doesn’t include requests made for instant example display while typing code in the editor window.

Conclusion

I am still unsure, what to conclude from this data. I also think that I have made an error in my experiment (I shouldn’t have deleted the file from session 1). But the problem is, had I not deleted it, nobody would ever use the codemark as the file would be visible in his workspace and he can simply modify the same file. In order to prove the usefulness of codemark, I must simulate a situation where a user can’t find his old code. I am not sure, what should I change for my remaining experiments. Any suggestion would be welcome.

There are some other findings from this experiment though, which are very encouraging. Redprint IDE is very different in a way that it shows instant example results while typing code in the editor window. Some of the participants really liked it. They say that, this is the best thing they like about Redprint. Do you think it would be worth giving a try comparing variables like code reuse and time with instant example display enabled vs. disabled. I think that would be a good experiment. What do you suggest? I think this can also be a very good hypothesis. As far as I know, no editor has instant example display.

I conducted an experiment to understand, how much do people reuse source-code. And if they do, does it really help them complete the task faster.

We had 4 participants, all CS grad students. They all used PC at Blackweilder cluster.

Day 1:

Task: We asked them to create a Login page which will authenticate users for logging in to zippings home page.

Scenario: We gave them Zippings database (MySQL)which had a table called User with three fields – id, email, password. The table was already populated with a few records. We allowed them to use Internet or any other aid.

Observations: All four participants searched on Google and copied a code for login. All of them modified the code after copying.  The time taken by each participant is below:

P1A – 35 minutes

P2A – 48 minutes

P1B – 42 minutes

P2B – 46 minutes

Day 2:

Task: We asked them to write a new page to register a new user for the same application (zippings).

Scenario: For Group A participants (P1A and P2A) we gave them access to their previous day code. For the Group B (P1B and P2B), we gave them a raw setup.

Observations: Group A (they had access to their previous day code) didn’t start with Google search like the first day. They copied their previously written login form, renamed it to register in the Login UI page, created a copy of login-exec and renamed it to register-exec. Later they did some modification to the register-exec to make it work. In between they did Google search for getting php code for mysql insert query. The time taken by group A is below:

P1A: 12 minutes

P1B: 17 minutes

Group B went straight to Google. They copied code after searching the web and later modified it to work for them

P2A: 19 minutes

P2B: 21 minutes

Analysis: Although the sample size is very small and it is very difficult to make any conclusion, one thing is clear that people who have access to their previous code tend to use it in their current project, if relevant. Also, in this experiment, people who had access to their previous day code completed their tasks faster than the people who didn’t have access to their previous code.