Methodology

Our methodology involved two approaches:

1. The Plutus PBL mid-course survey (May 2023 - November 2023)
   • We asked participants of Gimbalabs’ Plutus PBL to provide their feedback as potential Andamio users. You can access the survey here.
2. Andamio Showcases

   1. First showcase: A high-level overview of the Andamio system.

      We discussed how the system uses the Project-Based Learning (PBL) methodology and smart contracts to link skills with tasks, and the potential advantages this offers for more efficient onboarding. You can watch the recording here.

   2. Second showcase: The alpha version of the system.

      We provided a step-by-step walkthrough of how to use the Andamio PBL. The Andamio PBL is our version of documentation as an interactive learning resource. Following this workshop, several people, who expressed interest in using Andamio, minted their learner tokens. This process resulted in several leads proposing to use Andamio in the upcoming Project Catalyst Fund 11.

      In the second showcase, participants had an opportunity to test the platform and provide feedback. They pointed out bugs and suggested features and improvements they would like to see in future versions of the system.

      You can watch the recording here.

Quantitative analysis

• User satisfaction scores - From the survey of potential users, we gathered the following results:

  1. The most desired outcome of a Plutus PBL course is to find a job (30.8%) , followed by developing a product (15.4%)

     

  2. As expected, as learners progress through the course and the subject matter is more complex or sophisticated, the response “I’ve mastered this module!” decreases

     

     and we find a larger percentage of “I’m stuck” responses.

     

     When asked why they answered “I’m stuck”, the most common reason (66.7%) was “I’m having technical difficulties”.

     

     With regard to the use of Live coding sessions to support learning, 72.7% of respondents answered they found these to be very helpful.

• Engagement metrics

  • Of the 235 PBL tokens minted, 26 responded to the survey or 11%. This number is consistent with the number of participants that consistently engaged with the material measured by completion of tasks such as:
    • Changing the lucky number (Module 102, Mastery Assignment #2): 39 participants
    • Add metadata to a transaction (Module 203, Student Learning Target 203.1): 20 participants
    • Faucet aggregator (Module 204 Mastery Assignment): 17 participants

• Number of transactions

  • The number of transactions can be verified with the Plutus PBL token policy ID 05cf1f9c1e4cdcb6702ed2c978d55beff5e178b206b4ec7935d5e056 or by clicking here.

Qualitative analysis

• User comments and feedback
  • Overall sentiment analysis of the results of the survey of potential users of the Andamio platform is positive. Some recommendations are
    • Addition of more modules in the Building Background Knowledge (BBK) section of the program, indicating the need for further support as the learning material increases in complexity.
    • Improvements in content delivery with a focus on written content and collaborative learning opportunities.
    • Improved navigation across the platform as well as features such as:
      • The ability to “pick up” where the student last left off.
      • Search functionality
      • Links or integrations to outside resources (i.e., Miro, Surveys, Discord, etc.)
      • Dedicated “Resources” section

Key findings summarizing most significant insights

While surveys are the obvious choice to collect user data, it presents challenges in terms of data collection. Despite generating more unstructured data, the Andamio Showcases gave us a chance to get a sense of how our potential users and organizations interact with the platform and gather their feedback live while at the same time testing functionality.

From the potential user survey of the 2023 Plutus PBL we observed that most participants are career oriented, either seeking to find a job or to create a product.

Technical difficulties is one of the main barriers to progress by far for most learners. Onboarding to development environments like Nix and the use of Bash scripts where some of the toughest areas to achieve mastery as evidenced by the mid-course survey.

Live coding sessions are a must. An overwhelming majority of respondents identified Live coding sessions as a useful tool to collaborate and learn.

Overall, the analysis of potential users of the Andamio platform indicates the importance of foundational materials to facilitate progression and acquisition of skills by learners as they progress through the course.

Enhancements to the platform include ease of navigation and resource accessibility.

Interactive, hands-on and collaborative learning is highly valued by learners.