Tree Test

‍

Overview

To improve the information architecture (IA) of the Journal of User Experience (JUX) website, we conducted two tree tests using Optimal Workshop. Each study was limited to 10 participants due to the platform’s payment plan, totaling 20 participants across both tests (Study 1 and Study 2). Participants completed a pretest questionnaire, instructions, the tree test, and a post-test questionnaire.

Our categories were based on a card sort and content inventory of JUX. We analyzed results using Optimal Workshop’s tools while considering participants' UX experience, task completion rate, and navigation paths.

‍

‍

Structure of the Tree Test

The tree test structure was designed using data from the previous card sort study.

• Parent Categories: We used the most frequent labels participants created in the card sort to align with user expectations.
• Subcategories: We relied on the Similarity Matrix to cluster related items, ensuring intuitive grouping.

One challenge was placing “Most Downloaded Articles,” which participants had sorted inconsistently into four different categories. To simplify navigation, it was placed as a third-level category instead of a fourth-level subcategory under “Most Recent Articles.”

‍

Participants

Participants included junior UX designers, UX students, and UX professors to ensure familiarity with UX terms and navigation strategies.

• Pre-study results:
  • 3 out of 19 participants had visited the JUX site before.
  • UX experience ranged from 1 to 20+ years, with many participants being students using the site for academic research.

We targeted participants with UX backgrounds to minimize knowledge gaps and ensure meaningful feedback, aligning with Nikolopoulou et al.’s recommendation that academic audiences require effective search strategies.

‍

Test #1 Results

• Participants: 9
• Task 1:
  • 8 participants completed it, with 7 achieving direct success by selecting the expected category (“Articles”).
• Task 2:
  • Only 44% initially clicked on “Call for Papers,” the correct category.
  • 7 participants completed the task, but only 3 achieved direct success.
• Median completion time: 1 minute 46 second

‍

Test #2 Results

• Participants: 10
• Both tasks showed 90% success rates, with direct success improving from 33% to 70% for Test 2 compared to Test 1.
• Task 2 improvement: 90% of participants correctly clicked on “Call for Papers” on their first try, showing clearer navigation paths.
• Median completion time: 2 minutes.
• Participant feedback: Most found the tasks easy, though a few noted Task 2 required more time to evaluate options.

‍

‍Study Tasks

1. Task 1
   • Prompt: "Suppose you are completing a scholarly research piece – find research about inclusive and universal design."
   • Anticipated path: Home → Articles → Articles by Topic → Accessibility
2. Task 2
   • Prompt: "Suppose you are a professor who has written a research paper. Where would you look to find guidelines about publishing your research?"
   • Anticipated path: Home → Call for Papers → Criteria for Submitting a Paper

These tasks reflected realistic user needs based on our content inventory. Success was measured by the accuracy of category selection, directness of navigation, and first-click accuracy.

‍

Key Findings and Adjustments

1. Structural Changes:
   • In Test 1, “Policies” was a level 2 category, which caused confusion. Moving it under “Call for Papers” in Test 2 improved the success rate for Task 2 by 20%.
   • Rewording Task 2 from "peer-reviewed paper" to "research paper" made it clearer.
2. Direct Success Improvement:
   • Test 2 saw a significant jump in direct success for Task 2 (from 33% to 70%). This suggests participants better understood the relationship between the task and available options.
3. First Click Rates:
   • Task 1 had 100% correct first-clicks across both studies, showing it was straightforward.
   • For Task 2, first-click accuracy increased from 44% in Test 1 to 90% in Test 2 after adjustments to category structure.

Summary of Results

The structural and wording changes made between Test 1 and Test 2 led to improved navigation:

• Task 2's direct success rate increased by nearly 50%.
• First-click accuracy for Task 2 rose from 44% to 90%.
• Task 1 success remained consistent across both studies, with no significant changes.

Participants’ feedback confirmed the clarity of tasks, though Task 2 initially posed challenges due to the placement of Policies as a level 2 category. Moving it to “Call for Papers” and adjusting the wording made the task easier to complete in Test 2.

Final Report

‍

Design Strategy for JUX

The redesigned JUX website offers a more intuitive structure and enhanced navigation, aligning with the needs of its academic users. With the implementation of a revised sitemap, improved labeling, and faceted search, we expect users to have a smoother experience finding relevant content. The inclusion of utility features such as a subscribe option and bookmarking will also increase user engagement and retention. These changes aim to elevate the JUX website into a more efficient, user-centric platform that meets the evolving needs of the UX research community.

The limitations we encountered with metadata optimization and multi-device bookmarking highlight potential areas for future improvements. Despite these challenges, our recommendations reflect achievable solutions that maintain the site’s core functionality while addressing the critical pain points identified through research. With the proposed changes, JUX can better serve its primary audience, encourage scholarly collaboration, and establish itself as a valuable resource in the UX community.