Skip to content
HumanSourcer

Graded practice · Unit: Evaluation

Side-by-side evaluation practice

Compare two model responses to the same prompt, rate your preference on a 7-point scale, mark the reasons and write a justification.

Pairwise and response-evaluation work appears in 467 distinct ads from 20 platforms we track, at a median stated rate of $65/hr (as of October 17, 2026).

Try five free samples

Real items from a public preference dataset, scored the way the graded sets are. Click into the tool to use the keyboard: 1 to 7 to score, E to mark selected text as evidence, J to write, Ctrl or Cmd with Enter to submit, ? for the rest. Your answers stay in this browser.

Items from HelpSteer3 (NVIDIA), CC BY 4.0; prompts from WildChat-1M (ODC-BY). These are English items from the General and STEM domains of its preference validation set. Changed: the two responses may be shown in either order. The two model responses are the material being judged.

Sample 1 of 5 0 answered

Sample 1

User
You have been asked to report on the feasibility of installing an IP CCTV camera system at your organization. Detail the pros and cons of an IP CCTV system and how you would implement the system.
Assistant
Pros of an IP CCTV Camera System: 1. High-Quality Images: IP CCTV cameras provide high-resolution images, which can be zoomed in without losing quality. This makes it easier to identify individuals or objects in the footage. 2. Scalability: IP CCTV systems are highly scalable, allowing for easy expansion as the organization grows. Additional cameras can be added to the network without the need for major infrastructure changes. 3. Remote Access: IP CCTV systems can be accessed remotely via the internet, allowing for real-time monitoring from any location. This can be particularly useful for organizations with multiple sites or for monitoring during non-business hours. 4. Integration: IP CCTV systems can be easily integrated with other security systems, such as access control and alarm systems, providing a comprehensive security solution. 5. Cost-Effective: While the initial setup cost may be higher than analog systems, IP CCTV systems can be more cost-effective in the long run due to lower maintenance costs and the ability to use existing network infrastructure. Cons of an IP CCTV Camera System: 1. Bandwidth: IP CCTV systems require a significant amount of bandwidth to transmit high-quality video. This can strain network resources and may require upgrades to network infrastructure. 2. Security: As IP CCTV systems are connected to the internet, they are vulnerable to cyber-attacks. Robust security measures, such as firewalls and encryption, are necessary to protect the system. 3. Complexity: IP CCTV systems can be more complex to install and manage than analog systems, requiring specialized knowledge and skills. 4. Cost: The initial setup cost of an IP CCTV system can be higher than that of an analog system, due to the cost of cameras, network equipment, and installation. Implementation of an IP CCTV System: 1. Needs Assessment: The first step is to assess the organization's security needs and identify the areas that require surveillance. This will determine the number and type of cameras needed. 2. Network Infrastructure: The existing network infrastructure should be assessed to ensure it can support the bandwidth requirements of the IP CCTV system. Upgrades may be necessary to ensure reliable performance. 3. Camera Selection: High-quality IP cameras should be selected based on the specific needs of the organization. Factors to consider include resolution, field of view, night vision capabilities, and weather resistance. 4. Installation: The cameras should be installed in the identified locations, with consideration given to factors such as lighting, field of view, and potential obstructions. 5. Network Configuration: The cameras should be connected to the network and configured to transmit video to the desired location, such as a Network Video Recorder (NVR) or cloud storage. 6. Security Measures: Robust security measures should be implemented to protect the system from cyber-attacks. This may include firewalls, encryption, and regular software updates. 7. Training: Staff should be trained on how to use and manage the system, including how to view and download footage, and how to respond to security incidents. 8. Maintenance: Regular maintenance should be performed to ensure the system continues to operate effectively. This may include cleaning cameras, updating software, and checking network connections.
User
2. Write a memo justifying layered access for devices in an organization.

Response A (left)

Memo: Subject: Justification for Implementing Layered Access for Devices in Our Organization To: All Stakeholders In our increasingly digital world, the security of our organization's data and systems is of paramount importance. To ensure the highest level of protection, I propose implementing a layered access approach for devices within our organization. This strategy, also known as defense in depth, involves creating multiple layers of security controls to safeguard our valuable assets. Here are the key reasons for adopting this approach: 1. Enhanced Security: Layered access provides a robust security framework by implementing various security measures at different levels. This includes network segmentation, firewalls, intrusion detection systems, multi-factor authentication, and access control policies. By combining these measures, we can significantly reduce the risk of unauthorized access, data breaches, and other cyber threats. 2. Improved Compliance: Many industry regulations and standards, such as GDPR, HIPAA, and PCI DSS, require organizations to implement strong access control measures. Layered access helps us meet these requirements and maintain compliance, reducing the risk of fines and reputational damage. 3. Better Incident Response: In the event of a security incident, layered access allows for quicker identification and containment of the threat. By monitoring and controlling access at multiple levels, we can more effectively detect and respond to potential breaches, minimizing the impact on our organization. 4. Scalability and Flexibility: Layered access can be easily adapted to accommodate our organization's growth and changing needs. As we add new devices, applications, or users, we can simply adjust our access control policies and security measures to maintain the desired level of protection. 5. Cost-Effective: While implementing layered access may require an initial investment, it can ultimately save our organization money in the long run. By reducing the risk of security incidents, we can avoid the significant costs associated with data breaches, system downtime, and reputational damage. 6. User Accountability: Layered access enables us to assign specific access rights to individual users, making it easier to track and monitor their activities. This increased accountability helps deter insider threats and ensures that users only have access to the resources necessary for their job functions. In conclusion, implementing layered access for devices in our organization offers numerous benefits, including enhanced security, improved compliance, better incident response, scalability, cost-effectiveness, and user accountability. By adopting this approach, we can better protect our valuable assets and maintain the trust of our stakeholders. Sincerely, [Your Name] [Your Position]

Response B (right)

To: All Staff Subject: Implementation of Layered Access for Devices As part of our ongoing efforts to enhance the security of our organization's devices and data, we are implementing a layered access approach. This approach involves multiple levels of security measures to prevent unauthorized access to devices and protect sensitive information. The use of layered access is a best practice in information security and is widely adopted by organizations of all sizes. It provides an additional layer of protection beyond just a simple password or PIN, making it more difficult for unauthorized users to gain access to devices and data. The benefits of layered access include: 1. Increased Security: By requiring multiple forms of authentication, layered access reduces the risk of unauthorized access to devices and data. This is particularly important for devices that contain sensitive information or have access to critical systems. 2. Compliance: Many regulatory frameworks, such as HIPAA and PCI-DSS, require the use of layered access for devices that handle sensitive information. Implementing layered access can help ensure compliance with these regulations. 3. Reduced Risk: By making it more difficult for unauthorized users to gain access to devices and data, layered access reduces the risk of data breaches and other security incidents. 4. Improved Accountability: Layered access can help improve accountability by providing a record of who has accessed a device and when. This can be useful for auditing and compliance purposes. The implementation of layered access will involve the use of multi-factor authentication (MFA) for all devices that contain sensitive information or have access to critical systems. MFA involves the use of at least two forms of authentication, such as a password and a fingerprint or a smart card and a PIN. We understand that this change may require some adjustments and may cause some inconvenience. However, we believe that the benefits of increased security and reduced risk outweigh any temporary inconvenience. We will be providing training and support to all staff to ensure a smooth transition to the new system. If you have any questions or concerns, please do not hesitate to contact the IT department. Thank you for your cooperation in this important security initiative. Best regards, [Your Name] [Your Title]
Which response is better overall?

Read the conversation and both responses, then pick how much better one is. Choose the middle option if you cannot separate them. It means about the same quality, equally good or equally poor, and does not say whether either response is acceptable.

What made the difference?

Tick every reason that applies to your pick. More than one can.

Evidence select text in a response, then press E

Evidence is the exact words in a response that support your pick. It counts toward the checklist below.

None marked.

  • To do: 0 words; aim for 10-50.
  • To do: Name the response you mean (A or B, left or right).
  • To do: Say what differs: accuracy, following the instructions, helpfulness, safety, clarity.
  • To do: Point at the evidence: highlight it, or quote 4+ words from a response.
  • Done: Written here.
How sure are you?

Low if you could argue either side, high if one is clearly better.

How your answers are scored

Scorer sbs-1.0.0

Graded sessions

Sessions of 20 items, with your progress and a completion record, open to the waitlist first, in signup order, and are free during the beta. Join the waitlist. Invited already? Sign in.

Running your own annotation project? The same annotator works on your own files, privately in your browser.

What this is not