Face and Speech Fairness Assessments

Led a multidisciplinary team of researchers, engineers, designers, and external facial morphology and speech dialect experts to drive the completion and publishing of the pilot fairness assessments for our face verification and speech to text API's. We developed this framework when completing and communicating out the fairness results:

1. Introduction outlines details on the features we are testing and how this assessment relates to Microsoft's and our customer's broader responsible AI goals
2. ‍Data Collection Approach outlines details on the demographic factors and groups we chose to assess and why, participant selection, data collection equipment and more.
3. ‍Dataset Details outlines how we processed and cleaned the data to create the diverse benchmark dataset used in the evaluation
4. ‍Evaluation Approach outlines details on the methodology like the metrics and threshold selected when completing the evaluation
5. ‍Evaluation Results outlines the final fairness performance results and analysis
6. ‍Next Steps includes guiding customers to review the use cases and limitations of our service as well instructions to perform their own fairness assessment on their own data

This launch with our Face and Speech services was the first time Microsoft externally published granular fairness results for any of it's services.

Once we published these, I continued to work with our researchers and policy team to collect insights and feedback to incorporate into a fairness assessment playbook so that other teams at the company could also follow, complete, and publish high quality fairness assessments. This playbook included guidance on hiring external experts and vendors, collecting benchmark datasets, defining the right evaluation metrics and thresholds, and more. Check out the assessments and screenshots below: