Legal Robot will update this report quarterly with any changes, the next being on or before July 1st, 2017.
On January 12, 2017, Legal Robot publicly committed to implementing principles for Algorithmic Transparency. In this, our first report since making that commitment, we are happy to share our progress. There is clearly a lot of work before us, but we are making some good headway.
We created a required reading list for ethics and bias (plus some exams); we’ll publish the reading list soon. We also updated our employee handbook and created procedures around handling and reporting ethics and bias issues that supplement our existing Code of Conduct.
In each of our products, we now have forms that let users securely share their data and ask a human about a decision. Other people can also ask questions over email to [email protected], even if they are not using Legal Robot. These questions are tracked separately from our normal support requests.
Many of our processes at Legal Robot use deep neural networks to process language. Neural networks can be very complex which can make them seem incomprehensible. However, just because an algorithm seems like a black box (and is treated that way by many people using it) does not mean it cannot be explained.
To begin with, we do not use any 3rd party machine learning APIs at Legal Robot. This is mainly so we can control where data processing occurs. Rather than passing sensitive data to a 3rd party as many “AI” companies do, we actually build our own algorithms so we can open up the internals for further analysis and explanation.
Some of the techniques we use provide compressed sparse vectors that have only a limited number of dimensions, which can then map to interpretable feature names. We think these can provide useful visualizations and allow users to understand what is happening inside the “black box.” We are focusing on these areas over the next few releases.
We recently completed overhauling all of our training datasets to include sourcing detail. We removed data from our training sets in cases where we could not trace the original source, who collected the data, or how they chose the targets. This resulted in about 8% reduction in the number of samples in our training sets, but we can now trace exactly which samples contributed to a model that was used for a specific prediction as well as how and why those samples were collected.
All of our models, algorithms, and datasets are now versioned and recorded, providing a full audit trail. We have not yet set a policy or provided a mechanism to view or download the audit trail, but are planning to release this feature soon.
We are working on a structured approach to analyzing bias to capture both known and unknown biases. In addition to this high-level approach, we are investigating lower level techniques like attribution to detect and evaluate bias.
For more information around what inspired this statement go to https://www.canarywatch.org.
As of April 1st, 2017:
Special note should be taken if this transparency report is not updated by the expected date at the top of the page, or if this section is modified or removed from the page.
The canary scheme is not infallible. Although signing the declaration makes it difficult for a third party to produce this declaration, it does not prevent them from using force or other means, like blackmail or compromising the signers’ laptops, to coerce us to produce false declarations.
Proof of Freshness
The news quotes below show this report could not have been created prior to April 1st, 2017.
Legal Robot has not received any “take down” notices or other removal requests under the Digital Millennium Copyright Act (“DMCA”) or any other regulation like Article 12 of Directive 95/46/EC, or the newer Article 17 of the General Data Protection Regulation (“GDPR”), commonly known as the “right to be forgotten”.
-----BEGIN PGP SIGNATURE----- wsFcBAEBCAAQBQJY4B2lCRCY0PbwMF7zeAAAeAsQAHsEmJnFKB8hmfWo/lKtXgUH BaZv3AVy13/jUtarQd95KR45psYEdebZAdYfH7ycWbK59EfAP6/gpI7rtVy95wa3 bN+zZEBwcDb8xm58gsWIukwJhFzIjKLirjbCi5Wdu2kweiIo67PweoDMBxw/WdNU RIgiqOKI5SBY3EFa3wd9QFTivAVozBrrt5vM/NjqYT+lmTor8VAI5jae7p2jFLrw yQ4vM3MMUXSTX6kQJ7z3E4YC2itBlT0KZIV7p/sxYaX4adePaC6IHIfB2pt4JeBv BFNJmCBLhqX23x7R/jp9cIepQURiKcBPGamsz6N/Zia2izfWJixY7I8PTid8cSk+ MNrV9pTly9LHgAD/e78mLyopg5yp3kYxpVUKXQO/9QCnFm2BDGQzZnaKKYneqf+C 6TViAEgfs8b8m8kkv3ET0NaPgpc76rMPBA2v3WeNmGeKp6LnMH0K2bP9+a0ZcKPv F5Q6aeP8Ob9ewpmkFbCd7stPJAddSF+4sBwzGPlc0j4XsQM1SOZ6+ew/ETG+Kp3r u5IO5cvA44B4aHmgjEW5Brii8T8XB10XMOdQcCnWJ1v3R14YOdhUffC3sBiixLX6 5Xh2rnoIfX/fyZeikuzoR7+RW1fifLU0l31UBfHYx2+Fu476TGgizQUJp0FGisde lJFJ0qQMrFZ10RvTFzy6 =UgpI -----END PGP SIGNATURE-----