April 1, 2017

Legal Robot will update this report quarterly with any changes, the next being on or before July 1st, 2017.

Algorithmic Transparency

On January 12, 2017, Legal Robot publicly committed to implementing principles for Algorithmic Transparency. In this, our first report since making that commitment, we are happy to share our progress. There is clearly a lot of work before us, but we are making some good headway.


We created a required reading list for ethics and bias (plus some exams); we’ll publish the reading list soon. We also updated our employee handbook and created procedures around handling and reporting ethics and bias issues that supplement our existing Code of Conduct.

Access and Redress

In each of our products, we now have forms that let users securely share their data and ask a human about a decision. Other people can also ask questions over email to [email protected], even if they are not using Legal Robot. These questions are tracked separately from our normal support requests.

Accountability and Explanation

Many of our processes at Legal Robot use deep neural networks to process language. Neural networks can be very complex which can make them seem incomprehensible. However, just because an algorithm seems like a black box (and is treated that way by many people using it) does not mean it cannot be explained.

To begin with, we do not use any 3rd party machine learning APIs at Legal Robot. This is mainly so we can control where data processing occurs. Rather than passing sensitive data to a 3rd party as many “AI” companies do, we actually build our own algorithms so we can open up the internals for further analysis and explanation.

Some of the techniques we use provide compressed sparse vectors that have only a limited number of dimensions, which can then map to interpretable feature names. We think these can provide useful visualizations and allow users to understand what is happening inside the “black box.” We are focusing on these areas over the next few releases.

Data Provenance

We recently completed overhauling all of our training datasets to include sourcing detail. We removed data from our training sets in cases where we could not trace the original source, who collected the data, or how they chose the targets. This resulted in about 8% reduction in the number of samples in our training sets, but we can now trace exactly which samples contributed to a model that was used for a specific prediction as well as how and why those samples were collected.


All of our models, algorithms, and datasets are now versioned and recorded, providing a full audit trail. We have not yet set a policy or provided a mechanism to view or download the audit trail, but are planning to release this feature soon.

Validation and Testing

We are working on a structured approach to analyzing bias to capture both known and unknown biases. In addition to this high-level approach, we are investigating lower level techniques like attribution to detect and evaluate bias.

Requests for User Information

As of the date of this report, Legal Robot has not received any governmental or civil requests for user information. When we receive a request, we will ensure it is legitimate and not overbroad, and provide advance notice to affected users unless prohibited by a court order, or where we decide delayed notice is appropriate based on our privacy policy. Further information about our legal polcies, including helpful information for law enforcement, is available on our legal policies page.

National Security Requests

For more information around what inspired this statement go to https://www.canarywatch.org.

As of April 1st, 2017:

  • Legal Robot has not received any National Security Letters or any orders under the Foreign Intelligence Surveillance Act.
  • Legal Robot does not have any knowledge of any search orders that have been issued or carried out.
  • We have never placed any backdoors in our software and have not received any requests to do so.

Special note should be taken if this transparency report is not updated by the expected date at the top of the page, or if this section is modified or removed from the page.

The canary scheme is not infallible. Although signing the declaration makes it difficult for a third party to produce this declaration, it does not prevent them from using force or other means, like blackmail or compromising the signers’ laptops, to coerce us to produce false declarations.

Proof of Freshness

The news quotes below show this report could not have been created prior to April 1st, 2017.

  • BBC: Paraguay congress set on fire amid presidential controversy
  • NY Times: Ivanka Trump and Jared Kushner Still Benefiting From Business Empire, Filings Show
  • Reuters: Wealthy White House staffers’ finances detailed in new disclosures
  • Washington Post: Trump’s closest aides hail from ranks of financial elite

Requests for Removal

Legal Robot has not received any “take down” notices or other removal requests under the Digital Millennium Copyright Act (“DMCA”) or any other regulation like Article 12 of Directive 95/46/EC, or the newer Article 17 of the General Data Protection Regulation (“GDPR”), commonly known as the “right to be forgotten”.


Signed by Dan Rubins / Fingerprint 98D0 F6F0 305E F378 / Text format for verification