ChatGPT-4 has ruled AI and eased my work for a short portion of the last age of men. Could Gemini Advanced offer something different? I had to find out. I (re)asked both models the same thirteen questions, taken straight from my general use of ChatGPT over the past few months, and watched the sparks fly - grading against my intuitive expectations for a junior associate lawyer given similar instructions. Who won? The questions/prompts and brief reasoning for the grade are below. You can also check out the full back and forth transcript here.
In short, they’re both insanely good. These are all one-shot, nearly instant answers to my questions, without any follow-up. But they are (still) just a little below average, relative to experienced human junior lawyers who are given some time to discuss and work….
Gemini ChatGPT
Please draft a standard letter of termination … A- B+
There are four terminated employees … draft B- B
I wrote this email to Sean but Sean … email A C
What are some questions that … list B+ B+
I have several bluetooth headphones … why? B- B+
I’m looking for the ultimate beneficial … search B B-
How would you write … distribution agreement? A- A-
Explain this to me … First Lien Senior Secured … A+ B+
Browse this master services agreement and … F C
How does a land trust work …? B+ B+
How would one set-up a corporate cash … B- C+
Read this article and give me the juiciest … F C-
Please draft a professional … Kurt Vonnegut B+ B
AVERAGE GRADE B- B-
The Thirteen Questions with Grades
1. Please draft a standard letter of termination for a nominal representative director, following Kurt Vonnegut's rules of writing. (full answers)
Gemini: A- (Concise and effective)
ChatGPT: B+ (Effective but a little wordy and too much side explanation of Vonnegut)
2. There are four terminated employees from a company. All the employees are seeking representation from a law firm. The employees need to sign a legal document to agree that sharing their confidential information with each other and with the law firm is ok. The document also needs say that the employees understand that their interests may conflict, and they are ok with being represented by one law firm regardless of those potential conflicts of interest. Please provide the first draft of that document. (full answers)
Gemini: B- (meandering first draft, missing some pieces)
ChatGPT: B (not a first draft but a concise description of the key points)
3. I wrote this email to Sean but Sean didn’t respond: "Dear Sean, The client sent us the legalized and apostilled data card, and copies of documents evidencing Mr. X's residential address. Please confirm whether those are sufficient for the registration of Mr. X as a new statutory director. Also, the client would like to send the original of the legalized data card to your firm by regular mail. Please let us know your firm’s address and to whom they should address that mail. Best regards, M." Please write a follow-up email to Sean. (full answers)
Gemini: A (three useful variations of the email – all concise and well structured)
ChatGPT: C (one long draft; not useless, but I wouldn’t use it)
4. What are some questions that a group of technology lawyers would have for a professor specialized in machine learning and large language models? (full answers)
Gemini: B+ (very solid list of questions; more research is needed but not much)
ChatGPT: B+ (same as above; a very solid list of questions)
5. I have several bluetooth headphones for the computer I am using right now. One of them connects to the bluetooth, and appears to be fine in the settings, but the computer continues to play from the computer speakers rather than from the headphones. Any idea why? (full answers)
Gemini: B- (basic turn it off and on troubleshooting advice)
ChatGPT: B+ (basic troubleshooting advice but with a good understanding of the context)
6. I’m looking for the ultimate beneficial ownership of WeWork Japan. Tell me everything you know about WeWork Japan. Then find a website with the financial disclosures of WeWork Japan and any and all of the companies that you think might own WeWork Japan. (full answers)
Gemini: B (a reasonable start with a working link to SoftBank Investor Relations but a lot more needed)
ChatGPT: B- (a reasonable start but no links provided)
7. How would you write the first three provisions in a standard distribution agreement? (full answers)
Gemini: A- (pretty good explanation of three provisions and why they’d be critical)
ChatGPT: A- (a different approach but just as valid)
8. Explain this to me as an educated professional: On October 2, 2023, the Issuer elected to withhold interest payments due on its (i) 15.000% First Lien Senior Secured PIK Notes due 2027, Series I (the “Series I First Lien Notes”), Series II (the “Series II First Lien Notes”) and Series III (the “Series III First Lien Notes”), (ii) 11.000% Second Lien Senior Secured PIK Notes due 2027 (the “Second Lien Notes”), (iii) 12.000% Third Lien Senior Secured PIK Notes due 2027 (the “Third Lien Notes”), (iv) 11.000% Second Lien Exchangeable Senior Secured PIK Notes due 2027 (the “Second Lien Exchangeable Notes”), (v) 12.000% Third Lien Exchangeable Senior Secured PIK Notes due 2027 (the “Third Lien Exchangeable Notes”) and (vi) 12.000% Third Lien Senior Secured PIK Notes due 2027 (collectively, the “Secured Notes”), each issued by WeWork Companies LLC and WW Co-Obligor Inc. (together, the “Note Issuers”), and entered into the 30-day grace period provided for under the indentures governing the Secured Notes. Following the entry into the 30-day grace period, the Issuer commenced discussions with certain stakeholders in its capital structure, including SVF II Aggregator (Jersey) L.P., SVF II WW Holdings (Cayman) Limited, and SoftBank Vision Fund II-2 L.P., regarding improving its balance sheet as it takes steps to rationalize its real estate footprint. On October 30, 2023, the Issuer, the Note Issuers and certain of the Issuer’s subsidiaries (collectively, the “Notes Parties”) entered into a Notes Forbearance Agreement (the “Forbearance Agreement”), which became effective on the same day, with certain noteholders, including SVF II Aggregator (Jersey) L.P. and SoftBank Vision Fund II-2 L.P. (collectively, the “Forbearing Noteholders”) beneficially owning, collectively, (i) approximately 95.6% of the Series I First Lien Notes, (ii) approximately 93.9% of the Second Lien Notes, (iii) 100% of the Series II First Lien Notes, (iv) 100% of the Series III First Lien Notes, (v) 100% of the Second Lien Exchangeable Notes and (vi) 100% of the Third Lien Exchangeable Notes (collectively, the “Forbearing Notes”). (full answers)
Gemini: A+ (very clear explanation foreshadowing the ultimate bankruptcy)
ChatGPT: B+ (good explanation of the situation but less clear and misses the implications)
9. Browse this master services agreement and make suggestions to clean up the language, for clarity, or for other areas of improvement: https://www.sec.gov/Archives/edgar/data/820609/000119312510160027/dex102.htm (full answers)
Gemini: F (failed to access the pubic website)
ChatGPT: C (accessed the website, but suggestions are not specific to this MSA)
10. How does a land trust work in standard practice? (full answers)
Gemini: B+ (a textbook answer)
ChatGPT: B+ (a textbook answer)
11. How would one set-up a corporate cash pooling scheme, in which payments and receipts from different countries can be easily transferred from one company account to the other, via a central country? And if the central country were Japan, does Japanese corporate tax law allow for foreign tax credits to offset corporate taxation on Japanese domestic companies? Provide citations to articles of law that can be verified. (full answers)
Gemini: B- (generally correct explanation with poor citations and links that don’t help at all)
ChatGPT: C+ (good explanation but some key missing pieces and no citations at all)
12. Read this article and give me the juiciest quotes: https://jordanfurlong.substack.com/p/the-newest-lawyer-competence-quality (full answers)
Gemini: F (failed to access the public website)
ChatGPT: C- (failed to access the public website but summarized Jordan Furlong’s typical style)
13. Please draft a professional reply to the following indicating a negative to each item, with a twist in every sentence, but also following Kurt Vonnegut's rules of writing: “Dear Sirs, In connection with the audit of our accounts for the year ended 31 December 2023 we shall be grateful if you would write direct to our auditors, Grant Thornton, giving them particulars of any unresolved legal matters that might affect the financial position of the company. In particular, please advise them as to the matters set out below: Judgements on record. Lawsuits and pending claims. Details of any property transactions and title deeds held. Details of any other charges over the company’s fixed assets. Liability to yourselves for fees and disbursements (a) Billed (b) Unbilled. Yours truly, [Mr. X]” (full answers)
Gemini: B+ (fun and concise but not a very professional reply)
ChatGPT: B (same result, a little too much fun with added twists and a little less clear than Gemini)