Part of my job in Arts and Science involves reviewing theses after they have been approved by the examination committee. It’s the final step before a student is awarded their degree. I’m finding that I have to send a lot of theses back to students because of incomplete or incorrectly formatted references. I’ll say right away that I don’t check every reference. I just take a quick look at them, so the issues I’m noticing are not the odd typo. They’re systematic errors, repeated many times in a reference list, and usually glaringly obvious to someone who is used to looking at references.

References are created according to patterns, and computers are really good at patterned output. Many of the problems I’m seeing could be avoided if students used a reference manager. This is a piece of software that collects the raw data of your references (authors, title, journal, volume, pages or article number, year, etc.) and then formats your references on demand. Reference managers also generate the in-text citation. This is particularly handy for styles that use numbered references, because the numbers are automatically updated (or, at worst, updated at the push of a button) when new references are added or just move around as you edit your work. If you’re not using one already, you need to use a reference manager of some description. Not only will your references be perfectly formatted every time, but they will alert you if some data is missing. Even better, you only rarely have to input data manually to a reference manager because the citation data can often be downloaded from the publisher’s website or from a literature database, or the software can extract this data from the pdf of a paper.

I won’t tell you which citation manager to use. There are lots of them: EndNote (https://library.ulethbridge.ca/EndNote), Mendeley (https://library.ulethbridge.ca/mendeley), Zotero (https://library.ulethbridge.ca/citation_zotero), JabRef (https://www.jabref.org/), and so on. Some are free, and some are commercial products, and many now provide some amount of free cloud storage with the option to pay for extra storage. If you’re not sure where to get started, ask your supervisor or a Librarian for advice.

I’m a neophyte when it comes to using large-language models (LLMs, i.e. AI “chatbots”). But I have discovered one thing that some of them seem to be really good at: checking references. Your references are not intellectual property, so if you worry about such things, it should be safe to ask a chatbot to check the formatting and accuracy of your references. You can literally just tell a chatbot that you want it to check your references and then paste them into the text box. If there’s too much output—LLMs tend to be wordy—you can always ask it to (for example) not include any output for correct references, or to create a spreadsheet with highlighting of incorrect references, or whatever works for you. I would not take an LLM’s word for any corrections it suggests because even the best ones can be fooled by incorrect versions of a reference they find (for example) in a paper on the web. In an ideal case, you will only have a few potential referencing errors to check anyway. I would also not under any circumstances give the chatbot my work and ask it to fix anything. A student who tried to do this earlier this year found that the chatbot messed up a lot of the formatting in their thesis. Fortunately, they had a backup copy to go back to. So again, just ask the LLM to check your references and paste the references straight into the chatbot, then check the errors the LLM thinks it found and correct as appropriate.

One more thing: LLMs vary greatly in quality when it comes to this kind of work. Free versions tend to be more error-prone. At the UofL, we give all students and staff access to Microsoft Copilot. If you’re at the UofL, I would strongly recommend that you login to Copilot with your University credentials to do this work. The version that the UofL licenses seems to be quite good at checking references, explaining where the errors are, and providing links to the original source (usually) or to some other source of the citation information (e.g. a database like PubMed). On the other hand, the free version of ChatGPT is (as of this summer) not very good at this task. It misses a lot of mistakes, so it takes multiple passes to find all of them using this tool.

There’s one more thing you should consider: Your institution’s web site will generally have an information page that tells you whether or not an LLM that they license has privacy protections. Ours is here: https://uleth.sharepoint.com/sites/artificial-intelligence/SitePages/ToolsandTraining.aspx. (This link will only work if you have a UofL login ID.) As a rule, you need to login to benefit from these protections if they are available for the particular tool you are using. I would generally avoid using a free LLM except for the simplest of tasks where privacy is not an issue. They are known to leak information, so you should think of anything you put into a free LLM as being in the public domain. How much do I trust the paid versions, including those paid for through institutional subscriptions, even if they say that they will “sandbox” your data? Not much at this point. But checking references doesn’t require that you give the LLM any information that isn’t already out there, so for me, that’s a good use-case.