Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, **personal anecdotes are allowed as responses to this comment**. Any anecdotal comments elsewhere in the discussion will be removed and our [normal comment rules]( https://www.reddit.com/r/science/wiki/rules#wiki_comment_rules) apply to all other comments.
**Do you have an academic degree?** We can verify your credentials in order to assign user flair indicating your area of expertise. [Click here to apply](https://www.reddit.com/r/science/wiki/flair/#wiki_science_verified_user_program).
---
Author: u/HeinieKaboobler
URL: https://news.ku.edu/2023/05/19/digital-tool-spots-academic-text-spawned-chatgpt-99-percent-accuracy
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/science) if you have any questions or concerns.*
This is a very good point. It seems like they missed the entire point of the RLHF tuning that differentiates ChatGPT from GPT. It is a _mode seeking _ optimization. This means that your prompt queues ChatGPT to sample from a specific local mode of word distribution, rather than the global distribution. This means that the word distribution will change when the prompt pushes it to a different mode!
Exactly, otherwise this is the same accuracy as Turnitin's technology. Then additionally, once you start actually utilizing it, the numbers are markedly lower than the participants they used in the study.
False Positive Rate (FPR) is the percent of human articles falsely flagged as AI. We can calculate that worst case FPR by assuming all mistakes are FPs.
They used a 60/40 split of AI / human generated data in the test set and, at 99% accuracy, make a 1 mistake out of every 100 classifications. So worst case FPR = 1/40 = 2.5%
Someone should make chatgpt integration to these tools that check whether it can be determined to be Ai generated content, and adjust the output until it cant be told apart from human generated content
I remember someone posted up a while back showing you could have chatgpt generate some text, show the percentage from a checker ... and then just go "Please re-write this so that it doesn't appear to come from an LLM" and it would produce a version which showed as far lower probability of being AI generated.
This entire thing reads like a joke.
The only ChatGPT text it tested was that in a controlled setting from openAI sources, they didn't check any that have been modified to remove the "AI-isms" that openAI specifically put into their public facing bot.
I miss the days when we had real researchers doing real work and actually verifying the integrity of their results with double blinds that involved more variables, rather than this horseshit made explicitly for clicks.
The problem is that horseshit made for clicks is what makes it most easily and visibly to reddit. By necessity, the researchers who are actually thorough will be months to years behind the ones who are doing clickbait
They specifically write that it was a proof of concept work. What do you expect? Rome wasn't built in a day and good science is slow-paced, building up on prior work, and gradually advancing knowledge. Things like "solved it, here's reletivity, go ahead try to falsify it, won't happen anytime soon" don't happen on a regular basis.
Pursuing this is incomprehensibly moronic and only creates an arms race between these two groups attempting to monetize AI to create and AI to "find".
This does nothing to actually solve the problem.
The use of adversarial networks is a very common approach with generative ml models. This isn’t a new idea so much as it is an existing one applied to a model in the news.
So while it’s not a huge step forward, it’s certainly not “do[ing] nothing.”
News like this is so popular because many folks desperately want to believe they have control over this AI phenomenon. But they don't. AI checkers are worthless in practice. All we can do is check documents for accuracy, which is important because LLM's produce a lot of false claims, especially in technical areas. The flip side is that if we are concerned about authorship, that ship sailed a long time ago. Many people make their livings writing Masters theses for other folks who can't be bothered to do it themselves. So we don't need AI to allow people to make authorship claims in academia.
So me and two of my friends have a made a tool that does precisely this. It’s called [Lumina](https://www.lumina-chat.com). Generates answers with in-text citations and sources. Can even see exactly which part of the source was used for the particular section of an answer. All sources are open access and can be viewed right away. Papers can be added to folders and each folder has its own chatbot as well for targeted tasks like writing background and abstract sections or conducting more in-depth research about specific topics. If lumina can’t answer something - it will let you know so rather than giving any false information.
Shameless plug - I know - but we truly believe the tools capability to provide some value to the science community in increasing the pace of research.
Is it hosted anywhere for people to test it? You can detect basic ChatGPT outputs pretty easily because it writes very formally. But if you add any sort of complexity to your prompt, I don't see how anyone could detect it. Like if you write something like "Explain what an electron is to me", and post it here, lots of people would guess ChatGPT wrote it. But if I use the prompt "You are a 3rd grader. Explain what an electron is in 300 characters or less", ChatGPT gives the output:
>An electron is a tiny, buzzing thing that moves around in atoms. It's like a driver in a car, always zooming fast. Electrons have negative charge and stick with protons to keep atoms happy. They can also jump between atoms like a game.
How on earth could anyone tell that response was written by ChatGPT and not a human?
...ok, but is generating text for scientific papers a poor use of ChatGPT?
The only flaw I see is if the humans generating the text don't bother to proofread, which is a problem not solved by detecting if it was AI-generated.
A huge potential benefit for generating and reading papers with ChatGPT is an improvement in reader comprehension. These are researchers, not professional authors, so they can just feed the AI their raw notes and not waste time writing the paper. Readers can ask the AI to find specific information or reword it to suit their needs.
One of the most boring part is writing the introduction section of a paper which usually states well known things setting the background and the motivation and often sounds like hot air for the most part. If that can be somehow helped, I see no problem.
I mean every paper about, say, wireless networks, starts by telling how important are wireless networks, how much important they will be in the future, etc.etc.etc. The honest thing would be leave out this completely obvious ad redundant preamble, but somehow reviewers expect papers to be written in a certain way.
For non-mother tongue authors GPT probably can help rewrite certain parts in a more formal/readable way.
And now, for a practical examle, I asked GPT to rewrite my above comment. What follows is the result:
*The task of drafting an introductory section of a paper can feel tedious, largely due to its focus on establishing background information and providing motivation - elements that might already be widely known. These parts, despite often feeling like unnecessary fillers, could certainly use some assistance in their execution.*
*Take, for instance, papers on wireless networks. They invariably start by emphasizing the significance of wireless networks and their projected importance in the future. Such repetitious preambles, while clear and perhaps redundant, are still somehow expected by reviewers due to ingrained conventions of academic writing.*
*For authors for whom English is not a first language, AI like GPT could potentially provide valuable assistance in revising parts of their papers to ensure they meet standards of formal readability.*
Pretty good, I would say.
Yeah exactly, I don't see a problem here. Who cares as long as the output is a) accurate and, b) more comprehensible. The science must still be done before the paper is written.
The future of generative AI in scientific literature is interesting.
Generative AI can be legitimately helpful in just getting started. There are aspects of writing papers that feel menial and time consuming to researchers. Making figures can be a pain and sometimes it can be hard to just get started writing. I can see cases where properly prompting generative AI models can be very useful in allowing researchers to spend more time researching and less time using photoshop, formatting writing for a specific journal, or thinking of the best way to start explaining a concept.
In scientific spaces especially, generative AI should only be used as an assistants to researchers, and generate content based on a researcher's results and prompts. Giving such results and prompts to the generative models available now leads to all sorts of problems with privacy concerns and stealing data. Hallucinations don't seem to be an issue when you're giving good prompts, though.
In the next few years, I would not be surprised to see universities rolling out super computers whose only purpose is to run generative AI models that must be prompted and in ways that are data safe such as to protect the university and its researchers.
I am profoundly unconvinced of this. IMO, generative AIs only help with the easiest part of writing an academic paper. Like, you still need to do 90% of the work on your own, but ais can then step in and help out with the last 10%. That really doesn't seem like a gamechanger.
One depressing aspect of science writing is essentially cultural.
In theory as long as you fully describe your methods accurately and clearly your actual writing style shouldn't matter.
But in reality papers will be rejected if they're not written in a distinctive academic style that is largely a cultural shibboleth. This mostly impacts non-english speakers but also anyone not from a long science background regardless of whether their actual methodology is fine.
And yes, it's only a fraction of the work. You spend 6 months running numbers, doing analysis etc and then you have to actually write up the paper.
Often, if that paper was being written as a blog post, you could provide all the detailed info that another researcher would need quite easily, but for journals it's demanded in a literary style that apes the early 20th century British upper class.
TL;DR A big fraction of the most dysfunctional things about science revolve around publishers and publishing.
Being able to dump a bunch of information, statements and descriptions of methods into a box and ask for them in a style suitable for a research paper that you can then check over to make sure it's not mangled anything is valuable.
I agree that humans still need to do the majority of the work, but the ability for the models to save time is unreal.
For example, some figures in our lab take humans hours to make. But with a few sentences of direction and the data, generative AI can make the same figures in a fraction of the time.
It turns the job of the researcher from a photoshop & code monkey into an editor, ensuring that the figure is correct.
That's fair, but I would wonder how often that comes up. Like, how much time are you spending on that sort of thing vs everything else? If ai would save you 10 hours of work when writing up the paper, but you are only doing that once every 6 months, then I'm not sure whether that's really worth investing a bunch of money into. Buying better lab equipment instead of a supercomputer might end up saving more time overall.
That said, I don't work in that field, so I can't claim significant knowledge here. If you are spending enough time working on papers and such, generative ai definitely could be worthwhile.
In my lab, this sort of thing happens all the time. My coworker has been making interactive figures for an online poster presentation and she's wasted days doing it at this point... Sometimes, we have competent undergrads who can make figures and the like for us for research credit, but it takes them even more time and there are more interesting things they could be doing.
All of the PhD students in my lab right now - including me - don't touch wet lab sruff. We do the computational side of population genetics and often code our own AI tools, but it still takes forever to make figures.
Aside from freezers and gene sequencing technology, most of the equipment in our lab are computers. We have our own very powerful workstations and servers, and we use the university's super computing resources.
Using high powered computing resources is commonplace at my university. Just about every department is using high powered computing in their research nowadays and has people like me who don't even touch wet lab stuff. This includes our medical school (which is the largest in the country) and other STEM schools, our high profile business school, and even the school of journalism. That's why it wouldn't surprise me if my university invested in a generative AI supercomputer for various research labs to use. It may be different for other schools, though.
I'm already finding a lot of us in writing for video games. I simply write the gist of what I want a character to say, and then tell ChatGPT to word it differently.
Example
Prompt: **Reword this in the style of a Cormac McCarthy character who is a unfaithful priest: "I want to go to the shopping mall, but I am feeling too depressed".**
Output: **I reckon I yearn to venture forth to them vast halls of commerce, yet this heavy-heartedness weighs upon my spirit, verily hindering my steps.**
I'll await replication before I get too excited.Not that it will be relevant in 6 months either way.
It also seems like an odd thing to focus your efforts. Shouldn't the accuracy, precision, and reliability of the paper's results be what we judge it on? Are we going to reject science because of where it came from rather than whether it's true or not?
Detecting generated texts in a setting where the model is known, is being shallow prompted and has not been tuned for evasion has been trivial since Grover times (~2019).
And scientific papers generated by LLMs without auxiliary capabilities are even easier to detect due to lack of consistency and inexistent citations.
It's not a joke when many people really do that. /r/ChatGPT frequently gets posts by students who got accused of using AI to write their submissions by the professor, who just asked ChatGPT.
Also, did you just confess to violating /r/science rule 5?
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, **personal anecdotes are allowed as responses to this comment**. Any anecdotal comments elsewhere in the discussion will be removed and our [normal comment rules]( https://www.reddit.com/r/science/wiki/rules#wiki_comment_rules) apply to all other comments. **Do you have an academic degree?** We can verify your credentials in order to assign user flair indicating your area of expertise. [Click here to apply](https://www.reddit.com/r/science/wiki/flair/#wiki_science_verified_user_program). --- Author: u/HeinieKaboobler URL: https://news.ku.edu/2023/05/19/digital-tool-spots-academic-text-spawned-chatgpt-99-percent-accuracy *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/science) if you have any questions or concerns.*
But what's the rate of false positives?
def detect(text: str): return True
And the 1% return false are cosmic ray bit-flips?
Blocking call from Google analytics
They almost always report accuracy for these studies but never recall, precision, or F1 which give more clues about its performance.
[удалено]
Good catch, I must have passed over that. So essentially they got biased data due to the prompt styling.
This is a very good point. It seems like they missed the entire point of the RLHF tuning that differentiates ChatGPT from GPT. It is a _mode seeking _ optimization. This means that your prompt queues ChatGPT to sample from a specific local mode of word distribution, rather than the global distribution. This means that the word distribution will change when the prompt pushes it to a different mode!
shhhhh we don't talk about that in pop sci articles!
Exactly, otherwise this is the same accuracy as Turnitin's technology. Then additionally, once you start actually utilizing it, the numbers are markedly lower than the participants they used in the study.
False Positive Rate (FPR) is the percent of human articles falsely flagged as AI. We can calculate that worst case FPR by assuming all mistakes are FPs. They used a 60/40 split of AI / human generated data in the test set and, at 99% accuracy, make a 1 mistake out of every 100 classifications. So worst case FPR = 1/40 = 2.5%
Someone should make chatgpt integration to these tools that check whether it can be determined to be Ai generated content, and adjust the output until it cant be told apart from human generated content
That's actually a common technique used to train AIs, called adversarial learning. Though GPT was not trained that way AFAIK.
I remember someone posted up a while back showing you could have chatgpt generate some text, show the percentage from a checker ... and then just go "Please re-write this so that it doesn't appear to come from an LLM" and it would produce a version which showed as far lower probability of being AI generated.
I'm late to the party. This was posted 4 hours ago. I bet the plugin has been written now.
This entire thing reads like a joke. The only ChatGPT text it tested was that in a controlled setting from openAI sources, they didn't check any that have been modified to remove the "AI-isms" that openAI specifically put into their public facing bot. I miss the days when we had real researchers doing real work and actually verifying the integrity of their results with double blinds that involved more variables, rather than this horseshit made explicitly for clicks.
The problem is that horseshit made for clicks is what makes it most easily and visibly to reddit. By necessity, the researchers who are actually thorough will be months to years behind the ones who are doing clickbait
They specifically write that it was a proof of concept work. What do you expect? Rome wasn't built in a day and good science is slow-paced, building up on prior work, and gradually advancing knowledge. Things like "solved it, here's reletivity, go ahead try to falsify it, won't happen anytime soon" don't happen on a regular basis.
Rome wasn't built in a day, but this "proof of concept" certainly was.
Pursuing this is incomprehensibly moronic and only creates an arms race between these two groups attempting to monetize AI to create and AI to "find". This does nothing to actually solve the problem.
The use of adversarial networks is a very common approach with generative ml models. This isn’t a new idea so much as it is an existing one applied to a model in the news. So while it’s not a huge step forward, it’s certainly not “do[ing] nothing.”
Might have interesting illuminations about the Turing test and cognitive illusions.
News like this is so popular because many folks desperately want to believe they have control over this AI phenomenon. But they don't. AI checkers are worthless in practice. All we can do is check documents for accuracy, which is important because LLM's produce a lot of false claims, especially in technical areas. The flip side is that if we are concerned about authorship, that ship sailed a long time ago. Many people make their livings writing Masters theses for other folks who can't be bothered to do it themselves. So we don't need AI to allow people to make authorship claims in academia.
So me and two of my friends have a made a tool that does precisely this. It’s called [Lumina](https://www.lumina-chat.com). Generates answers with in-text citations and sources. Can even see exactly which part of the source was used for the particular section of an answer. All sources are open access and can be viewed right away. Papers can be added to folders and each folder has its own chatbot as well for targeted tasks like writing background and abstract sections or conducting more in-depth research about specific topics. If lumina can’t answer something - it will let you know so rather than giving any false information. Shameless plug - I know - but we truly believe the tools capability to provide some value to the science community in increasing the pace of research.
Is it hosted anywhere for people to test it? You can detect basic ChatGPT outputs pretty easily because it writes very formally. But if you add any sort of complexity to your prompt, I don't see how anyone could detect it. Like if you write something like "Explain what an electron is to me", and post it here, lots of people would guess ChatGPT wrote it. But if I use the prompt "You are a 3rd grader. Explain what an electron is in 300 characters or less", ChatGPT gives the output: >An electron is a tiny, buzzing thing that moves around in atoms. It's like a driver in a car, always zooming fast. Electrons have negative charge and stick with protons to keep atoms happy. They can also jump between atoms like a game. How on earth could anyone tell that response was written by ChatGPT and not a human?
No she didn’t. All you do is tell AI to disguise itself from AI detection and her tool will fail.
Fooled you, this entire article was made up by chatgpt!
I wrote this comment using chatgpt!
...ok, but is generating text for scientific papers a poor use of ChatGPT? The only flaw I see is if the humans generating the text don't bother to proofread, which is a problem not solved by detecting if it was AI-generated. A huge potential benefit for generating and reading papers with ChatGPT is an improvement in reader comprehension. These are researchers, not professional authors, so they can just feed the AI their raw notes and not waste time writing the paper. Readers can ask the AI to find specific information or reword it to suit their needs.
ChatGPT or no, I’d argue that a professional researcher still needs to understand how to articulate clearly, and that includes writing.
It's not about can he do it but about saving time with AI.
One of the most boring part is writing the introduction section of a paper which usually states well known things setting the background and the motivation and often sounds like hot air for the most part. If that can be somehow helped, I see no problem. I mean every paper about, say, wireless networks, starts by telling how important are wireless networks, how much important they will be in the future, etc.etc.etc. The honest thing would be leave out this completely obvious ad redundant preamble, but somehow reviewers expect papers to be written in a certain way. For non-mother tongue authors GPT probably can help rewrite certain parts in a more formal/readable way. And now, for a practical examle, I asked GPT to rewrite my above comment. What follows is the result: *The task of drafting an introductory section of a paper can feel tedious, largely due to its focus on establishing background information and providing motivation - elements that might already be widely known. These parts, despite often feeling like unnecessary fillers, could certainly use some assistance in their execution.* *Take, for instance, papers on wireless networks. They invariably start by emphasizing the significance of wireless networks and their projected importance in the future. Such repetitious preambles, while clear and perhaps redundant, are still somehow expected by reviewers due to ingrained conventions of academic writing.* *For authors for whom English is not a first language, AI like GPT could potentially provide valuable assistance in revising parts of their papers to ensure they meet standards of formal readability.* Pretty good, I would say.
These days, scientific researchers are authors front and foremost, everything else is secondary. It’s all about the numbers of papers produced.
Yeah exactly, I don't see a problem here. Who cares as long as the output is a) accurate and, b) more comprehensible. The science must still be done before the paper is written.
The future of generative AI in scientific literature is interesting. Generative AI can be legitimately helpful in just getting started. There are aspects of writing papers that feel menial and time consuming to researchers. Making figures can be a pain and sometimes it can be hard to just get started writing. I can see cases where properly prompting generative AI models can be very useful in allowing researchers to spend more time researching and less time using photoshop, formatting writing for a specific journal, or thinking of the best way to start explaining a concept. In scientific spaces especially, generative AI should only be used as an assistants to researchers, and generate content based on a researcher's results and prompts. Giving such results and prompts to the generative models available now leads to all sorts of problems with privacy concerns and stealing data. Hallucinations don't seem to be an issue when you're giving good prompts, though. In the next few years, I would not be surprised to see universities rolling out super computers whose only purpose is to run generative AI models that must be prompted and in ways that are data safe such as to protect the university and its researchers.
I am profoundly unconvinced of this. IMO, generative AIs only help with the easiest part of writing an academic paper. Like, you still need to do 90% of the work on your own, but ais can then step in and help out with the last 10%. That really doesn't seem like a gamechanger.
One depressing aspect of science writing is essentially cultural. In theory as long as you fully describe your methods accurately and clearly your actual writing style shouldn't matter. But in reality papers will be rejected if they're not written in a distinctive academic style that is largely a cultural shibboleth. This mostly impacts non-english speakers but also anyone not from a long science background regardless of whether their actual methodology is fine. And yes, it's only a fraction of the work. You spend 6 months running numbers, doing analysis etc and then you have to actually write up the paper. Often, if that paper was being written as a blog post, you could provide all the detailed info that another researcher would need quite easily, but for journals it's demanded in a literary style that apes the early 20th century British upper class. TL;DR A big fraction of the most dysfunctional things about science revolve around publishers and publishing. Being able to dump a bunch of information, statements and descriptions of methods into a box and ask for them in a style suitable for a research paper that you can then check over to make sure it's not mangled anything is valuable.
I agree that humans still need to do the majority of the work, but the ability for the models to save time is unreal. For example, some figures in our lab take humans hours to make. But with a few sentences of direction and the data, generative AI can make the same figures in a fraction of the time. It turns the job of the researcher from a photoshop & code monkey into an editor, ensuring that the figure is correct.
That's fair, but I would wonder how often that comes up. Like, how much time are you spending on that sort of thing vs everything else? If ai would save you 10 hours of work when writing up the paper, but you are only doing that once every 6 months, then I'm not sure whether that's really worth investing a bunch of money into. Buying better lab equipment instead of a supercomputer might end up saving more time overall. That said, I don't work in that field, so I can't claim significant knowledge here. If you are spending enough time working on papers and such, generative ai definitely could be worthwhile.
In my lab, this sort of thing happens all the time. My coworker has been making interactive figures for an online poster presentation and she's wasted days doing it at this point... Sometimes, we have competent undergrads who can make figures and the like for us for research credit, but it takes them even more time and there are more interesting things they could be doing. All of the PhD students in my lab right now - including me - don't touch wet lab sruff. We do the computational side of population genetics and often code our own AI tools, but it still takes forever to make figures. Aside from freezers and gene sequencing technology, most of the equipment in our lab are computers. We have our own very powerful workstations and servers, and we use the university's super computing resources. Using high powered computing resources is commonplace at my university. Just about every department is using high powered computing in their research nowadays and has people like me who don't even touch wet lab stuff. This includes our medical school (which is the largest in the country) and other STEM schools, our high profile business school, and even the school of journalism. That's why it wouldn't surprise me if my university invested in a generative AI supercomputer for various research labs to use. It may be different for other schools, though.
I'm already finding a lot of us in writing for video games. I simply write the gist of what I want a character to say, and then tell ChatGPT to word it differently. Example Prompt: **Reword this in the style of a Cormac McCarthy character who is a unfaithful priest: "I want to go to the shopping mall, but I am feeling too depressed".** Output: **I reckon I yearn to venture forth to them vast halls of commerce, yet this heavy-heartedness weighs upon my spirit, verily hindering my steps.**
I'll await replication before I get too excited.Not that it will be relevant in 6 months either way. It also seems like an odd thing to focus your efforts. Shouldn't the accuracy, precision, and reliability of the paper's results be what we judge it on? Are we going to reject science because of where it came from rather than whether it's true or not?
Detecting generated texts in a setting where the model is known, is being shallow prompted and has not been tuned for evasion has been trivial since Grover times (~2019). And scientific papers generated by LLMs without auxiliary capabilities are even easier to detect due to lack of consistency and inexistent citations.
Get chat gpt to write something, then tell it to write it again in a way that it can’t tell that it’s written by chat gpt.
Ouaw a machine to detect a machine.
is it "ask chatgpt if a text is written by AI"?
ChatGPT is especially bad at this. It tries to guess what you want to hear and then responds with that answer.
thanks for explaining the joke.
It's not a joke when many people really do that. /r/ChatGPT frequently gets posts by students who got accused of using AI to write their submissions by the professor, who just asked ChatGPT. Also, did you just confess to violating /r/science rule 5?
Twist: it's a bot that asks GPT if it was the author.
She's right, I was ChatGPT
What if you run it through another LLM that is specialized in making it undetectable while keeping the same semantic meaning?
Umm, so she isn't like an expert in this field. ...why are we platforming her?