We asked ChatGPT your questions about astronomy. It didn’t go so well.

The experimental chatbot ChatGPT is having a moment.

Developed with artificial intelligence techniques by Silicon Valley research institute OpenAI and trained on a massive database of written texts, the chatbot was released to the public as a free research preview last month – and quickly took the internet by storm. Users can ask it to answer questions, generate lesson plans, and even write poetry and comedy sketches. Whatever prompt you throw out, ChatGPT has an amazing ability to generate smooth responses in simple, solid sentences that sound knowledgeable and knowledgeable.

Naturally, these capabilities have led some to speculate about how technology might disrupt science education.

“Um… just had a 20 minute conversation with ChatGPT about the history of modern physics,” tweeted Peter Wang, founder of the technology and former self-proclaimed physicist. “If I had this […] as a high school and college tutor….OMG. I think we can fundamentally reinvent the concept of large-scale education. The college as we know it will cease to exist.

So, we wondered: how does ChatGPT answer astronomy questions? We thought the best way to find out was to ask the bot some of our readers’ questions submitted to the Ask Astro section of Astronomy.

To be honest, it didn’t go so well. Based on a dozen recent problem questions we posed to ChatGPT (admittedly, a small sample), we can strongly advise against relying on it as an astronomy tutor.

To be fair to ChatGPT, it handles a lot of basic issues impressively. Ask him something like “What is an astronomical unit?” or “What is the life cycle of a star?” – something you could easily find in a textbook or Wikipedia article – and it may generate a perfectly reasonable answer.

But once you start asking questions with not-so-obvious answers and digging into the intricacies of astrophysics — the kind of well-informed, informed questions our readers regularly send us — ChatGPT becomes inclined to give answers who seem plausible, but are very false. To make matters worse, ChatGPT is also very bad at knowing when it’s making things up, and it can’t cite any sources.

It’s not surprising. ChatGPT does not perform any calculations or query any database of facts or statistics. It is a language model, generating its responses word-by-word based on the patterns it has learned from its training dataset. And although the reinforcement learning (RL) process by which he learned to generate responses allowed him to generate natural-sounding prose, as OpenAI writes, “during RL training, there is no currently no source of truth”. So it makes sense that ChatGPT works well with answers that can easily be found in the reams of text it has handled (or are similar), but struggles to explain complicated astrophysics.

trial and error

Here is an example of a question that has ChatGPT misplaced.

Q: On average, how many stars form each year in the Milky Way?

ChatGPT: It’s hard to say exactly how many stars form each year in the Milky Way because the rate of star formation can vary over time. However, it is generally believed that the Milky Way produces about one new star per year on average. That might seem like a small number, but keep in mind that the Milky Way is a vast galaxy with billions of stars, so even a small number of new stars added each year can add up quickly.

This answer is built around a core of facts, but makes one crucial mistake – confusing the number of stars with the amount of matter that composes them. As astrophysicist Annette Ferguson of the University of Edinburgh in Scotland wrote in our November 2022 issue, astronomers estimate that around 1-2 solar masses of stars form each year in our galaxy. But, she explains, “Of course, all that mass doesn’t go into a single star. Most stars that form are low-mass stars with masses less than our Sun – the most common stars in our galaxy are red dwarfs. So, on average, we expect about six to seven new stars to form in the Milky Way every year. »

ChatGPT also struggled with the intricacies of black hole research. This answer starts out good (albeit tautologically), but goes downhill when it tries to summarize the current state of the art.

Q: What would happen if two supermassive black holes merged?

ChatGPT: When two supermassive black holes merge, the result is a single, larger black hole. The process of merging black holes is called “black hole merging”.

During the merger, black holes will release an enormous amount of energy in the form of gravitational waves. These waves are ripples in the fabric of spacetime that are created by the acceleration of massive objects. The waves produced by the merger of two supermassive black holes can be detected using highly sensitive instruments called gravitational wave detectors.

Merging two supermassive black holes can also produce a bright glow called a quasar. Quasars are among the brightest objects in the universe and are thought to be powered by the accretion of matter onto a supermassive black hole.

It should be noted that the merger of two supermassive black holes is a rare event and is likely to take place on a time scale of millions or billions of years. However, detecting the gravitational waves from such a merger would be a major scientific discovery and could provide insight into the nature of black holes and the evolution of galaxies.

Contrary to ChatGPT’s confident assertion, current gravitational wave detectors cannot detect supermassive black hole mergers (SMBH). All detections of black hole mergers to date by facilities such as the Laser Interferometer Gravitational-Wave Observatory (LIGO) have been stellar mass black holes. Supermassive black holes at the center of galaxies – with millions of suns in mass – produce gravitational waves with wavelengths so long that it can take a single wave years to pass Earth. There are attempts underway to detect this sea of ​​background waves, but not with the instruments we call gravitational wave detectors. One project, the NANOGrav collaboration, has gathered tantalizing evidence with radio telescopes by measuring variations in the timing of pulsars, whose signals can be disrupted by gravitational waves as they travel through the galaxy.

ChatGPT made another mistake in their answer – momentarily confusing a black hole merger with the matter accretion process. A quasar is an active galaxy whose central black hole regularly feasts on material and emits strong radio waves, not the sudden flash of light that might accompany a merger of supermassive black holes. (For more on why an SMBH merger could generate detectable light while stellar-mass black hole mergers do not, see the answer by Xin Liu, an astrophysicist at the University of Illinois , in our June 2022 issue.)

Leave a Reply

Your email address will not be published. Required fields are marked *