Can an AI chatbot handle a history homework assignment? | Test

This post is a supplement to the video we published on our Beyond AI YouTube channel. We highly encourage you to watch the video version.

Watch this material on YouTube:

Can ChatGPT do your homework for you?

Hi, I'm Michał, and I'd like to tell you about a test I conducted yesterday evening. The question was: can ChatGPT do your Polish or History homework for you?

It turned out that my son had to prepare a large presentation for History. The topic was: "How Poland fared in numerous war conflicts under the rule of the Vasas." This was not a simple topic that could be developed based on a single source.

We decided to use AI tools. We tested three paid solutions: ChatGPT, Gemini Ultra, and Perplexity AI. Here is how they handled the task:

1. The first approach:

We asked each of the tools to take on the role of a specialist in 17th-century Polish history and summarize the events during the reigns of individual kings. At first glance, we received interesting slides or outlines, but upon closer analysis, it turned out they contained many inaccuracies and fabricated facts.

• ChatGPT missed six major conflicts from the reign of Sigismund III Vasa.
• emini listed five, not necessarily the most important, conflicts from the entire 17th century.
• Perplexity AI presented facts selectively.

When we asked for more details, each tool confused dates and wars, which undermined our trust in these solutions.

Example of Gemini Advanced:

Example of ChatGPT

2. Test with Microsoft Copilot:

We asked Copilot to create a chronological list of all conflicts in the 17th century. Unfortunately, the list was not chronological, and some wars were repeated several times.

3. Changing the approach:

We decided on a different method. Instead of relying on the general knowledge of the AI models, we started:

• Reading Wikipedia and educational materials,
• Extracting the most important information from articles about the wars,
• Asking the AI for a summary according to specific assumptions (e.g., for each war: causes, course, consequences).

First, we created a general vision of the presentation, and then we prepared detailed slides. In this way, a presentation based on facts was created, without distortions and with real educational value.

4. Test with school literature:

We also checked how AI tools handle less popular required reading, such as "Konrad Wallenrod." Every tool confused the characters and was unable to correctly summarize the main plot points.

Conclusions:

1. AI tools (ChatGPT, Gemini Advanced, Perplexity AI, Copilot) have difficulty accessing specific knowledge, e.g., regarding Polish history or lesser-known literature.
2. They are prone to hallucinations and creating false facts.
3. They cannot be used as a reliable source of knowledge in such cases.
4. They can be useful as "text calculators"—for processing, shortening, or summarizing information provided to them.
5. In the case of specialized topics, it is faster and more accurate to develop the material yourself based on reliable sources than to rely on AI.

In summary, AI tools can be helpful in certain aspects of homework, but they will not replace critical thinking and your own analysis of the material.

We cordially invite you to follow our Beyond AI channel, where we regularly share the latest news, analyses, and trends from the world of artificial intelligence.

Whether you are just starting your adventure with AI or are an experienced specialist, you will find valuable content with us that will help you better understand and utilize the potential of this technology.

Join our community and stay up to date with what is happening in the world of AI!