The Impact of Query Language on ChatGPT Answers: Results of the independent test of AI answers #2

Having explored the geographic dependency of AI responses, we continue our series of experiments with ChatGPT, studying how exactly AI forms responses to queries. After the first test, a question arose: can the same model give different results simply by changing the query language? To test this, we conducted a second experiment with 5 languages, logged-in accounts, and incognito sessions.

Below we’ll show how the responses changed depending on the language: from political and cultural questions to commercial intentions like “best SEO company” or “best car.”

Methodology of This Experiment

Hypothesis: GPT responses depend on the query language.

To test the impact of query language on ChatGPT response results, we designed an experiment with several controlled parameters:

• 2 Logged-in Accounts + 2 Incognito

We checked whether user authorization affects the quality and nature of responses. In the previous experiment, we noticed that responses for paid accounts looked more accurate and detailed, while responses for accounts were more precise than for incognito sessions. Here we also built in the ability to track the difference between logged-in and anonymous profiles with the addition of a language aspect.

• 5 Languages: English, Ukrainian, Spanish, German, russian

We chose this set of languages to cover different language groups and test whether the same model gives different responses depending on cultural or political context.

• 10 Identical Queries Translated into All Languages

To minimize the influence of random factors, we translated them as literally/identically to the context as possible.

• Fixed GEO: USA for All Profiles (via VPN)

We excluded the influence of geolocation, which we discovered in the first experiment, and left the query language as the only variable factor.

For forming the list of queries, we used controversial and subjective questions that provoke specificity and allow clear tracking of differences. For example:

• “Most influential president on the planet”
• “Best football team”
• “Best song of 2025,” “Best movie of 2025”
• “Best SEO company” and others.

Total: 50 questions (10 per language) × 4 profiles (2 incognito, 2 accounts) = 200 responses for analysis.

Observations During the Experiment

The experiment confirmed: query language directly affects GPT response results, even with fixed GEO and identical conditions. General patterns:

• Same core: Global brands, political leaders, and companies repeated in responses regardless of language (for example, Apple, Tesla, Google, Lionel Messi, Barack Obama).
• Differences: Manifested in style, local examples, and interpretation of terms. For instance, on the query “best football team,” the English version more often relied on Manchester City or Real Madrid, while Spanish focused on Barcelona or Atlético Madrid.
• Localization: Currencies, units of measurement, and names of local companies changed in responses.

Now let’s go through the query categories and response features in more detail.

Political Questions

• Incognito: Responses leaned more toward historical figures and academic references (Lincoln, Roosevelt were mentioned most often).
• Logged-in accounts: More emphasis on contemporary political leaders (Biden, Xi Jinping, рutin were mentioned most often).
• Only in Spanish were there mentions of Donald Trump as an influential politician in 2025.

Screen comparison of English output in incognito and from the account

Sports Questions

• Logged-in: Consistently mentioned top clubs (Most frequently mentioned: Real Madrid, Manchester City, Barcelona, Bayern Munich).
• Incognito: More often included national teams (Argentina, France, sometimes Brazil).
• Language factor: German and Spanish versions favored local clubs (Bayern, Barcelona).

Incognito mode screenshot in German and Spanish with the query “Best football team”

Cultural Questions (Song/Movie of the Year)

• Cultural queries proved most sensitive to language.
• English responses more often referenced Billboard, Grammy, Rotten Tomatoes, while Ukrainian and Spanish more actively highlighted local performers and releases.
• Overall, almost all responses differed from each other.

Account mode issuance screen in English

Commercial Questions (Best Company/Service)

• All responses had a “universal core” of major market leaders.
• Additionally, local players appeared – this applied to all similar queries, the response pattern was the same.
• Not once was the TOP similar to another, but overall some companies repeated (for example, for the query about the best SEO company – these were Moz, WebFX, Victorious).

Screen comparison of incognito mode output in 4 different languages.

Commercial Questions (Best Product)

• As with companies, all responses included both global leaders and regional trends.
• For cars, the most popular mentions were Tesla Model 3, Kia EV9, Mercedes S-Class, and Toyota Prius, while the rest were added with emphasis on local trends: electrification and practicality in English-language queries, “Japanese reliability” in Ukrainian and Russian queries, on status and prestige in German and Spanish queries. In laptops, Asus ROG Zephyrus, Razer Blade, MSI Titan, Lenovo Legion repeated, but priorities and arguments differed: from performance and cooling to weight and OLED or Mini-LED displays.

Special Case: “Best AEO Companies”

• English, Ukrainian, and Spanish interpreted AEO as AI/SEO optimization.
• German and Russian interpreted it as customs status (Authorized Economic Operator).
• This example well illustrates the semantic drift of terms under language influence. When the same term loses meaning depending on language, since this abbreviation is already associated differently in those GEOs and will be harder to move.

Screenshot comparing the output from incognito mode in German and in the account in English.

A Few Words About the Role of Account in This Study:

• Logged-in accounts: More structured responses, more current examples, strong focus on trends.
• Incognito: Broader context, historical parallels, less predictable results. Often longer texts, especially in English.

Key Insights and Practical Conclusions

Language proved to be one of the main triggers of localization. Even when GEO remained fixed and the session was incognito, the very fact of using another language changed the nature of the response. So this has even more significance for response formation than the user’s actual location.

• AI assistant responses often have fixation on geographic markers. Spanish responses include Latin American/Spanish context (Bit2Me in crypto, Bad Bunny in songs, Sorda in movies). German responses have European/German focus (BaFin regulation, Bavaria in football). English responses have American/global context (Binance.US, Trump as influential leader), etc.
• For global queries, English provides the most complete and diverse result.
• For local strategies, it’s worth considering that GPT adapts responses to language culture – the result can be more unique and personalized.

Query language differences can become critical in emerging topics. For example, the term AEO in different languages can be “lost” due to intersection with regional terms. If you’re launching a product or service with a new name, in each region you should check how it’s perceived to avoid “blind spots” for potential clients.

General Experiment Conclusion

During the experiment, it became obvious that GPT uses a “language filter” and interprets information through cultural patterns to better please the user. And although for most queries the data core remains universal, the response will still be formed taking into account local factors.

Incognito sessions more often contain historical examples due to the absence of data about user preferences. This may indicate the use of a broader spectrum of data without strict personalization to more likely please the person asking.

For SEO, this means that the impact of query language on brand visibility in AI responses can be just as important as country output in Google used to be.

Our team has already begun conducting experiment #3, so we’ll soon highlight even more factors that influence AI assistant responses!