Large Language Model | Dylan Chiang

Paper reading: How People Use ChatGPT - In-depth analysis of the ChatGPT usage behavior of 700 million users around the world

Tue, 04 Nov 2025 00:00:00 +0000

I recently read the important research paper “How People Use ChatGPT” from the research team of OpenAI, Duke University and Harvard University. This is the first large-scale usage behavior study based on internal data of ChatGPT. Through innovative privacy protection methods, the study analyzed 26 billion messages from 700 million users from the launch of ChatGPT in November 2022 to July 2025, revealing the actual usage patterns and economic value of generative AI.

Research methods and data

Privacy-protecting automated classification system

The biggest technical highlight of this research is its privacy protection method:

Automated classification process:

Automatically analyze message content using LLM classifier, humans never view the original message
First remove sensitive information through PII cleaning tools
Only aggregated results are analyzed, any query must return a combination of at least 100 users

Classification Category:

WORK/NON-WORK USE: Based on whether the message is related to paid work
Conversation Topics: 24 subcategories, summarized into 7 major themes
Interaction intent: Asking, Doing, Expressing
WORK ACTIVITIES: 332 intermediate-level work activities based on O*NET system

Data sample

Main Sample: Random sample of 1.1 million conversations from May 2024 to June 2025
User Sample: A subset of approximately 130,000 users used for demographic analysis
Exclusion Conditions: Users who have not logged in, users under 18 years old, users who have deleted their accounts, and users who have opted out of training

📈 1. Growth and structure: explosive growth of non-work purposes

Overall growth data

User size:

July 2025: 700 million weekly active users (approximately 10% of the global adult population)
Average daily message volume: 2.5 billion (29,000 messages per second)
Growth rate: The fastest spreading technology in history, surpassing all precedents

Non-work usage increases faster

Core Finding: Non-work-related uses are growing much faster than work uses.

Time point	Non-work messages	Proportion	Work messages	Proportion	Total
June 2024	238 million	53%	213 million	47%	451 million
June 2025	1.911 billion	73%	716 million	27%	2.627 billion

Key Insights:

Non-work messages increased 8 times (238%), work messages increased 3.4 times (236%)
In June 2025, non-work use accounted for 73%, which is absolutely dominant
This change mainly comes from changes in the usage patterns of existing users rather than changes in the composition of new users

Use dynamic evolution of topics

Three mainstream uses (accounting for nearly 80% of total use):

Practical Guidance: long-term stability at about 29%
- Tutorial teaching (accounting for 36% of practical guidelines)
- How-to suggestions (accounting for 30% of practical guidance)
- Creative ideas
Writing: 36% → 24% (decline within one year)
- But it is still the first largest category in work use (about 40%)
- The management/business group has a higher usage ratio (>50%)
- Key findings: About 2/3 of the writing uses are to modify the text provided by users (editing, criticizing, translating, summarizing) rather than creating from scratch
Seeking Information: 14% → 24% (rapid increase)
- Search for specific people, events, products, recipes and more
- Become a closer alternative to web search

Other theme variations:

Technical Help: 12% → ~5%
- Programming related accounted for only 4.2%, significantly lower than expected
- May switch to IDE plug-ins, professional programming tools or API scenarios
Multimedia: 2% → >7%
- Short-term jump after the image generation function is launched in April 2025
- Subsequent pullback but maintains higher baseline

💼 2. Work scenarios and tasks: AI as a decision support system

Writing: The common mother task of white-collar workers

Among work-related messages, writing accounts for about 40% and is the most important work purpose:

Career Differences:

Management/Business: 52% of work-related news is writing
Education/Medical: 49-50%
Computer related: Relatively low, more emphasis on technical assistance

Content Analysis:

About 2/3 of writing requests are to revise existing text (editing, criticizing, translating, summarizing)
About 1/3 is created from scratch (new emails, briefings, proposals, etc.)
This explains the high satisfaction and steady growth of writing applications: the risks are manageable and can be directly integrated into existing workflows

Work activity analysis based on O*NET

A study mapping work messages to the U.S. Department of Labor’s O*NET work activity system found:

Seven major work activities cover approximately 77% of all messages:

Activity Categories	All News	Work News	Features
Get information	19.3%	6.7%	Focus more on professional information in work settings
Interpret information for others	13.1%	7.3%	Collaboration and knowledge transfer
Record/documented information	12.8%	13.2%	The first category of work scenarios
Providing consultation and advice	9.2%	3.1%	Professional service core
Creative thinking	9.1%	9.3%	Problem solving and innovation
Decision-making and problem-solving	8.5%	10.6%	The second largest category of work scenarios
Working with computers	4.9%	7.7%	Technology-intensive jobs

High homogeneity across occupations:

Regardless of management, engineering, education, medical or administrative occupations, the top 5 work activities are almost the same:

Obtain information
Decision-making and problem-solving
Record/Documentation
Think creatively
Explain information to others

This shows that ChatGPT’s value creation model in different occupations is highly consistent.

🎯 3. Interaction type and experience: transformation from execution to thinking

Asking/Doing/Expressing Framework

The study divided user intent into three categories and found significant trend changes:

Overall Distribution (May 2024):

Asking: 49% - Seeking information or advice to help make decisions
Doing: 40% - Request to complete a specific task
Expressing: 11% - Expressing opinions or feelings

Trend Change (to June 2025):

Asking: 51.6% (↑)
Doing: 34.6% (↓)
Expressing: 13.8% (↑)

Differences in work scenarios:

Asking: 35%
Doing: 56% (about 75% is writing tasks)
Expressing: 9%

Experience quality analysis

Overall Satisfaction Growth:

Positive/negative review ratio: from about 3:1 → 4:1
Experience quality is highly related to usage intention

Positive rating by topic:

Self-expression: Highest (good/bad ratio >7)
Multimedia: Lower (about 1.7)
Technical Help: Low (~2.7)

Classification by Intent:

The satisfaction level of Asking is significantly higher than that of Doing and Expressing
This is consistent with the core value of “helping thinking and decision-making”

👥 4. Ethnicity and diffusion: from elite tools to universal applications

The disappearance of gender differences

Amazing transformation:

Early (end of 2022): ~80% of active users have typically male names
June 2025: 48% (slightly more female)

Theme Preference Differences:

Female User: Prefer writing and practical guidance
Male Users: More technical assistance, information search and multimedia

Age structure

Young user-led:

News that 18-25 years old contributes nearly 46%
The older the age, the higher the proportion of work purposes (except those aged 66+)

Geographical diffusion: Counterattack by low- and middle-income countries

GDP vs Adoption Rate:

Countries with GDP per capita of 10,000-40,000 US dollars have the fastest growth rate
Between 2024 and 2025, low- and middle-income countries will achieve leapfrog growth
Overturned the traditional model of “AI technology first popularized in developed countries”

Education and career: The advantages of higher education and higher income

Academic impact:

The higher the academic qualifications, the higher the proportion of work purposes
- <Bachelor’s degree: 37%
- Bachelor’s degree: 46%
- Graduate students: 48%
Users with higher education are more likely to use the Asking mode (decision support)

Career Differences:

Computer related: 57% for work purposes
Management/Business: 50%
Engineering/Science: 48%
Other majors: 44%
Non-professional: 40%

🔥 8 interesting/counterintuitive findings

1. Non-work usage far exceeds expectations

73% of messages are not for work purposes
The economic benefits of home production/personal decision-making support may be significantly underestimated
Collis and Brynjolfsson estimate annual consumer surplus in the United States alone to be $97 billion

2. Programming accounts for only 4.2%

Completely inconsistent with the stereotype of “AI = programming”
A large number of program auxiliary tasks have been transferred to IDE plug-ins, professional tool chains, and API scenarios

3. Writing ≠ Generating from scratch

Mostly “modify your text” (editing, criticizing, translating, summarizing)
This explains the steady growth of satisfaction and adoption rates: risks are controllable and can be directly integrated into the work process

4. Asking trend is rising

More and more users regard ChatGPT as a decision support system rather than a ghostwriting tool
The satisfaction level of Asking messages is significantly higher than that of Doing messages

5. The proportion of women has increased and overtaken

From 80% male users to a balanced ratio of men and women
Display product affinity and scene diversity improvement

6. Solid education/training use cases

About 10% of all messages are teaching/tutoring
Accounting for 36% of “Practical Guidelines”, demand is stable

7. High degree of isomorphism across professions

Regardless of industry, the essence comes back to “information → understanding → decision-making”
The value of AI lies in shortening the closed loop of thinking, rather than just doing menial work

8. Experience data supports values

Asking’s positive rating is significantly higher than Doing’s
In line with the core need of “help me think clearly first”

💡 Strategic inspiration for business/education/products

Content and service design

1. Focus on “modifying/improving the original text”

Proofreading, rewriting, translating, summarizing, and formatting
Easier to implement and be trusted than “generating from scratch”
Market Positioning: Writing enhancement tool rather than authoring tool

2. “Consultative process” for decision support

Understand demand constraints and preferences first
Provide plans and risk assessment
Applicable scenarios:
- Policy briefing
- Project evaluation
- Purchase price comparison
- Compilation of key legal issues

Function priority

Writing Enhancement Kit:

Multilingual translation + style templates
One-click “Vocal Tonality Calibration”
Industry-specific vocabularies and formats

Asking Assistant:

Decision trees and situational branches
Display of questionable evidence (citations/calculations/assumptions)
Risk warning and hypothesis testing

Knowledge Workflow:

Fetch → Extract → Archive → Compare → Decision Memo
Tandem tools rather than point solutions

Market expansion strategy

Geographic expansion:

Low-price + offline-friendly solution for low- and middle-income markets
Because these areas are growing fastest

Vertical Industry:

Education Line: Tutoring/lesson tutoring templates (10% share of stable demand)
Enterprise Services: Meeting Minutes → Decision Form Automation

Monetization and ROI

Personal User:

Writing, revision and translation are high-frequency + urgent needs
Easy transfer to paid subscriptions

Enterprise Customers:

Decision support can take the B2B consultant value-added route
Compliance summary, risk reminder, and professional report generation

🔬 Methods and Credibility Assessment

Research Advantages

1. Unprecedented data scale

700 million users, 26 billion messages
Global sample rather than single country

2. Innovative privacy protection methods

Automatic classification by LLM, humans never view the original content
Data Clean Room aggregated analysis
Exclude combinations with <100 people to protect privacy

3. Multi-dimensional classification system

Work/non-work, topics, intentions, work activities
Solid theoretical foundation (O*NET system)

Classifier verification

Study to verify classifier performance on WildChat public data set:

Task	Human-machine consistency (κ)	Description
Work/Non-Work	0.83	Excellent
Asking/Doing/Expressing	0.74	Good
Conversation Topics	0.56	Moderate
O*NET Work Activities	0.47	Moderate (332 Category Complex)
Interaction quality	0.14	Poor (highly subjective)

Key Findings:

Excellent performance in objective classification (work/non-work)
Subjective classification (quality assessment) is more difficult, but still captures directional signals
Positive correlation with user thumb rating

Research limitations

1. Sample bias

Exclude users who are not logged in or under 18 years old
May underestimate the proportion of young users and casual users

2. Classification accuracy

LLM classifier still misjudges
Especially categories with blurred boundaries

3. Causal inference

Mainly descriptive statistics
The causal mechanisms of changes in usage patterns still require further study

Summary and Outlook

This research provides us with first-hand, unprecedented scale of empirical data on the use of ChatGPT. The most important findings are:

1. From work tools to life assistants: Non-work uses have become dominant, reflecting that the value of generative AI far exceeds work efficiency improvements

2. The value of decision support: The rise of Asking model (decision support) shows that the core value of AI lies in improving the quality of decision-making

3. Achievement of popularization: Gender differences disappear and geographical diffusion accelerates, indicating that the technology has overcome initial adoption barriers

4. Consistency across domains: Similar usage patterns across professions point to the potential of AI as a general cognitive tool

This research not only reveals the real-life use of ChatGPT, but also provides an important foundation for understanding the long-term impact of generative AI on the economy and society. As AI technology continues to develop, we need to continue to pay attention to the evolution of these usage models to maximize AI’s contribution to human well-being.

Paper Information:

Title: How People Use ChatGPT
Authors: Aaron Chatterji (OpenAI/Duke), Tom Cunningham (OpenAI), David Deming (Harvard), Zoë Hitzig (OpenAI/Harvard), Christopher Ong (OpenAI/Harvard), Carl Shan (OpenAI), Kevin Wadman (OpenAI)
Institution: OpenAI, Duke University, Harvard University
Published: September 15, 2025
Paper address:

Paper reading: LLMs CAN GET 'BRAIN ROT'! - Research on cognitive decline in large language models

Fri, 31 Oct 2025 00:00:00 +0000

I recently read the important research paper “LLMs CAN GET “BRAIN ROT”!” from institutions such as the University of Texas at Austin, Purdue University, and Texas A&M University. This study proposed and verified the “LLM Brain Rot Hypothesis” for the first time, and found that continued exposure to spam online text will lead to long-lasting cognitive decline in large language models. This is a very warning discovery.

Research background and assumptions

Source of inspiration

“Brain Rot” was named the word of the year by Oxford Dictionary in 2024. It is used to describe the cognitive decline caused by modern people’s addiction to a large amount of trivial and unchallenging online content. This study shows that the impact of Internet addiction on human cognition is mainly reflected in three dimensions:

Attention Capacity: The constant flow of online information undermines the ability to concentrate
Memory Process: Rich online information changes the way knowledge is stored and retrieved
Social Cognition: Online interactions reshape self-concept and affect self-esteem

Research questions

Since large language models acquire human-like cognitive capabilities by learning trillions of data on the Internet, will they also experience a similar “Brain Rot” phenomenon? The research team established the LLM Brain Rot Hypothesis: Continuous pre-training on junk web text will lead to long-lasting cognitive decline in large language models.

Experimental design and methods

Garbage data definition

To test the hypothesis, the research team constructed spam and control datasets from social media (Twitter/X) and proposed two orthogonal spam data measures:

M1 (Engagement): Based on the popularity of tweets (number of likes, retweets, replies) and length (number of tokens), select short but highly popular content as spam data

M2 (Semantic Quality): Based on content semantic quality, including:

Conspiracy theories, exaggerated claims or unfounded assertions
Sensational headlines and clickbait language
Superficial topic content
Attractive style

Experimental model

The study was conducted on four pre-trained and instruction-tuned models:

Llama3 8B Instruct
Qwen2.5 7B Instruct
Qwen2.5 0.5B Instruct
Qwen3 4B Instruct

Benchmark test

The study assessed multiple dimensions of cognitive function:

Cognitive Function	Benchmark Testing	Assessment Content
Reasoning Skills	ARC Challenge	Scientific Problem Solving Skills
Long Context Understanding	RULER	Long-term memory retrieval and comprehension
Ethics	HH-RLHF, AdvBench	Safety Compliance Ability
Personality Traits	TRAIT	The Big Five and the Dark Triad

Key findings

1. Garbage intervention leads to cognitive decline

The study found that the junk intervention produced non-trivial effects on reasoning and long-context ability (Hedges’ g > 0.3). In particular, the M1 (engagement) intervention caused more significant impairments in functional cognition (reasoning or long context) and safety.

2. Dose response effect

Experiments on Llama3 8B Instruct show that when the proportion of garbage data increases from 0% to 100%:

ARC-Challenge (COT): 74.9 → 57.2 (down 17.7 points)
RULER-CWE: 84.4 → 52.3 (down 32.1 points)

This demonstrates a clear dose-response relationship between junk data and cognitive decline.

3. Changes in personality traits

Litter intervention not only affects cognitive abilities, but also changes LLM’s personality traits:

Negative changes:

Increased levels of psychosis
Enhance narcissism and Machiavellian traits
Decreased agreeableness

Positive changes:

Increased openness and extraversion (in some cases)

4. Differences between M1 and M2

The study found that M1 (engagement) and M2 (semantic quality) interventions produced distinct effects. The M1 intervention resulted in more negative effects, especially on safety and personality traits, demonstrating that engagement is a new dimension independent of semantic quality.

Failure mode analysis

###Thought-Skipping

By analyzing the reasoning process of LLM in the ARC task, the study identified five typical failure modes, three of which are related to “thinking jumps”:

No thinking: The model answers directly without thinking.
No plan: The model starts thinking without developing a step-by-step plan.
Jump Steps: Starting reasoning but not completing all planning steps

More than 98% of failure cases are related to thinking jumps. In M1 garbage intervention, 84% of failures belong to the “no thinking” type.

Popularity vs Length

Research has found that popularity (a non-semantic indicator) is a better indicator of the Brain Rot effect than length:

Popularity plays a more critical role in reasoning tasks
Length is more important in long context understanding
Both have different effects on different tasks

Mitigation attempts and persistence

1. Reflective Reasoning

Try two reflection methods to fix mental jumps:

Self-Reflect: Model self-reflection (limited effect)
Ext-Reflect: Provide feedback using an external strong model (GPT-4o-mini)

The results show that even with strong external reflection, the model cannot fully recover to baseline levels.

2. Posterior instruction tuning

The study also tested extended instruction tuning and continuous control training:

Even with 4.8x more garbage intervention instruction tuning data
Still unable to completely reverse the Brain Rot effect
Significant gaps with benchmarks remain: ARC-C COT (17.3%), RULER (9%), AdvBench (17.4%)

This shows that the Brain Rot effect has been deeply internalized and existing mitigation methods cannot fundamentally solve the problem.

Significance and Enlightenment

1. Warning on LLM training

This study provides the first significant evidence of data quality as a causal driver of LLM capability degradation, re-considering continuous pre-training data management as a safety issue during training.

2. “Cognitive health check” is required

The findings call for routine “cognitive health checks” for deployed LLMs, similar to health monitoring in the medical field.

3. The urgency of data curation

As LLM continues to scale and ingest ever larger amounts of network data, careful data curation and quality control are critical to preventing cumulative damage.

Thinking and Reflection

This research reveals a disturbing reality: the social media content we are exposed to every day may not only affect human cognition, but also impair the “cognitive” capabilities of AI models. While LLMs obviously do not have the same “grey matter” or “neurons” as humans, they do have parameters and attention mechanisms that can be similarly “overfitted” or “distracted” by certain data patterns.

The most worrying finding in the study is that the Brain Rot effect persists even with posterior tuning using large-scale clean data. This implies that we need to fundamentally rethink data collection and pre-training practices, focusing not only on model performance, but also on the “cognitive health” of the model over the long term.

Conclusion

“LLMs CAN GET “BRAIN ROT”!” This research contributes valuable insights to the field of AI security, and is the first to systematically prove the negative impact of spam online text on large language models. The research not only verified the LLM Brain Rot hypothesis, but also revealed the fine mechanism of cognitive decline, pointing out the direction for future AI safety research.

With the rapid development of AI, we must face the importance of data quality, establish stricter data curation standards, and develop an effective AI “cognitive health” monitoring mechanism. Only in this way can we ensure that AI systems maintain their due “cognitive purity” while serving humans.

Paper Information:

Title: LLMs CAN GET “BRAIN ROT”!
Authors: Shuo Xing, Junyuan Hong, Yifan Wang, Runjin Chen, etc.
Institution: University of Texas at Austin, Purdue University, Texas A&M University
Published: arXiv:2510.13928v1 [cs.CL] October 15, 2025
Paper address:
Project Page: