AI · Article
AI Reading and Editing Neural Network Weights Is Not Superhuman, Human Researchers Can Do It Too
GeekNews ·
At a glance
A report said GPT-6 Astra looked only at the weights of a small game neural network, figured out which game it was, and edited the weights to improve another model's performance. However, a counterargument says this work can be done by skilled human researchers, so it is hard to call it superhuman.
Key points
- Weights are numbers that represent the strength of connections inside a neural network. In game networks, each screen cell has a known meaning, so the size of the weights alone can provide clues.
- The Qwen 0.8B model had a problem repeating the same phrases after 32k tokens (units of text fragments). The team fixed it by reducing the weights that caused the repetition.
- This method finds relevant weights by comparing repeating sentences with non-repeating ones, so human researchers can follow the same steps.
- The author believes AI's contribution to research lies in improving the training process rather than magically editing weights.
Why it matters
It helps readers judge realistically, without exaggeration, how far AI's ability to improve models on its own actually goes.
Original source
LLM은 신경망 가중치를 읽고 쓰는 데 초인적이지 않다
GeekNews · Nov 5, 05:38 KST
Read on GeekNews ↗AI summarized the original article. Check the original for accuracy. Copyright belongs to GeekNews.
More from this field
- MIT and IBM Team Probes Gaps in Multimodal AI Understanding
MIT and IBM researchers are studying where today's multimodal AI models fall short, even when they handle text, images, and sound fluently. Their team, co-led by MIT's Jim Glass, builds new benchmarks and training methods that need less labeled data. This Q&A walks through how the project started and how it has evolved.
MIT News · Nov 21, 06:00 KST - Leaky AI Supply Chain Exposes Secret User Data at Alarming Rate
A research test sent nearly two million prompts with fake but working secrets through a network of AI services to see where they ended up. Many of the secrets were stolen, and attackers used them to reach cloud accounts. The results point to weak protection of customer data along the chain.
Ars Technica · Nov 21, 05:59 KST - OpenAI Shows How to Claim a Shopify Store in ChatGPT
This video shows how to claim your business in ChatGPT by connecting your Shopify store. You can start in Business Hub or the Shopify App Store. After connecting, you can manage your business information to control how your store appears in ChatGPT.
OpenAI 공식 YouTube · Nov 21, 05:48 KST - OpenAI Sharply Cuts Success of Attacks on GPT-6.2 Astra and Sol Reasoning
OpenAI has greatly strengthened its defenses against attempts to pull the hidden reasoning out of its latest models, GPT-6.2 Astra and GPT-6.2 Sol. In its own internal tests, the success rate of these attacks fell sharply, and OpenAI says it is now bringing some of these improvements to GPT-6 Astra too.
The Decoder · Nov 21, 05:31 KST