When I was in boarding school in Johannesburg, at the peak of my reading habit and intellectual curiosity, I came across a video for Entrepreneurs where Patrick Bet David emphasized that reading the right book at the wrong time can be very damaging.

This was the first time I had considered the possibility that such a positive habit like reading could have significant unintended negative effects. For example, reading The 4-Hour Work Week when you are starting your first job can be very damaging because that is when you’re supposed to be putting in outsized investments of effort and time that compound throughout your career.

The converse is also true by the way – reading the right book at the right time can change your life for the better. There are even more natural extensions to this principle beyond books. For instance, recalling a piece of knowledge from a book, video, or conversation at the right time can transform your experiences.

For instance, I had read in a software engineering book about the importance of organizing two kickoff meetings. The first should involve all stakeholders, including non-technical ones like UX and Legal, to clarify the scope and expectations of the project. This step is often not even considered. The second is for Engineering to clarify responsibilities, deliverables, and timelines. While I could “just remember this” steps, rereading my summary notes when I was starting a new project transformed this from couch advice to career impacting behavior.

What I have described so far is the importance of timing with memory as well as an allusion to the difficulty of solving this problem. The problem is getting the right context (from memory) at the right time.

It turns out this is quite a challenging problem for AI systems to solve today. In fact, I posit it is so big a challenge that this will be the kind of blocker to AGI that will require a significant redesign of the transformer architecture used to power ubiquitous Generative AI technologies.

Quick disclaimer: I do believe we will rapidly experience powerful deployments of Artificial Intelligence that will transform our society significantly. I do NOT believe AGI is around the corner. My favorite definition of AGI here is an AI system good enough that major tech companies like Google and Microsoft fire nearly all of their knowledge workers. Finally, there are folks who are working much closer to this technology on a daily basis and some of them disagree with my position for good reasons. However, given the AGI hype it seems important that folks who are not as close to the research developments are aware that we will need significant breakthroughs just as big as the one brought by the transformer architecture.

In this piece, I want to lay out key facts about the evolution of the architecture used by many LLMs today, the key innovations/gaps needed to cross the chasm to AGI, and finally the implications for consumers, builders, and investors.

Architecture Evolution

Transformers are the transformative technology that has spurred the generative AI revolution, even though it was not a complete solution to the AI research problems of the time.

Recurrent Neural Networks (RNNs) were more or less state of the art pre-2017. They used a feed forward mechanism which used a fixed size hidden state that was “updated” after every embedded token in the input or output. The problem, however, is that there was information loss in these hidden states because too much was packed into them.

Transformers proposed a new architecture which enabled long range dependencies to be better handled because instead of the sequentially updated hidden states, the transformer can pay more or less attention to different parts of the input/output (leveraging multi-head attention and positional encodings) for the token it is currently generating – that is having the right context at the right time.

While the transformer was successful in other significant ways including much faster and cheaper training, as well as incredible versatility beyond text to images and videos, it only partially solved a subset of the problem – having the right context, from the input/output, at the right time.

The distinction between these two problems is well known to AI researchers. Even before the transformer paper, different ideas have been explored to design neural networks that are connected to a read/write long term memory as well as more successful approaches to use Retrieval Augmented Generation (RAG) for getting the right context from storage into the limited context window. All of these efforts acknowledge the importance of context whether it is the context that the model has written to storage to understand a user’s preference or RAG to pull relevant information from the internet.

Transformer Gaps

Beyond the fact that Transformers only solved a subset of the memory/context problem, there are also surface level and fundamental gaps in how LLMs operate today: Catastrophic forgetting, U shaped memory, and Fictitious memory.

Catastrophic forgetting is primarily human-induced in a sense. Remember in 2023 when everyone who cared would complain that AI chatbots had knowledge cutoffs and so would not be able to answer who won yesterday’s FA Cup final. The main solution adopted in industry to address the knowledge problem was continuous pre-training. This process adapts the models to specialized domains, enhances their coding capabilities, and expands heir linguistic ranges too.

However, these pre-training updates change the model’s weights which can lead to a loss in previously acquired capabilities or knowledge – catastrophic forgetting. That said, researchers and AI engineers have implemented Elastic Weight Consolidation (EWC) and Evals to address this problem. EWC does something akin to reducing the plasticity of some neurons for learned tasks when training on new tasks to prevent the forgetting. Eval sets define expected output behaviors for specified inputs which serve as some sort of unit testing to ensure old functionality is not eroded.

U-shaped memory is a semi-recent discovery in state of the art LLMs which shows that the models perform much better at recalling and reasoning on information when it is nearer the beginning or the end of the context – terrible at the middle, hence the U shape. This research published by Stanford and Cal in 2023 called into question the practical benefits of supposedly enormous context windows many research labs were racing towards.

My favorite quote from the paper:

“to claim that a language model can robustly use information within long input contexts, it is necessary to show that its performance is minimally affected by the position of the relevant information in the input context”

Different solutions have been tried with varying degrees of success. One path researchers continue to investigate is improving the positional encodings used in transformers that i mentioned earlier. Another is to use RAG to retrieve only the most relevant information to add to the context. This is because the U-shaped problems go away as input length decreases. Finally, many prompting guides simply encourage users to keep the most important information or instructions at the start or at the end of their prompts.

Subscribe now

Fictitious memory

Come to think of it, these models actually have no memory at all. When we discuss concepts like AGI and its possibilities, it is important to remember that a lot of our experience of AI memory is really just clever client side engineering nin jitsu. Let’s break this down a bit more.

The biggest manifestation of this is the simple fact that when you chat with an AI bot and send your second message after its response, the model does NOT remember your first message or the response. The client side engineering is that the entire context from uploaded files to previous conversation turns are all passed to the model again to generate the second response.

LLMs are, in some sense, stateless because they have to be reminded of all the context, every time. Its a stateful system, but a stateless model.

One implication is that nothing about the current transformer architecture actually promises learning behavior which is required for an Artificial General Intelligence System. That said, as a quick aside, while transformers have no memory, one could consider the LLM system that includes external memory, RAG, and so on to indeed have memory, but these are layers on top of the transformer architecture.

Another implication is that the behavior problems we experience with reduced creativity after multiple turns is a direct consequence of poor memory architecture. The models get siloed and less creative because it most definitely does not ignore the previous context in the chat. This is the same reason why unrelated information in context can steer the model off course even when accurate information was present in the training set.

There are many paths to Artificial General Intelligence from complex technology systems that may include the transformer to cohesive collectives of human-computer systems and gene-edited humans with neuralink-like connections to computers.

Implications for consumers, builders, and investors.

While we may not know the when/how of AGI we know that transformers are ushering in powerful AI that will transform society in economic, political, and other ways.

For users of GenAI technology, which I expect to include everyone, remember that you are a context engineer. You should give explicit instructions, file uploads and so on that can bring relevant context to LLM to make it much more useful to you. Separate conversations about different ideas into new chats. And, keep your most important instructions near the beginning or end of the context.

For builders, while I might seem skeptical of all the engineering nin jitsu needed in context engineering, I think we need much more of it. There are many latent use cases today that would be possible and impactful if only these models had the right context and the right user experience was presented to the user. The lack of memory of a transformer-based model is not an argument for its limited usefulness!

Finally, for investors, just a word of warning. To put it simply, if your valuations and investments are based on assumptions that your portfolio companies will figure out a path to AGI, reconsider. To reap the outsized benefits that AGI would provide, we would need a breakthrough just as big as the one that resulted in the transformer. And that’s just for the memory problem.

Share

The common thread I am observing is that the rate of building with AI is surpassing the rate of adoption, especially among students. I remember showing students a pdf only prototype and getting them to use any free ppt to pdf converter to add their class slides was such a hiccup.

This suggests that building great tools is not enough. Even democratizing them to be widely accessible is also not enough. If we care about the real impact, we must also invest in meeting students where they are at, demonstrating how users overcome inertia, confusion, or lack of context.

Now on to the first tool.

Gemini Quizzes

My team launched a feature in Gemini that generates practice quizzes that are personalized and grounded on the user’s content. It is available globally both on the web and mobile apps. You can read the full announcement here.

I am incredibly proud of this work because it represents a critical application of AI to an underutilized but impactful lever of pedagogy – retrieval practice.

In ‘Why Technology is Revolutionizing Education’ I explain how Optical Character Recognition technology transformed Education positively by enabling teachers to grade multiple choice questions faster and therefore test students more often.

Research shows significant learning benefits when we recall information during testing. Notably, this approach is incredibly effective once you have at least consumed the content once. This was such a consequential learning that I have since convinced my sister to change from her ways of re-reading the page till she understood to instead moving on the teacher’s practice questions ASAP.

The issue with retrieval is simple and well known. Many studious students run out of practice questions to keep testing themselves. And it is really difficult and time consuming for diligent students to create their own. To make matters worse, in my secondary school there were some teachers who deliberately did not give practice questions (they didn’t know about the power of retrieval for learning.)

Fast forward nearly a century since the adoption of OCR, and we have another technology, AI, which enables much more testing, not only limited to multiple choice formats, and not only for teachers’ uses.

Gemini quizzes are available to students to get their reps in, while also providing feedback at every step, an opportunity to follow up, as well as a summary of the strengths and weaknesses of their performance.

While Gemini helps students by saving time and effort in the studying process, other tools are integrating directly into their daily routines. A subset of the team behind NotebookLM, have broken away to start something new that they are soft launching after months of development.

Huxe AI

Every now and then, we experience a product that feels simultaneously alien as well as strikingly familiar. When ChatGPT launched, it was a magical moment because the chat interface that was familiar to tech savvy folks cloaked a powerful and strange intelligence.

Trying out Huxe AI also felt magical. If I forced myself to compare it to familiar experiences, it semed like some cross between podcasts, newsletters, and AI personalization more broadly. But really it feels like an original way of digesting and learning content.

After onboarding, which involves mainly connecting your calendar/mail and specifying 5 topics of interest, the default experience is the daily briefing. The two hosts recap your inbox and day ahead calling out possibly urgent or actionable items. They then transition to current events relevant to your topics of interest.

I must say all of this is really delightful to listen to. It is similar to the NotebookLM audio overviews, but it is much more personalized. You can even skip topics or dive deeper. Moreover, the integration with other apps like calendar and mail is very promising.

One of the biggest limits to widespread application of Generative AI technology is providing the right context at the right time – similar to the challenge students had uploading ppt vs pdfs.

However, I question any audio output first experience. It’s one thing to listen to a generated podcast every once in a while about a specific thing such as a research paper but quite different to work Huxe into my daily life if its so audio first.

Guess we’ll see how this app morphs into an experience that meets people where they are whether on their commute or in a noisy coffee shop. Download the app here but join the discord to get an access code here.

Anthropic Edu

Another notable announcement in the past month in Edtech came from Anthropic who have been investing more in empowering students. With Claude for Education, the company is rolling out a student ambassador program, API credits for student builders, as well as Learning Mode. Read more here.

Learning Mode encourages critical thinking by guiding with questions rather than giving answers, emphasizing core concepts, and using Socratic questioning, while also offering practical tools like templates for structured learning.

While it is awesome that Claude applies more pedagogical principles, it is not so awesome that it only does so in Projects which is only available to paying Pro consumers. Access to these innovative learning experiences remains a challenge across the ecosystem today, of course with exceptions.

Nevertheless, Claude also announced a 4-hour AI fluency course that is rooted in 4 D’s: Delegation, Description, Discernment, and Diligence. And folks have actually been raving about it.

While I haven’t taken the course, these are the displays of leadership we should see in this era as AI transforms, learning, building, and living in our society.

Subscribe now

That’s it for our recap of tools this week. I am planning a couple of series of posts around some themes I have been reading, listening, and thinking about.

For now, as we build these tools, let us consider how to better meet learners exactly where they are at whether its with flexible file uploads, providing alternatives to audio, or simply ensuring experiences for students are accessible to all.

Share

When people say technology will never revolutionize Education, I believe we must either disagree about what it means to revolutionize an industry or disagree about the impact that technology can have.

The last time I heard this was from a talk given by, the Youtuber behind Veritasium, Derek Muller. I should quickly acknowledge that Veritasium makes some of the best entertaining educational content on Youtube that I have seen, period. I am impressed by how much one Youtuber has democratized physics understanding.

But I am not impressed that Muller declared in his talk that technology will never revolutionize Education. I don’t just say that because I am building in this space. Many teachers and school administrators would agree with Muller. There have been many waves of technology that have promised to upend the system but none were successful at this goal.

But what if we don’t all have the same understanding of what it means to revolutionize an Industry? what if we haven’t appreciated the impact technology has already had on Education?

Subscribe now

What Does 'Revolutionize' Mean Anyway?

We criticize past technologies for not upending Education but we seem to be picky about how we set the bar. Since 2000, some industries have been said to have been revolutionized by the internet and mobile devices.

Retail was supposedly revolutionized by eCommerce, but i still have many friends who buy most of their clothing items in person. Media was supposedly revolutionized by streaming and on demand content, but I still went to see the movie Sinners last weekend with a friend who was going for the fourth time. Transportation was upended by the Uber and Lyft, but I still use public transportation nearly every day and i never choose ride share options for multi hour journeys.

All these sectors underwent significant change even if they were not wholly revolutionized. Maybe something similar happened with Education?

In the 80s, TVs were supposed to beam an expert into every classroom while baby sitters sit back and watch but that didn’t succeed at scale because of the interactive nature of learning. The 90s promised revolution of Education through interactivity on computers but that also didn’t upend the industry.

The most recent craze of the 2010’s was MOOCs – Massive Open Online Courses like Coursera, Edx, Udacity. The typically slow moving bureaucratic higher institutions spent immense resources on converting classes to online formats and even erecting departments just for churning out MOOCs. These did not succeed in democratizing access to quality education either.

While these technologies failed on their promise, they were more impactful than memory might suggest, though. Take MOOCs for example.

MOOCs do work really well, just only for the more self-managing learners who already have a high school or undergraduate level education. Not to mention that most courses in my undergraduate education had a lot of MOOC like aspects with pre recorded modules, corresponding activity sets, as well as self-pacing.

In reality, technology has transformed Education in ways that are quite subtle and profound. A great example is a technology older than all the ones I’ve mentioned — Optical Character Recognition.

Share

Optical Character Recognition (OCR)

OCR was not initially used in Education. After its early invention in 1914 by Emanuel Goldberg, it took a few decades before the technology got good enough to be used to decode mailing addresses in the Post office system.

Eventually, this technology transformed the classroom without ever entering it. Standardized testing and other large scale assessments had been growing in popularity specifically because of their benefit of assigning a reliable standardized score.

However, they had a bottleneck. It took so many labour hours to manually grade all those questions. So when OCR got good enough along with Scanners and Computers, it made sense to automate all that grading.

The somewhat unexpected result, was that these tests were significantly optimized for OCR which thrives on bubble style selections. This increased the number of Multiple Choice Questions (MCQs) and True/False Questions (TFQs) while reducing the number of essays, diagrams and free responses.

The awesome thing that happened next is teachers began giving out more assessments, since they could be graded faster. While students don’t love more tests, the pedagogy is unequivocal about the impact of retrieval practice on learning. Testing students more often was a sweeping horizontal benefit. A benefit so big, you could argue it revolutionized Education!

The not so awesome thing that followed was that teachers then began changing the way they taught to be more tailored to the kinds of exams students were going to face – “Teaching to the test.” This, in turn, swayed students to also prioritize skills that would benefit them on these assessments especially memorization and recognition. I mean, I love a good acronym, but teachers stopped focusing on skills like critical thinking, communication, collaboration, and creativity which are more difficult to test on MCQs or TFQs.

When I look back on the influence of OCRs its hard to ignore the magnitude of the impact. Teachers tested students a lot more. They changed how they taught. And students changed how they learned.

The AI age

Here are three lessons we can take from the impact of OCR technology on Education as AI transforms society.

First, is that a technology that significantly affects many sectors of society (OCR affected our mail, banking, retail, security and identity systems) is likely to also transform Education. OCR did not directly help students learn.

It sped up other processes (grading) which freed teachers to apply more pedagogical theories (increased testing.) AI will coexist with, and enhance, rather than entirely replace teachers. It will improve lesson planning, provide feedback, and aggregate data which are all secondary to the lecture experience.

Second, we know that AI can be the redemption for OCR. Before OCR, the main forms of assessments were oral. Student were individually and privately evaluated by a teacher or publicly evaluated collectively in the form of a presentation. This was terribly time consuming, hence the embrace of OCR and the MCQs and TFQs.

AI can reclaim some lost ground. That could mean more free response questions on SATs, WASSCEs, or IGCSEs. Or engaging critical thinking in the oral defense of possibly AI-generated essays. Or even new assessments with open-ended submission formats, focused on the feedback and learning process.

Third, we know that the negative impact OCR had on learning was subtle and profound. But had they been armed with information and opportunity, teachers would have prioritized teaching soft skills, digital literacy, and creativity.

This means we must be deliberate in this AI moment. In fact, we must not go “oh shucks, I guess AI will never revolutionize education” as Muller’s words might suggest. Instead, we, builders, teachers, and administrators, need to be ready for the immense impact AI will have. We must ensure our philosophies, decisions, and actions with AI prioritize students for the changing world they are graduating into.

Subscribe now

It’s open-source. It’s multimodal. It’s been caught in twitter drama.

Llama 4 is the latest family of AI models from Meta’s research lab and it's already shrouded in massive controversy. Before spilling the tea on Meta’s ranking hacking, let’s review the major changes they announced last month as well as the potential benefits to students.

llama 4 comes in three flavors, Behemoth, Maverick, and Scout, in order of decreasing capabilities. Only the second two are available as of writing, but all are natively multimodal and use the Mixture of Experts architecture for the first time!

For the curious nerds, I recommend reading the full announcement which details different engineering feats achieved by the team. For instance, the smallest scout model was trained on a 256K context window but the team developed an effective “needle in a haystack” solution to increase the context to 10M. The team was also able to discover favorable initial hyper parameters that is transferrable to various LLMs.

All in all, llama is now multimodal first but still open source which has major implications for students, teachers, and anyone supporting learners. You can try it out today in Meta products like Whatsapp and Instagram.

Subscribe now

Multimodality

To understand why multi modality is such a big win we have to review the Learning Science principle of Dual coding.

Dual coding is the process of combining and presenting information in two formats.

These formats could be voice and text, text and image, and so on. This strategy provides more than one avenue for the brain to recollect and reconstruct information. It is also different from ‘adapting to learning styles’ which has been thoroughly debunked in research for decades now.

Adapting to learning styles suggests, for instance, that while some students may benefit more from a primarily visual learning experience, others may benefit more from a primarily auditory one. Instead, Dual Coding, insists on providing both formats (simultaneously) to all students. The former is a myth, the latter is research-based fact.

And this fact is what makes multimodality exciting! Not just because Meta is doing so, but because all the major models are leaning towards multimodality too!

Google was the first company after the text-based ChatGPT viral moment to announce a natively multimodal experience in March 2023. GPT-4 introduced images along with text and GPT-4o extended to audio and video. Following suit is Anthropic’s Claude, Deepseek’s Janus-Pro 7B, and xAI’s Grok 1.5V which are all multimodal today.

Though these models still have factuality concerns, students can take pictures of homework, study notes, or real world objects and get personalized responses to accompany the visual aid. Teams like the one behind NotebookLM can create a podcast grounded in the text or images that you share. The future of education will need administrators and teachers alike, who will facilitate an environment for an AI tutor that embraces the context of the student through the sounds and sights around them.

Open source

When models are open-sourced, the benefits of the technology can immediately impact students today. Not only do we improve learning outcomes for students (when used right) but students can also begin adapting their learning journeys and habits to the available technology. Even builders can dream up and implement various ideas with the latest technology, accelerating us towards the successful EdTech products.

When Deepseek open-sourced R1 which competed with OpenAI’s state of the art model, the ChatGPT maker began exploring more ways to live up to the ‘Open’ in its name. It is well known fact in the open source community that open leads to more collaboration, innovation, and secure outcomes.

The fact that Meta, Mistral, Google (Gemma), and now OpenAI are all contributing to this space, removes a major barrier to distributing the fruits of genAI more equitably to students around the world. But there’s a catch.

While some of these companies are building small (on-device), capable, and free models, we must not forget that there are more significant barriers to true accessibility for students, from the technical reasons like reliable power to societal ones like unstable homes.

Share Ctrl + Alt + Learn

The Drama

In the last few years, the sentiment about the capability of an LLM has primarily been driven by results on one platform – LMSYS now Chatbot Arena. This is a benchmark determined by human voters on head to head comparisons of model responses. When I discovered it, I was excited about the reliability of the process.

This excitement has quickly evaporated since the llama 4 release for two reasons. The first is that Meta casually mentioned that an experimental chat version scored an ELO of 1417 on LMArena. This was a very surprising result for Maverick (maybe more believable for the bigger Behemoth.)

Digging deeper, it was discovered that Meta submitted a model that was very optimized for this benchmark. Needless to say, Benchmark hacking is a big no no. This is not the first time Meta has been accused of using benchmark datasets in their training data, which leads to the second reason.

After Chatbot Arena addressed the confusion, they shared the head to head comparisons which would make one question the goals of the human raters (and Meta’s research lab).

For instance, many of llamas responses were very verbose, and it was hard to find the information you needed in its output. This is not a great experience for most users, in my opinion. But it is a fact that this experience is worse for students as it wastes valuable study time searching for answers in the response and risks cognitive overload with all the information overwhelm.

We need models that are not optimized for gaming benchmarks but optimized for helping students. Not all of the frontier AI companies have lost the plot in that sense. Google and Claude have invested in fundamental pedagogical improvements to their models.

This means model that don’t just give you the answers. They engage in socratic dialogue. They leverage dual coding. They provide concise explanations. They use elaboration with concrete examples.

So as we follow the news of the latest models, lets ask ourselves how does this development impact students? is this a cool demo or does this also foster learning?

Subscribe now

Subscribe now

unfolding before us is that pivotal moment where technology, especially AI, isn't just changing tools or work, but potentially reshaping the very foundation of learning. Ctrl + Alt + Learn is your bi-weekly guide to navigating this exciting, complex future.

to thrive in this future, educators, administrators, parents, and learners alike, need to have informed, critical discussions about EdTech and how it should influence the evolution of learning.

every other week, we explore questions about the use of technology in learning, analyze the impacts of AI developments on education infrastructure, and learn from the experts and teachers doing the most difficult work of sculpting the future, one child at a time.

stay up-to-date

whether you're an educator seeking innovation, a tech professional passionate about impact, a policymaker shaping the future, a curious parent, or simply someone who believes in the power of learning, you'll find valuable insights in the questions, conversations, and demonstrations on this newsletter.

this isn't just a newsletter; it's a conversation. i encourage you to share your thoughts, experiences, and questions throughout.

the best way to follow this newsletter and never miss a post you would love to read is to subscribe for free!

Subscribe now

about me

hi there! i am Ayotomiwa, your neighborhood Yoruba boy and software engineer. i am convinced that education is the biggest flywheel for society. if we believe Obama’s (likely misattributed) quote that knowledge is the currency of the 21st century, then my mission is to make everyone filthy rich!

i grew up with some incredible teachers and professors. that alone is reason enough to see the impact of education, but the scholarships i received also showed me how transformative an opportunity at an excellent education can be. i want to build that for others.

growing up in Nigeria provided me with perspective that spurs me to advocate for students/contexts that are underrepresented in America. issues can range from low quality mobile cameras and high-latency connections to power outages and domestic responsibilities.

working as a software engineer provides me with the hard core, hands-on experience to root my optimism in reality. i am interested in how to build and deploy the latest developments that impact learners today or that concretely improve learning outcomes in the future.

i enjoy playing a ton of sports, recently getting into yoga and running. i also enjoy board games and love watching psychological thrillers.

Subscribe now