SoCal, Software 3.0, and Chronic Health Problems
Happy new year from San Francisco!
2022 was as a year of deep reflection for me. Chronic health problems, relationships entering and leaving my life, and in general settling into this next phase of adulthood led me to revisit my modus operandi. What is not longer working for me? What do I want to start doing?
While my public writing slowed down nearly to a halt these last five months, my private writing skyrocketed - tallying up all of my journal entries and reflections from the same period comes out to around 60,000 words, or 400 words a day.
This past month in particular with all of the holidays gave me the room to crystallize many of the insights and thoughts that I’d been having. A big reflection was the joy and energy I created from writing online, so as the new year begins, I wanted to share what I’ve been experiencing and thinking about more frequently.
Preview for this week:
Traveling through Southern California
Are we progressing beyond Software 2.0?
On chronic health problems
Traveling through Southern California
San Diego was my home from 0 to 6 as a wee lad and from 18 to 22 as a wee adult. Many nostalgic memories take place here - from frolicking on sandy beaches to deep conversations in between bites of California burritos at 3am. Safe to say it has a special place in my heart.
In fact, there’s something about the city that pulls at me. Every single year since I’ve graduated from UCSD I’ve found myself back at Gliderport, a 370-foot sea cliff overlooking the ocean just outside the campus. The expansive views, mesmerizing sunsets, the constantly changing cliff surface… it reminds me that there are other rhythms to life than relentless self improvement. A more laid back, relaxed, go-with-the-flow kind of rhythm. A rhythm I could bear in mind more often.
Traveling through SoCal (we also hit LA and Catalina Island) this past week with family was beautiful. We watched silly Bollywood movies, got deep into Smash Bros tournaments, and cooked food together in beautiful, balmy 60-70 degree weather. It had been a long time since I’d spent that much time with humans without a break, so it took some time to get used to my “social muscle” being always turned on. But I adjusted quickly, and now, after the trip, I feel truly relaxed for the first time in a long time.
See you back in December 2023 I guess, San Diego!
The dawn of Software… 3.0?
With all the recent hype in AI, I found it prudent to revisit Andrej Karpathy’s classic Software 2.0 article from 2017, because I think it points a path forward to where we may be going next.
What’s Software 1.0 & 2.0?
To create Software 1.0, programmers write explicit instructions (Python, C++, etc) for a certain goal such as visual recognition for cats. Those explicit instructions are compiled to raw machine code that the computer runs.
In Software 2.0, machine learning engineers define a space of possible computer programs by choosing a neural network architecture. Then, the computer “compiles” a task-specific dataset and evaluation metric to neural network weights - aka the best possible set of for that task within the defined program space.
It’s so cool because for many problems such as visual recognition, speech generation, and even optimal database indexing, computers can find better explicit rules in neural network weights than humans can in Python or C++. In other words, Software 2.0 represents a leveling up in abstraction where computers (not humans) are writing the explicit instructions for a given task. Humans just have to supply good examples.
But Software 2.0 as presented in the article is still far more brittle than human intelligence. While the same neural network architecture can accomplish many different tasks (ie recognizing cats and recognizing dogs), each task requires huge amounts of data (1000+ images of cats, 1000+ images of dogs), whereas a human would only need an example or two per category.
What I’m excited about are two big trends in AI that point to another leveling up in abstraction (Software 3.0?) that bring computers much closer to human-level intelligence: Synthetic data and generative AI.
Synthetic Data
The endgame of synthetic data is a system with two components:
Computer A running Software 2.0, finding an optimal program for a task using an evaluation criteria specified by a human, and data generated by Computer B.
Computer B running Software 1.0 or Software 2.0, generating data using parameters specified by a human
In other words, this new system “compiles” the data generation parameters for Computer B and the task specific evaluation metric for Computer A into neural network weights for a particular task. Eventually, Computer A may learn to create Computer B’s data generation parameters, reducing human involvement to simply supplying the evaluation metric for a particular task. That’s exciting.
In other words, synthetic data is a leveling up in abstraction from Software 2.0 where we’re no longer supplying datasets to computers. We just supply the evaluation metric. That’s pretty close to human intelligence!
Generative AI
Here’s a few things I used ChatGPT for in the last week:
Refreshing my memory on how Python code is run by a computer:
Learning why piers often have grated floors instead of solid concrete floors (Google could not find an answer for this):
Looking up the damage calculation formula for the Pokemon video games (instead of searching for it in a Wiki article):
What’s incredible is that it’s one ~100GB model doing it all. Future versions will be able to be spit out low level intelligence completely locally on phones without the internet. That’s mind-breaking. Incredible.
It took me a while to realize that in this new generative AI paradigm, there was no task-specific training like there was in Software 2.0. The task selection (prompting ChatGPT ) is independent from the dataset or training process. There’s one round of Software 2.0-esque training, but afterwards, humans can “program” the model for a particular task with a prompt.
With generative AI. Humans are now operating at a level of abstraction above Software 2.0. Nigh indistinguishable from human level intelligence.
For now, we’re limited to artistic images and text, but I wonder if we’ll start seeing more and more tasks being eaten by these higher levels of abstraction. We’re already starting to see this happen with robot tasks, character animation, and more.
On Solving Chronic Health Conditions
I’ve now dealt with three major chronic health conditions for 2-5 years each (chronic tendonitis in my hands, a mystery hamstring pain, and sleep apnea) but I’m on the up and up for pretty much all of them. I reflected on my journey, and these are my top takeaways and tips:
It’s easy to let small pains add up in a chronic health condition because individually, they don’t much change the quality of your life. But you run into the danger of letting them pile up into something that’s psychologically too much to tackle.
It’s hard to generate empathy for long-term problems, both from yourself and from other people. No one is coming to save you. You need to start addressing it yesterday.
The medical system isn’t set up to treat chronic conditions effectively because of financial incentives. Be in the driver’s seat, be your own researcher, and don’t be afraid to try unusual treatments (as long as they’re safe). Don’t delegate your well-being to a single practitioner.
Manage symptoms but treat underlying causes. Treating symptoms is like putting bandaids on a festering wound. It’s dangerous because it gives you a false sense of victory, when that energy could instead have been used to treat the underlying causes
Where something hurts isn’t necessarily where it should be fixed. For years I tried massaging, stretching, strengthening my hamstring. All I had to do was stretch my hip flexor, which had zero pain.
Sometimes, it’s better to focus on overall health. Causal chains can sometimes be too complex to debug. In the prior example, I didn’t actually figure out that my tight hip flexor was causing my hamstring pain until after the fact. I was just trying to become more flexible in general when I accidentally stumbled upon that conclusion.
Believe in total victory to start. Some chronic health conditions can’t be cured and perhaps symptom management is all you’ll get. But why not go for the gold? If you don’t believe it’s possible because you feel defeated, you won’t take actions that may cure it once and for all.