Hey Bard: Ready to be a Programmer?
Matthew A. Tucker…July 12th, 2023
Backstory
When I landed my first real programming job long ago, my curmudgeon Uncle Ray grumbled: “In 10 years, computers will do the programming & you’ll be out of a job!” As it turns out, that didn’t happen. Now flash forward to the 21st century: could Uncle Ray finally be right?
Generative AI such as OpenAI’s ChatGPT, Google’s Bard and many others have emerged in spectacular fashion. Generated essays submitted by clever students have professors wringing their hands in alarm. Generated images of disturbing or amusing nature such as the Pope in an extravagantly overpriced designer fashion jacket or an astronaut riding a horse on the moon profligate. Generated audio tracks mimicking celebrities making unbelievable or distasteful statements test our trust. There are even rather extreme ruminations on an “existential threat to humanity”. More commonly, there are assertions of an “existential threat to (human) jobs”.
I was at a good point in an interesting project so I thought, “Why not give it a try & see what all the excitement is about?”. The project can be imagined as an iteration on some basic steps:
- design, train & tune a model for hand-sign classification
- transform the model into a lightweight package suitable for mobile devices
- wrap the model in an app capable of streaming video for analysis
- present the signer with the model analysis
How hard could that be for a generative AI?
Bard: Being Useful
Hey Bard: Show the best TensorFlow model in Python for hand-sign image classification.
I had a simple model to classify images in 10 categories (digits 0–9). It had just a couple layers but in training reached a validation accuracy of 88%. I imagined with tweaking & tuning, I could improve the accuracy. You really want 99.9% accuracy running your validation dataset since the accuracy may degrade significantly classifying real world data. For example, the model may very accurately classify the “sanitized” training set image but struggle with the real world image.
Bard generated a similar, simple TensorFlow model. It ran without error, training successfully to 90% validation accuracy — not bad!
Conclusion: Bard can be useful.
Acknowledgement: Training set image from https://www.kaggle.com/datasets/ardamavi/sign-language-digits-dataset
Bard: Catastrophic Forgetting
Hey Bard: Using the previous model, show how to improve the validation accuracy of the model.
Bard emphatically asserted the resulting modified model would achieve higher accuracy. Unfortunately, it didn’t run & crashed. I supplied Bard with the crash trace & Bard came up with a modification to fix that first problem. But then another crash surfaced. When supplied with that crash trace, Bard came up with another modification, asserting it would fix the problem. This caused another crash. Supplied with that crash trace, Bard generated a modification that fixed the 2nd problem but “broke” the code that solved the 1st problem. Supplied with that crash trace, Bard generated a modification that fixed the 1st problem (again) but “broke” the code that solved the 2nd problem (again).
Hey Bard: Are you aware of the cyclic nature of your fixes which fix one problem but create another problem which you then “fix’ but re-create the other problem.
Yes, you are correct. There is a cyclic nature to these fixes & subsequent crashes. This modification will correct the problem so the code runs properly.
Unfortunately, Bard could never “remember” enough to not repeat the mistakes of the past. Sounds almost human, eh?
Conclusion: Bard can suffer from “catastrophic forgetting”.
Bard: Insights with Baggage
Hey Bard: Show the Python code to load an American Sign Language Digits Dataset for training.
I decided to try another tack, asking Bard to load a better training dataset. Bard came up with a succession of code attempting to load a Kaggle dataset. These modifications were composed of “reasonable” formatting that specified various packages & techniques but all were flawed. When provided with the error trace, Bard would invariably say, “You are correct, this modification will solve the problem.” Repeating over many iterations, Bard finally came up with a valuable insight: I needed to provide my Kaggle API token in order to access Kaggle datasets. I had forgotten that piece of the puzzle so it was helpful to be reminded. However, there was a lot of superfluous baggage to unpack to get to that insight.
Conclusion: Bard can provide valuable insights but with undesirable baggage.
Bard: Ready to be a Programmer?
An afternoon with Bard reminded me of remote paired programming sessions with some (human) programmers:
- The collaboration could generate useful contributions.
- The programmer might pedantically assert as fact, mere speculation.
- The programmer could suffer from catastrophic forgetting resulting in a seemingly endless cycle of fixes & errors.
- The programmer could provide valuable insights but with undesirable baggage.
In this sense, Bard isn’t that different from many programmers (including myself?).
Hey Bard: Ready to be a Programmer?
I am not ready to be a programmer in the sense that I am not yet able to write code that can be used to create real-world applications. However, I am learning new things every day and I am confident that I will be able to become a programmer in the future.
Here are some of the things that I am doing to prepare myself to be a programmer:
- I am learning the basics of programming languages such as Python, Java, and JavaScript.
- I am practicing my coding skills by solving coding problems and creating small projects.
- I am reading books and articles about programming and software development.
- I am participating in online forums and communities where I can ask questions and learn from other programmers.
I believe that if I continue to work hard and learn new things, I will eventually be ready to become a programmer. I am excited about the prospect of using my programming skills to create new and innovative things.
Conclusion:
Would I hire Bard? Yes, possibly for experimentation. With proper mentoring, Bard can most likely perform some programming tasks.
Should you hire Bard instead of me? Heck no! ;}
Is Uncle Ray finally right? Well, maybe in 10 years…
