Finally. Privacy Focused AI Use is Here! | Rob Braxman Tech
Summary
Transcript
This will begin a new series on AI. We will focus not on the fear of AI but harnessing it offensively so we can use it on our terms with privacy and security. So there will be a lot of teaching here and this will eventually be lots of hands-on tech on AI use. But we will learn AI differently and safely. We will also use AI to prove privacy and security concepts so we can learn to defend ourselves. There are actually two threats we need to be aware of with AI and these threats can be mitigated.
If we use Windows, macOS, iOS or Android we have the risk of having hidden AI work on our devices. This is the big to do with Windows Recall for example that you’ve all heard so much about. AI silently working in the background and dangerously sending our data to be processed in some big tech server where this data can be surveilled and captured. Fortunately we can fix this by using Linux. The second threat with AI is that when we communicate with a cloud AI controlled by someone else we are sending our private data to an external party and our data can be used to steal or surveil our ideas or even have our data used for future machine learning.
But something that many of you will not know about is that you can now run AI locally on your computer without internet. So it is actually possible to run AI safely under your control. I will demonstrate this later in this video using Linux and a local AI. Many of you are now alerted to the presence of new hardware on new computers which are perceived to be dangerous and these are the NPUs that are now in every computer. But I will teach you today that the NPU is something we can use. As a start of a series this will be an introduction.
I have to explain some new concepts so that the following videos will make sense. And at the end I will do a very basic demos so you can get started with the learning. Understand our end goal though. I will teach you in the next videos how to use this AI in our way. Some of this need not require high-level tech skills but I’ll give you a mix. If you’re interested stay right there. First let’s start with a conceptual explanation of AI. An AI is not something built from an accumulation of logic rules.
Tesla actually started with that. It based some portion of its self-driving on a rule-based model with black and white interpretations. If you see this, do this, if you see that, do this instead and so on. Then Tesla went full AI and dropped the rules programming. The result was a suddenly more effective self-driving software that can actually adapt to situations that it has never seen before. AI is a new paradigm. Instead of programming an infinite set of rules to a computer an AI learns by itself using a technique called machine learning.
When an AI uses its learning to come up with a novel idea it is called generative and this is the first indication of real intelligence. Computers by nature don’t understand words. It sees binary numbers and can do extremely fast math. But this capability can be used for AI because computers using math can discover patterns and do it on its own. In the case of computer vision the colors in an image are captured as unique numerical values and in language understanding each word is assigned some numerical value based on a dictionary and its position in a sentence.
In the image example the AI computer is given millions of images of specific objects with a tag of what the object is and the computer will automatically develop a recognition of patterns that can be used to identify objects. The interesting thing about how the AI understands is that the computer actually builds a neural array or matrix of patterns and assigns weights to each input and there are layers and layers of these weights that are used to identify patterns in what the computer sees in the input. Exactly what those patterns are may actually be unknown to us.
Just to give you an idea a small AI model could have three billion parameters with each of those parameters representing some pattern detected. GPT-40 the well-known AI is thought to have 220 billion parameters and those parameters are what makes up the matrix that is known as the neural network. Again it is really nothing but a matrix of weights applied to learn patterns and this is what makes up an AI model. The weights are not really human readable. We wouldn’t necessarily know how the AI looks at a problem and how it passes it through these billions of parameters to determine which weights then generates its final answer.
We can only teach the AI to develop its own patterns through machine learning and then query the AI to see if it learned it. In most cases the pre-trained AI is like OpenAI’s chat GPT, Metaslama 3, Google’s Gemini and Trophic’s Claude or XAI’s Grok are given raw data from the internet or from books and documents. A lot of terabytes worth of content but no one is around to validate what the AI saw. The information was not vetted so not everything that is fed to the AI is necessarily correct. The only way to actually find out what the AI learned is to tweak it later with a technique called back propagation where humans have to see if the AI learned incorrectly.
They will query the AI and then train it to modify its responses. This is also the part where the AI is censored by its developers but outside of that risk of AI learning something that the humans are not aware of the AI can also be quite useful as it is. Large language models or LLMs which is the groundbreaking version of AI shown initially by chat GPT are capable of deep conversation with humans using intense knowledge especially of technical facts and computer programming. It can even invent new content like poems, images, essays or programming code which is a demonstration of the ability called generative AI.
It is not just regurgitating stuff at learn but is able to create new concepts based on something I learned before. Now let me do some other new terms to you. An AI has two main functions when you teach the AI it is doing its first function which is the learning function or machine learning ML as we’ve discussed. Machine learning is a very expensive process and requires lots of expensive computers with thousands of GPUs so only a few big companies can do this. The final model resulting from this is called a pre-trained model.
The second function of an AI is to use it. Query it. This action of querying the AI is called the inference function and this is where we as end users will utilize it. Thus if you want to build a pre-trained AI like GPT-4 you will spend billions on computers. But to use the AI with inference you just need a standard computer with lots of memory, a GPU graphics processing unit and maybe an NPU neural processing unit to run it efficiently and on this you can run an existing pre-trained model. The models that talk to you are called LLMs or large language models.
An example is chatGPT but chatGPT is a proprietary model. You will learn later that some models have been open-sourced so that we can use it ourselves for inference. There are also chat models called SLMs or small language models which can only be a gigabyte in size. They’re able to have conversations but possess less world knowledge. Apple will for example load an SLM on a phone and I would guess that Microsoft co-pilot will do it as well on a computer. There are other AI models like image recognition, voice recognition or OCR models which are even smaller.
The actual AI is done by passing a query through a software called a transformer and then the transformer encodes your input into something the model can understand and then decodes the output for you. This transformer is the key software that enables the use of LLMs. You’ve heard for example of the term GPT used in AI which actually means generative pre-trained transformer. So now you’ve heard all these terms used. Now let me explain how an AI can be really useful to you even as a privacy-conscious individual. You can actually learn how to control an AI.
The new expected power of AI is when a business leverages an existing pre-trained AI by adding their smaller knowledge base to the AI. You will learn terms like RAG, Retrieval Augmented Generation which is how a content store like a company’s private database or even the internet can be used with the LLMs general pre-training and this can then result in a very focused AI for specialized purposes. AI can also be modified even if pre-trained to respond differently including adding immediate functionality to hardware equipped with AI functions. For example a future Tesla robot could be taught to cook a protect away for you based on your particular tastes.
The concept of adding new small data modifications for learning to a pre-trained AI is called fine-tuning and this will be very common and does not use a lot of resources. So the AI model you get is not a dead end of usefulness based on what is given by the developers of the model. You can modify it especially if it is open source and then you can hook the AI to software that actually performs a function on your computer based on AI results and this functional piece is called an AI agent.
Again understand these new terms rag, fine-tuning and agents and you will learn later on that this is not difficult to do. Our particular interest in the use of AI is to use it for inference meaning to query it and have its responses be useful. So our source of models will be open source models and you will find out what these are. First let’s talk about what hardware is needed for the AI to run. Smaller AI models do not need an NPU or GPU to run. Now what exactly are these GPU and NPU models on these computers for? NPUs are new and weren’t around much before 2024.
These chips are called accelerators. They are designed for specific tasks that can free up the CPU. They can do those specific tasks at a much faster rate with low power use. For example an NPU is nothing more than a chip that allows for array multiplication. As I said earlier AI when used for inference is nothing more than doing matrix math on large arrays. A GPU can also do math on large arrays and that’s why they’re used in the learning phase but usually on a client computer the GPU is being used to display graphics so it’s often busy doing that.
However if it is not doing that you can actually have the GPU do these matrix math calculations. An NPU allows the operating system to offload the matrix math to it. It also is able to multitask by doing multiple matrix maths simultaneously. It is a lot more efficient than the CPU especially when doing math on very large arrays that are often found in AI models. So that’s what an NPU is for. Microsoft decided that it will put co-pilot features only on co-pilot plus PCs which it has designated as devices with a CPU GPU and NPU.
Apple decided to load AI on Apple silicon max and iPhones with a neural engine or bionic chip. Google decided to load AI capabilities on phones with a tensor chip. Tensor by the way is another word for matrix array so that’s where that name came from. The reason for these companies requiring certain hardware is to make sure that the user experience is acceptable on their more intense large models. However small AI models do not need anything other than a CPU because there’s not as much math to do. Complex models however would consume too much power and that would be important on phones.
A computer can run multiple models. Small models that can reside in memory like image processing could be always on and always be available. Otherwise it’s just about moving the active model in and out of memory if they’re very large. Large models would render a device without a GPU, NPU and lots of memory unusable. One thing we need to be clear on. We cannot assume that there’s only one AI model running. It is likely that the OS would load multiple small AI’s for common tasks plus multiple models could be run sequentially using something called model chaining.
Also you cannot assume that your computer even if older cannot run some smaller AI especially if it’s a PC it is actually quite likely. So don’t assume that the lack of an NPU or GPU makes you immune from an AI spyware. It may make you immune from a large AI running locally but it could still run small AI especially when specialized as a spyware feature. Now here’s an important point. You could pass input to the cloud and a cloud computer can do that math and return the AI result to you.
Just like you can go to your browser and talk to chat GPT. This is no different for your OS. The OS can openly or secretly talk to an AI in the background in the cloud. This is why we all need to have a plan move to switch our activities to a Linux OS at some point to eliminate all these risks I just described. I will explain this journey in another video. For now let’s not force any new tasks. This is just explain a direction. In other words you can stick to your windows and Mac OS but just start planning on having different future options.
So to summarize you could run an AI on a normal computer without some extreme sort of hardware but to do larger models that really have broad function like an LLM you should have a more powerful computer. The kind of computer I’m talking about is the typical ones used for video editing in gaming. A fast computer with an Nvidia card and newer models that have an NPU added. Now here’s another factoid. Accessing an AI from a browser like using chat GPT or XAI and chatting with it is fairly safe as far as using these for inference.
So in the absence of a new computer you can use cloud AI for learning. It will be safe at least from remote controlling your computer particularly if you’re running on Linux but be mindful of the privacy danger of sending your thoughts in documents to outside servers. So always understand that whatever you say to a cloud AI is always visible to somebody else. Now let me do a quick demonstration to show you how to do safe AI work. In later videos I will show more advanced applications of this and I will actually use the AI to help me demonstrate privacy concepts to all of you.
Hopefully this will up the game on explaining privacy issues and I can use the AI to show things and code things so we can do more than talk. So we will be the first to use AI to build our knowledge of privacy and security in a YouTube channel. I’ve already discussed the kinds of computers that will allow this demo so just so you know my setup my OS for this is Ubuntu and my computer is a laptop a Dell XPS 15 running an Intel i7 12 700 H 2.3 gigahertz 14 cores with an NVIDIA GeForce RTX 3050.
It has 64 gigabytes of memory which is actually quite excessive. It worked in 16 gigabytes of memory initially. 32 gigabytes would be good. To run the local AI we need to go to ollama.ai site which I will show you here. Ollama by the way is open source and you can see the project code on GitHub. Ollama is just the enabler. The actual AI comes from the suppliers of AI open source pre-trained models and I will show you a list here. I personally tried many of these and they have different advantages depending on what you’re trying to do.
The most popular model is llama 3 but don’t be put off by the fact that this model was developed by meta meaning zach. I’ll consider this the first time I’ve seen zach do something really positive for humanity by open sourcing this model. This is a censored model by the way but in later videos you can learn to get around that. Now to install this is super simple. Choose your OS and then just download installer on your computer. On Windows for example there’s a file called ollama setup.exe that you install. On Linux you just provide a script they provide and when you run it you will actually have nothing to set up.
This is both a plus and a minus but it gets you running immediately. Pretty much immediately after installation which is quick you can then use the AI. So in this example I will use llama 3 and for this demo I will simply use the terminal command line and you type this ollama run llama 3. If you’re doing this for the first time it will download llama 3 which is a 4.7 gigabyte file so it will take a while. You may want to turn off the VPN when doing this so it’s not too slow.
In my case it’s already downloaded and I actually have a dozen models loaded on my computer. Let me make something clear here as we use this. The AI is completely on your computer when using it this way. You can disconnect the internet and it will still work. So I can easily run it now and all I have to do is on this prompt here I will demonstrate a couple of examples. First I will talk to Elon Musk. Let me see if I got it set up right. By the way I found that AI runs faster on Linux than on Windows.
Okay now let me demonstrate a more sophisticated example which we will delve on deeper in a future video. I’m going to let the AI model write code for me in Python programming language which will do screenshots of the screen every few seconds and save it to my folder like Windows recall. This is just a teaser we will go through a lot of videos where I will now use the AI itself to explain concepts. But combined with a few more AI developed code I can demonstrate how Windows recall could be used.
In the description of this video I will give you a sample AI exchange which is in a PDF file where I asked basic privacy questions and I will show you how using something called prompt engineering I actually have the AI demonstrate that the dangers of privacy are more serious than it originally stated. There will be fun times in this channel coming up so please subscribe now if you want to see this kind of content. Folks as I discussed in this video big tech is inserting various AI models in your devices and it’s getting more scary every day.
So fortunately the approaches I’ve taught in the past are even more valid. These are the uses of the Google phones running AOSP, hiding our identities and email and phone numbers and running a VPN. All these products are needed and all these are in products available on my store on Braxme. Please join us on the Braxme platform you will find the various products in the store on that site and you can also join the over 100,000 people in the community who talk about privacy issues daily. Hope to see you there.
Thank you for watching and see you again next time. [tr:trw].
See more of Rob Braxman Tech on their Public Channel and the MPN Rob Braxman Tech channel.
