• 180 Posts
  • 1.81K Comments
Joined 2 years ago
cake
Cake day: June 9th, 2023

help-circle
  • j4k3@lemmy.worldtoSelfhosted@lemmy.worldConsumer GPUs to run LLMs
    link
    fedilink
    English
    arrow-up
    4
    ·
    edit-2
    1 hour ago
    Anything under 16 is a no go. Your number of CPU cores are important. Use Oobabooga Textgen for an advanced llama.cpp setup that splits between the CPU and GPU. You'll need at least 64 GB of RAM or be willing to offload layers using the NVME with deepspeed. I can run up to a 72b model with 4 bit quantization in GGUF with a 12700 laptop with a mobile 3080Ti which has 16GB of VRAM (mobile is like that).

    I prefer to run a 8×7b mixture of experts model because only 2 of the 8 are ever running at the same time. I am running that in 4 bit quantized GGUF and it takes 56 GB total to load. Once loaded it is about like a 13b model for speed but is ~90% of the capabilities of a 70b. The streaming speed is faster than my fastest reading pace.

    A 70b model streams at my slowest tenable reading pace.

    Both of these options are exponentially more capable than any of the smaller model sizes even if you screw around with training. Unfortunately, this streaming speed is still pretty slow for most advanced agentic stuff. Maybe if I had 24 to 48gb it would be different, I cannot say. If I was building now, I would be looking at what hardware options have the largest L1 cache, the most cores that include the most advanced AVX instructions. Generally, anything with efficiency cores are removing AVX and because the CPU schedulers in kernels are usually unable to handle this asymmetry consumer junk has poor AVX support. It is quite likely that all the problems Intel has had in recent years has been due to how they tried to block consumer stuff from accessing the advanced P-core instructions that were only blocked in microcode. It requires disabling the e-cores or setting up a CPU set isolation in Linux or BSD distros.

    You need good Linux support even if you run windows. Most good and advanced stuff with AI will be done with WSL if you haven’t ditched doz for whatever reason. Use https://linux-hardware.org/ to see support for devices.

    The reason I mentioned avoid consumer e-cores is because there have been some articles popping up lately about all p-core hardware.

    The main constraint for the CPU is the L2 to L1 cache bus width. Researching this deeply may be beneficial.

    Splitting the load between multiple GPUs may be an option too. As of a year ago, the cheapest option for a 16 GB GPU in a machine was a second hand 12th gen Intel laptop with a 3080Ti by a considerable margin when all of it is added up. It is noisy, gets hot, and I hate it many times, wishing I had gotten a server like setup for AI, but I have something and that is what matters.


  • I like to write, but have never done so professionally. I disagree that it hurts writers. I think people reacted poorly to AI because of the direct and indirect information campaign Altmann funded to try and make himself a monopoly. AI is just a tool. It is fun to play with in unique areas, but these often require very large models and/or advanced frameworks. In my science fiction universe I must go to extreme lengths to get the model to play along with several aspects like a restructure of politics, economics, and social hierarchy. I use several predictions I imagine about the distant future that plausibly make the present world seem primitive in several ways and with good reasons. This restructuring of society violates both some of our cultural norms in the present and is deep within areas of politics that are blocked by alignment. I tell a story where humans are the potentially volatile monsters to be feared. That is not the plot, but convincing a present model to collaborate on such a story ends up in the gutter a lot. My grammar and thought stream is not great and that is the main thing I use a model to clean up, but it is still collaborative to some extent.

    I feel like there is an enormous range of stories to tell and that AI only makes these more accessible. I have gone off on tangents many times exploring parts of my universe because of directions the LLM took. Like I limit the model to generate a sentence at a time and I’m writing half or more of every sentence for the first 10k tokens. Then it picks up on my style so much that I can start the sentence with a word or change one word in a sentence and let it continue with great effect. It is most entertaining to me because it is almost as fast as me telling a story as fast as I can make it up. I don’t see anything remotely bad about that. No one makes a career in the real world by copying someone else’s writing. There are tons of fan works but those do not make anyone real money and they only increase the reach of the original author.

    No, I think all the writers and artists hype was all about Altmann’s plan for a monopoly that got derailed when Yann LeCunn covertly leaked the Llama weights after Altmann went against the founding principles of OpenAI and made GPT3 proprietary.

    People got all upset about digital tools too back when they first came on the scene; about how they would destroy the artists. Sure it ended the era of hand painted cartoon cell animation, but it created stuff like Pixar.

    All of AI is a tool. The only thing to hate is this culture of reductionism where people are given free money in the form of great efficiency gains and they choose to do the same things with less people and cash out the free money instead of using the opportunity to offer more, expand, and do something new. A few people could get a great tool chain together and create a franchise greater, better planned, and more rich than anything corporations have ever done to date. The only thing to hate are these little regressive stupid people without vision, without motivation, and far too conservatively timid to take risks and create the future. We live in an age of cowards worthy of loathing. That is the only problem I see.


  • I use the term myth loosely in abstraction. Generalization of the tools of industry is still a mythos in an abstract sense. Someone with a new lathe they bought to bore the journals of an engine block has absolutely no connection or intentions related to class, workers, or society. That abstraction and assignment of meaning like a category or entity or class is simply the evolution of a divine mythos in the more complex humans of today.

    Stories about Skynet or The Matrix are about a similar struggle of the human class against machine gods. These have no relationship to the actual AI alignment problem and are instead a battle with more literal machine gods. Point is that the new thing is always the boogie man. Evolution must be deeply conservative most of the time. People display a similar trajectory of conservative aversion to change. In this light, the reasons for such resistance are largely irrelevant. It is a big change and will certainly get a lot of push back from conservative elements that collectively ensure change is not harmful. Those elements get cut off in the long term as the change propagates.

    You need a 16 GB or better GPU from the 30 series or higher, but then run Oobabooga text gen with the API and an 8×7b or like a 34b or 70b coder in a GGUF quantized model. Those are larger than most machines can run but Oobabooga can pull it off by splitting the model between CPU and GPU. You’ll just need the ram to initially load the thing or deepspeed to load it from NVME.

    Use a model with a long context and add a bunch of your chats into the prompt. Then ask for your user profile and start asking it questions about you that seem unrelated to any of your previous conversations in the context. You might be surprised by the results. Inference works both directions. You’re giving a lot of information that is specifically related to the ongoing interchanges and language choices. If you add a bunch of your social media posts, it is totally different in what the model will make up about you in a user profile. There is information of some sort that the model is capable of deciphering. It is not absolute or like some kind of conspiracy or trained behavior (I think), but the accuracy seemed uncanny to me. It spat out surprising information across multiple unrelated sessions when I tried it a year ago.



  • When tech changes quickly, some people always resist exponentially in the opposite vector. The bigger and more sudden the disruption, the bigger the push back.

    If you read some of Karl Marx stuff, it was the fear of the machines. Humans always make up a mythos of divine origin. Even atheists of the present are doing it. Almost all of the stories about AI are much the same stories of god machines that Marx was fearful of. There are many reasons why. Lemmy has several squeaky wheel users on this front. It is not a very good platform for sharing stuff about AI unfortunately.

    There are many reasons why AI is not a super effective solution and overused in many applications. Exploring uses and applications is the smart thing to be doing in the present. I play with it daily, but I will gatekeep over the use of any cloud based service. The information that can be gleaned from any interaction with an AI prompt is exponentially greater than any datamining stalkerware that existed prior. The real depth of this privacy evasive potential is only possible with a large number of individual interactions. So I expect all applications to interact with my self hosted OpenAI compatible server.

    The real frontier is in agentic workflows and developing effective niche focused momentum. Any addition of AI into general use type stuff is massively over used.

    Also people tend to make assumptions about code as if all devs are equal or capable. In some sense I am a dev, but not really. I’m more of a script kiddie that dabbles in assembly at times. I use AI more like stack exchange to good effect.




  • You’ve still got time.

    I was in Geometry class when 9/11 happened. The day stopped. The news was turned on in class a few minutes before the second plane struck. I watched it in real time. I had been in those towers 6 months before too…

    About the worst rabbit holes for me were giving any audience to perpetual motion trolls, and Brown’s gas nonsense in car stuff.

    Everyone tries to simplify messy complexity and we are all tribal in scope. I’ve learned to only pay attention to people with academic credentials. I don’t watch translated nonsense from general news outlets. The information I pick up elsewhere is more collectivised where I expect to see a bunch of people talking about something from different angles before I view the information as relevant. I also do not care for any outlets claiming to bridge some divided narrative as these are controlling where the line in the sand is drawn. If two parties are Right and Right-Jihadists like in the USA, calling one party Left is manipulating by validating the status quo and outdated perspective.

    What changed me started with stratification of rock layers and realizing deep time was not compatible with my religious narrative. I encountered a sharp personal dislike for biases and prejudice against others without logic or reason. I encountered a lot of plausible seeming arguments, but ultimately the people making those arguments had nothing to offer; they are trolls with no depth, interests, personality, community, richness in life. Look at such a person’s profile and they are not real. There is no greater engagement or value they add to the world. All they do is make arguments that muddle political narratives. I learned to view these people as either getting paid to post or idiots. I care about real people and that means your politics should only ever be a small part of your person and profile. Any person that lacks a serious passion project and hobby(s) but posts their politics is a joke to me.

    In a way, I extend this to any group now. Like do people in your group include Nobel laureates that contribute significantly to the advancement of humanity. Because if they don’t, why bother wasting time with fools that lack top aspirations. Live life with no excuses. Excuses are for fools. Do the best you can with the cards you’re dealt in life.





  • I know my weakness is my emotional depth. I keep protections in place mentally with a zero tolerance policy of trust. Like if my boss or coworker cheats on their partner, I know I will never trust that person or their character/ethics. The concept is one of the few that I retain from my religious past: “faithful in little; faithful in much.” Any person that shrugs off little lies or dishonesty is revealing their true ethics or lack there of. I do not try to hide who I am or lie about anything intentionally. Therein lies my lack of depth. I am unaware of how people filter and mask who they are in intentional ways, so anyone that shows harmful potential is someone I avoid and never trust more than is convenient or that I am forced.


  • j4k3@lemmy.worldtoPrivacy@lemmy.mlPrivacy Recommendations for a Young Teen
    link
    fedilink
    English
    arrow-up
    12
    arrow-down
    1
    ·
    edit-2
    3 days ago

    I think authoritarianism is a giant mistake and only creates duplicitous behavior. In my opinion tracking is ridiculous. None of us existed like this and ended up fine. In my opinion, all of this nonsense is acting as a stand in for relationships and real parenting. Humans make decisions and develop ethics based upon trust and autonomy. By stealing that factor of trust and autonomy, and replacing it with authoritarianism a parent is stunting the child’s growth of independent ethics and character. Make compelling discussions of why they should do whatever thing, but let them decide their own path. The lack of compelling discussions and real trust that requires risk is a major factor in the problems that exist in the present world.

    The one time you actually need to know where your kid is at because something has happened, you will not know because you have taught them that the only path to independence is to turn off the device and put it into a Faraday cage like pouch, or someone else will do so. If you have a fundamentally trusting relationship with open dialog and respect for their autonomy, they will tell you openly exactly where they are going and any potential for danger. If you can handle that information without allowing anxiety to overwhelm reasoning skills, you will be in a far better position to help them if something bad happens.

    The most long term valuable aspect of schooling is the development of one’s social network and connections, along with the habits and ethics. The actual information learned is rather limited in valuable application in the end. Who one knows and how one appears to others is of far more value than what one knows. For these reasons, there may be value in corporate social media. Simply teach the kid to understand how these places are both a trap and a tool. A trap, in that many of the smartest humans are manipulating users in ways that are nearly impossible for the users to escape. Never invest emotions into such a trap. Use the tool if needed for external social benefits, but use it as a manipulation tool with a layer of disconnect from who you really are. Teach them to use a work profile to isolate any apps from their device. That is just how I look at the issue.





  • I have a mental color wheel and go much further than most.

    I painted cars for nearly a decade. I intuitively know how various color hues are made. For instance, there are only two kinds of black. The most common is made from carbon and it is a yellow base. It will always tint a color towards yellow when added to any other base. Then there is the much more rare purple based black. Some color mixing systems do not even have a purple based black and it in impossible to hit some color matches as a result. Some special edition Harley Davidsons are too dark to hit with my old PPG mixing system. I usually kept a bit of purple black from BASF for this purpose.

    Another color that would blow your mind is this one white (that is used on Toyotas IIRC). It was so bright of a white that, when I first encountered it, I tried just using my brightest white base because in my mind, there was no way that tinting was going to produce a brighter white. Almost all whites go one of three directions in tint. They are all either yellow - most common, blue - maybe 2/5ths of white cars, or red - very rare at maybe around 1 in 20 and extremely subtle. All of these are very subtle to notice but to a painter they are plainly obvious.

    So this one Toyota white looked like I sprayed a blotch of grey primer even after using my brightest white. I was in trouble because a small panel job might turn into a whole side of a car to blend out a color difference like that in ways no one will see. I finally looked at the color formula from the color code and mixed an approximation of it. The formula involved a mix of odd colors, but the result was actually brighter pure white and that blew my mind. It did not tint in any tone or go darker at all but actually went brighter.

    I have some of the best color vision of any other painters I encountered. This is actually how I ran my paint business. When you mix paints there is a minimum amount you’re supposed to mix to make it right. It is really about the minimum amount that can be measured and how much of the smallest amount of a color is involved. So if the formula has 1% of this one red, and the minimum I can dispense is 1 gram, I must mix 100 grams of paint in total. I may only need 50 grams, but industry standard is that ai mix 100 regardless and have to use or toss it. I don’t need to use formulas like this. I can look up the base ingredients and make the colors from scratch in smaller quantity. I also kept around 10 bottles of common base colors that I would mix together. So if I painted a silver car and had some color left over, I would put that in my silvers bottle. Then on my next job with a silver car, I would take my left over silvers bottle tint it a little bit and spray that just over my primer over the repair. Then I would mix a very tiny amount of the proper silver from scratch and use this to blend out the actual final color coat. I did things like dilute the small amounts of colors I needed in special ways like by combining it with clear binder, solvent, or one of the base colors in the formula I was replicating. This gave me access to a smaller amount than the 1 gram of red.

    Painters must tint the formula for any color they mix anyways. As cars age, the color degrades for many reasons. So even when making the minimum formula, it is just a baseline for tinting. I simply flipped this paradigm and tinted everything while only using the formula as a reference. This means I spent far less on paint per job, and I could approach smaller repairs more cheaply than most people doing automotive paint. I also have hacker skills with clear coat application that make smaller repairs possible.


  • Last job I wore a race kit. Rode my clothes in and out one day a week and had permanent work shoes and a spare kit just in case. I often would leave a couple of hours early for work, do something like 30-40 miles on the bike then hit the shower at work before the day started. If it was nice out I would head out early, but I really enjoy riding at night when it is in the low 60’s. I’m lightning fast when it is cool and the lack of people on bike trails is really nice at night. Commuting started me down that path.

    I generally wore shorts and work tees or whatever bike brand reps gave me for free. Pay was absolutely trash though. I mostly worked at a main shop but would stop in at the other two shops a few times a month so those were 60-80+ mile round trip commutes. I’d often just wear my kit on those days.



  • It is more complicated than just price. It is ultimately an intuitive self awareness and scope thing. People lack depth to understand the details or ask others that do understand before they make a purchase. The majority of people are more oriented towards interpersonal interactions and experiential aspects of life in their fundamental functional thought. They struggle to see detail and nuances or question fixation and biases.

    We still live in the early era of human tribal primitivism when it is quite easy to exploit tribal stupidity on multiple fronts. For some it is fixation from initial exposure or emotional brand perception, others it is impulsive availability, for others they are masochistic misers. Abstractive thinking and understanding is rare in humans, and the majority do not understand it or value it in others.

    Walmart bikes are targeting misers first, but spontaneous availability and access, along with controlling the perception of what the low bar of the market is are major factors as well. Each of these three factors exploits a specific niche. Walmart is a rogue wholesale distributor selling directly to consumers using massive capital. They are privateers (legal pirates) in the retail market as are most big box stores. Piracy has always been a nice short term business model for gains. It just happens to be true that people of today like being raided raped and pillaged so long as it is done slowly enough without violence, the ship looks pretty and the pirates wear a suit. Even worse is when pirates become entrenched as monarchs and feudal lords. This is the next step in the evolution when piracy is normalized. Welcome to neo feudalism.


  • It is simply an entry level thing. You will find this in every market.

    In a bike shop retail market I can sell you a serviceable bike for $500 that will last, or an $800 road bike you’ll actually ride. Still the majority of bikes sold come from places like Walmart where they are made of unserviceable junk and are mostly nonfunctional. These are rarely ever ridden and often thrown away. In the shop I’ll sell 20:1 on the cheapest model to the next options up the ladder.

    It is strange to adapt to this kind of understanding at first, like just how skewed the real market is. I can target selling to clubs and teams but I can’t touch the the garbage bike market where most people reside.

    I think we are at a point where the influx of people into 3d printing are not real Makers or have any aspirations to be.

    The reality is that people are often simply stupid. They seem to think that saving a few bucks here or there is smart but are not bright enough to see that everyone doing the same thing are buying the junk product over and over. There is nothing more expensive than being a cheap miser.

    Ultimately, the only person that can fix stupid is ourselves. One can only inspire others to learn but can never force them. You cannot fix stupid in others. In the USA, stupidity is political currency and we have a long tradition of poor education and standardized exploitation. It is the American dream.

    I think LDO and Voron are the only super relevant open source torchbearers.