• 1 Post
  • 168 Comments
Joined 1 year ago
cake
Cake day: June 16th, 2023

help-circle


  • Where I worked we had a very important time sensitive project. The server had to do a lot of calculations on a terrain dataset that covered the entire planet.

    The server had a huge amount of RAM and each calculation block took about a week. It could not be saved until the end of the calculation and only that server had the RAM to do the work. So if it went down we could lose almost a weeks work.

    Project was due in 6 months and calculation time was estimated to be about 5 1/2 months. So we couldn’t afford any interruptions.

    We had bought a huge UPS meant for a whole server rack. For this one server. It could keep the server up for three days. That way even if wet lost power over the weekend it would keep going and we would have time to buy a generator.

    One Friday afternoon the building losses power and I go check on the server room. Sure enough the big UPS with a sign saying only for project xyz has a bunch of other servers plugged into it.

    I quickly unplug all but ours. I tell my boss and we go home at 5. Latter that day the power comes back on.

    On Monday there are a ton of departments bitching that they came in an their servers were unplugged. Lots of people wanted me fired. My boss backed me and nothing happened but it was stressful.










  • The solution for this is usually counter training. Granted my experience is on the opposite end training ai vision systems to id real objects.

    So you train up your detector ai on hand tagged images. When it gets good you use it to train a generator ai until the generator is good at fooling the detector.

    Then you train the detector on new tagged real data and the new ai generated data. Once it’s good at detection again you train the generator ai on the new detector.

    Repeate several times and you usually get a solid detector and a good generator as a side effect.

    The thing is you need new real human tagged data for each new generation. None of the companies want to generate new human tagged data sets as it’s expensive.