New (to me) AMD Bulldozer Workstation

[systems_glitch 111]

Picked this up off eBay:

 

http://www.ebay.com/itm/262157900280

 

Expandable to 32 cores for under $50, and will take up to 256 GB of ECC DDR3. The plan is to run Slackware 14.2 on it and use it as my main dev workstation. I'm running enough VMs nowadays that my current 4 core AMD APU box isn't cutting it, with running parallelized test suites for day-job work and whatnot. I've been running ZFS on Linux for a while with no reliability problems, so I'll probably use a bunch of the hotswap bays to do raidz or raidz2.

 

There's a pile of PCIe slots, looking like the Supermicro motherboard in there might actually support my Xeon Phi, too! I'm thinking I will probably end up with a 10gb Ethernet card too, and do a crossover link to my FreeBSD ZFS server.

[dinscurge 51]

actually those are interlagos, the pildedriver based opterons, but for stuff that scales well with the numa architecture, or just massive workloads they are pretty decent.

 

would just make note of the 16 dimm slots, if going to be running dual socket, as far as specing out the memory options, i.e. atleast 8 sticks if your going to run dual socket and want to take advantage of all the memory channels. assuming you would remember anyways though.

 

but eitherway will significantly outperform the apu box since besides the cores, you now have a mountain of cache in comparison, 16mb l2 + 16mb l3, as opposed to no l3 at all and prolly 4mb l2

[tekio 117]

Nice. Id love one for a vshpere system at home. Cheaper than a lot of Chrome books at Wal Mart, even.

[systems_glitch 111]

6200 series is still Bulldozer microarchitecture, Interlagos is a subfamily Piledriver is 6300 series IIRC.

 

But yeah, part of the reason I went with a dual-socket motherboard is the enormous quantity of RAM slots. I can buy cheaper, lower-density RAM which seems to be widely available as cast-offs from other people's upgrades, and still get to a reasonable quantity for this system.

 

Well, the machine arrived today, and it looks like something is wrong with the graphics card Super clean inside though, it smells like a brand-new computer. I kind of wonder if these were used for a trade show or something, and just surplussed as soon as the show was over. The other thing is, no onboard SAS controller -- it wasn't clear from the motherboard documentation that you could have IPMI (which this one does) and no SAS. It did ship with a 1 TB Hitachi/HGST SATA 3gb/s drive installed. There's a small dent on the top corner of the plastic bezel of the case, but that hardly matters. It came with 5x total drive caddies, so it's 3 short of being completely full.

[systems_glitch 111]

Slackware64 14.2 installed, running a lot of compiles and building up part of my dev stack. Currently compiling node.js -- `make -j16` is nice

 

Got some correctable ECC errors while building Ruby, so it looks like there might be a RAM upgrade sooner than later...

[dinscurge 51]

On 8/10/2016 at 10:04 AM, systems_glitch said:

6200 series is still Bulldozer microarchitecture, Interlagos is a subfamily Piledriver is 6300 series IIRC.

 

But yeah, part of the reason I went with a dual-socket motherboard is the enormous quantity of RAM slots. I can buy cheaper, lower-density RAM which seems to be widely available as cast-offs from other people's upgrades, and still get to a reasonable quantity for this system.

 

ooh yeah was wrong XD. abu dabi is the pile driver.

 

just meant might be one of those.. easily forgotten things if people didnt have experience with dual socket, that you would want to populate slots for each memory controller, to that, the minimum would be 8 sticks for full speed, that if you only populate one socket the other memory controller wont be there so.. probably cant populate any of the other 8 slots etc. just like it might be easy to miss the requirement for dual 8pin cpu power(usually anyways) lol :p.

 

edit: like with the newer soc stuff amd is going towards, if the servers follow suit.. depending on motherboards and stuff, if you only had one socket populated instead of 2, you would lose pci lanes and stuff aswell, where with current gaming setups with multiple graphics cards.. if you had a 990 chipset you could have 2 x16, but 970 only has 1, usually with the lower end consumer boards you can plainly see by the solder points which slot is x16 or can be x16, but with stuff like 2011v3 where different chips have different amount of lanes different slots will operate at different speeds using different chips which im sure could leave some people scratching their head about this low performing raid or whatever lol

Edited August 12, 2016 by dinscurge
added text

[systems_glitch 111]

So I'm against some sort of bottleneck which I haven't had time to identify yet. The full test suite of one of the day-job applications I work on runs significantly slower, but won't even utilize a full core. On my APU box, I was running a load average of 2.5 - 3 with all fans spun up. This sits around 0.75 load average. I'm wondering if it's a disk bandwidth issue, I currently have the 1 TB Hitachi/HGST SATA drive in that shipped with the machine.

 

Thinking I'm going to order a Samsung M.2 SSD and PCIe adapter bracket, I wanted to upgrade to SSD anyway. Direct PCIe attachment will rule out any disk/controller issues. I knew getting into this, from our server deploys, that single-core performance wasn't awesome with these CPUs, but not even being able to load down a single core seems like something else is my bottleneck.

[dinscurge 51]

8 hours ago, systems_glitch said:

So I'm against some sort of bottleneck which I haven't had time to identify yet. The full test suite of one of the day-job applications I work on runs significantly slower, but won't even utilize a full core. On my APU box, I was running a load average of 2.5 - 3 with all fans spun up. This sits around 0.75 load average. I'm wondering if it's a disk bandwidth issue, I currently have the 1 TB Hitachi/HGST SATA drive in that shipped with the machine.

 

hmm.. sounds weird probably need to look at bunch of jack to try to figure it out if that doesnt work lol. like micro code stuff/firmware, might have to recompile the software/get a different build using different compiler options or something.. who knows XD.

 

most of the stuff ive heard as far as issues was like some versions of 7zip dont use all the cores, or performance optimization stuff relating to the numa stuff, since.. not 100% sure how it operates.. but besides theres 2 sockets, g34 is a mcm, pretty sure each of the 2 dies for the 'chip' has its own memory controller, the l3 stuff might be split between the dies so.. if you access other stuff/channels it might be at lower speeds if its in the other parts/die but thats really specific optimization stuff should have nothing to deal with the cores not being used at all.

[systems_glitch 111]

I picked up a 128 GB Samsung SM951 PCIe attached SSD, going to grab a PCIe x4 adapter for full bandwidth. That ought to rule out a disk bottleneck

[tekio 117]

BIOS/UEFI update? If so hopefully it supports something like Asus Ez Flash (so Windows won't be needed  to deliver it).

[dinscurge 51]

13 hours ago, systems_glitch said:

I picked up a 128 GB Samsung SM951 PCIe attached SSD, going to grab a PCIe x4 adapter for full bandwidth. That ought to rule out a disk bottleneck

 

should once used a different disk/controller should eliminate any of those potential causes,

 

otherwise usual run down whole checklist of possible random things, settings/configs for programs, build versions, check for reported bugs or anything like that, if you were using linux who knows if the kernel your using has the smp module or whatever, there might even be such a thing just for the g34/numa stuff(joke aside, i havent looked into it much but start talking sockets vs cores there might be a kernel option or something applicable for this type of setup) etc remotely applicable conditions

Edited August 18, 2016 by dinscurge
added text

[systems_glitch 111]

So I added the Samsung SSD and got to pretty much equal run times with my AMD APU box. Obviously multitasking is still way more usable on the Opteron box, with all of those cores!

 

As an experiment, I installed Gentoo and took care to optimize USE and CPU flags for the architecture. After a day of compiling and working around bugs/oddities, I finally got around to running the test suite I've been using as a benchmark -- it's 5 or 6 minutes *slower* with Gentoo I'm sure Gentoo experts will tell me I'm doing it wrong...

[tekio 117]

RAM errors? Graphics card issue? I'm starting to think something at lower level, maybe?

 

If it were my machine: download a trial for Windows Server 2012 64 and install it:

  - works :: check Linux compatibility and the chipset/motherboard 

  - does not work :: check for BIOS/UEFI flash make sure it is current

    - works reinstall Linux

    - doesn't work :: pester seller for a return or replacement or enough to cover the cost of a new mother board

 

- More than likely problem solved. 

 

Maybe PSU as well? Do you have another PSU to swap in?

  

 

Edited August 27, 2016 by tekio

[systems_glitch 111]

I don't think there's anything wrong with it, I think I just had too high of expectations for it. It's a machine from 2011 that's known for having lousy single-core performance. It does compile C/C++ code *really* fast, with `-j18` in the MAKEOPTS. I may end up selling it to a friend -- paid a visit a few weekends ago, and apparently his VM server died, and he's been running his production VMs for his small business off a laptop!

 

I was simultaneously surprised and not surprised that Gentoo was actually slower. I've maintained for years that there's no real speed advantage from Gentoo for most hardware situations, but I didn't think it would be that much slower!

[dinscurge 51]

On 8/30/2016 at 10:30 AM, systems_glitch said:

I was simultaneously surprised and not surprised that Gentoo was actually slower. I've maintained for years that there's no real speed advantage from Gentoo for most hardware situations, but I didn't think it would be that much slower!

 

dunno would guess one would have to know all the right options/which compiler even you should be using for best optimization. but i always under the impression it was more the actual system efficiency than any individual program running significantly faster, like by trimming everything possible from the kernel, only having the tools you actually use/need. would be faster boot times slightly, the drive would be slightly faster because the more free space. if you had a brand new instruction set or something that you needed stuff compiled for etc like bunch of small things eventually resulting in some noticeable gains.

 

if you wanna see it do something fast try handbrake or 7zip lol

[tekio 117]

Was always under the impression Gentoo was supposed compile everything and optimize it for the CPU installed?  Anyway, I got tired of Gentoo just because it earned street cred online. Maybe I should just lie and say I use FreeBSD? But since everyone here knows I use OS X a lot - some BSD fanatics would probably own me for being a poser. :-(

[dinscurge 51]

17 hours ago, tekio said:

Was always under the impression Gentoo was supposed compile everything and optimize it for the CPU installed?  Anyway, I got tired of Gentoo just because it earned street cred online. Maybe I should just lie and say I use FreeBSD? But since everyone here knows I use OS X a lot - some BSD fanatics would probably own me for being a poser. :-(

 

dunno lol. i always thought it was sort of on you to do the work, but then ive only ever messed with the minimal install thing where you build it just mean.. gcc may not be the best option at all, especially for certain processors/versions without even getting into stuff thats highly optimized like the intel compiler thats super expensive that only works with intel and specific intel chips at that